Sunday, March 29, 2009

PostgreSQL, Slony-I, pgadmin, win32, First time config/installation

In one occasion, I have to use Slony-I for Database replication. As a starter, I look for my friend (Gooooogle) for help. She showed me some tutorial: some rather outdated with version or some having broken pic link. After quite a lot of struggle, I have managed to get it working, and would like to provide a better tutorial/notes for the installation.

Here is my software config:
PostgreSQL : 8.3.5-2
Slony-I : 1.2.15
OS: Windows XP, Windows Server 2003

Notice that my Postgres version of 8.3.5-2 is not available to be downloaded in official site, which only contain 8.3.5-1 (and higher version) in the mirror I have tried. (Minor version is not showing anywhere, AND 8.3.5-1 is NOT working for me somehow) I am not sure if the procedure will work on future versions but the general wisdom should able to apply to most situation.

My tutorial will be based on Dave's PostgreSQL Blog with extra information that help any first time comer to setup everything gracefully.

The target of the procedure is to setup a localhost replication between 2 database of your choice.

Initial Installation:

Install PostgreSQL server and run Application Stack Builder to add Replication : Slony-I support.
Start pgAdmin III, goto File -> Options, setup the Slony-I Path. On default case it should be C:\Program Files\PostgreSQL\8.3\share
Use pgAdmin, connect to the localhost server, and remember to save password.

Database Preparation:
If it is your first time playing around Slony-I, do not apply anything on the working database yet, which would probably screw things up.
With pgAdmin, backup with default setting a working database into an *.backup file.
Create 2 new database, which will be acting as Master and Slave, with names of your choice. I will use db_1 and db_2 which represent the master and slave respectively.
Restore db_1 with the backup file obtained above, using default option. (Full restore)
Restore db_2 with the backup file, using "Schema only" option.

Setting up Replication at pgAdmin

All the setting is located in Databases : db_1 / db_2 : Replication

Right click on db_1 : Replication and choose "New Slony-I Cluster"
Join existing cluster : UN-checked
Cluster name : sample
Local Node Left : 1
Local Node Right : Master Node
Admin Node Left : 99
Admin Node Right : Admin Node
Press OK.

Right click on db_2 : Replication and choose "New Slony-I Cluster"
Join existing cluster : Checked
Server : localhost
Database : db_1
Cluster name : sample
Local Node Left : 10
Local Node Right : Slave Node
Admin Node : 99 - Admin Node
Press OK.

Upcoming is path setup, which is not well documented in any reference.

Right click on db_1 : Replication : sample : Nodes : Master Node : Path and choose New path
Server : 10 - Slave Node
Connect Info : host=localhost port=5432 user=postgres dbname=db_2
Conn retry : 10
Press OK.

Right click on db_2 : Replication : sample : Nodes : Slave Node : Path and choose New path
Server : 1 - Master Node
Connect Info : host=localhost port=5432 user=postgres dbname=db_1
Conn retry : 10
Press OK.

Now we define Replication set.

Right click on db_1 : Replication : sample : Replication Sets and choose new Replication Set.
ID : 1
Comment : sample set

Now we choose the table to be replicated.
Right click on Replication Sets : sample set : Tables and choose New tables
Table : Any table of your choice
ID : 1
Index : Auto filled when choosing table.
Press OK.

Repeat the above for all the tables that you want to replicate with increasing IDs. If your Database contain hundreds of tables that need to be replicated, this manual action can takes hours. The following SQL can help this progress.

SELECT COL.table_name as table_name , CON.constraint_name as constraint_name FROM information_schema.columns as COL INNER JOIN information_schema.constraint_table_usage as CON ON CON.table_name = COL.table_name WHERE COL.table_schema='public' GROUP BY COL.table_name , CON.constraint_name ORDER BY COL.table_name;

and use the result to fill the following SQL.

SELECT _<ClusterName>.setaddtable(<MasterID>, <ID>, 'public.<table_name>', '<constraint_name>', '');

In our example,
<ClusterName> : sample
<MasterID> : 1
<ID> : increasing counter from 1
<table_name> : from above query
<constraint_name> : from above query

Notice that the query assume that there is one on one mapping between table_name and constraint_name. Otherwise, multiple matches will exist in result set. In my occasion, Hibernate is used and extra constraint is added to some of the tables, and I have added "AND CON.constraint_name not LIKE 'fk%'" in my WHERE clause to filter those constraints.

Now we create a new Subscription
Right click on db_1 : Replication : sample : Replication Sets : sample set : Subscription and choose New subscription.
Origin : 1
Provider : 1 - Master Node
Receiver : 10 - Slave Node
Can forward : UN-checked
Press OK.

Here ends the Database preparation.

Slon daemon setup

Slon daemon is needed to be running in all server involving the replication. The setting is quite tricky and differ from each other for different version of Windows being used.

To ease the installation, add slon.exe to the PATH environment variable.
Control Panel : System : Advanced : Environment Variables
Search for PATH in system environment and append ";C:\Program Files\PostgreSQL\8.3\bin" or other path if your installation is not on default.
Start command prompt and try "slon" to see if the path is correct. If successful, an help screen will be shown.

Register new service:
slon -regservice Slony-I

Create Engine Configure files:
Create folder C:\Slony
Create C:\Slony\db_1.conf with the following contents (2 lines)
cluster_name='sample'
conn_info='host=localhost port=5432 user=postgres dbname=db_1'

Create C:\Slony\db_2.conf with the following contents (2 lines)
cluster_name='sample'
conn_info='host=localhost port=5432 user=postgres dbname=db_2'

Add engine to the daemon:
Do the following in command prompt :
slon -addengine C:\Slony\db_1.conf
slon -addengine C:\Slony\db_2.conf

Double check the above with :
slon -listengines
You should see the path of the two conf files being listed.

Before starting the daemon, there are extra setting to be done for running the service successfully.
In folder C:\Documents and Settings\<Current User>\Application Data\postgresql,
there should be a postgres password file pgpass.conf. Slon need to access this file to connect to the database for replication. However as Slon is a daemon that do not run as current user, so the owner of Slon service should contain such a file.

Copy the file to C:\Documents and Settings\postgres\Application Data\postgresql
On default, this folder do not exist and have to be created manually (unless you login as postgres and saved password before). Make sure user "postgres" have read/write access to the password file.

Go to Control Panel : Administrative Tools: Services and look for "Slony-I"
Enter Slony-I's settings and access "Log On" tab.
Log on as "This account" and enter "postgres".

Now we are ready to start up the service. Either right click on Slony-I in services and choose Enable, or do "net start Slony-I" in command prompt.

Replication should takes place immediately if every setting above is correct.

Diagnose

In most cases if you are working on it for the first time, there would be some errors in the setting and the replication is not working. You can do one of the following to check up errors.

Using pgAdmin, access db_1 : Replication : sample : Nodes : Master Node, on the right panel, access Statistics tab. If replication is working fine, the numbers keep changing. Otherwise multiple "0"s will be shown. Statistics is also available for Slave nodes.

Using Event viewer (Control Panel : Administrative Tools : Event viewer, Slon's activities will be shown in Application session.

If Slon running as Service didn't show its log nicely to read, we can run Slon in command prompt in Debug mode, as follows:
slon -d 4 sample "host=localhost port=5432 user=postgres dbname=db_1"
slon -d 4 sample "host=localhost port=5432 user=postgres dbname=db_2"

Common Mistake

FATAL main: Cannot connect to local database - fe_sendauth: no password

This indicated that pgpass.conf is not found or not accessible.
Make sure pgpass.conf file exist for the user running the Slony-I service but not the current user. And make sure the user running the service should have full access the the password file. Also make sure that Service owner is correctly set up.

Conclusion

Setting up Slony-I for the first time really takes time and effort. I hope this guild can help whoever that want to start using Slony-I for replication on Windows.

Clearly, more in-depth knowledge in replication is needed for creating a real working Replication solution. One can find more information in the reference session below.

Good luck for your Slony-I setup.

References
http://www.slony.info/documentation/
http://www.pgadmin.org/docs/1.8/slony.html
http://pgsnake.blogspot.com/2007/09/setting-up-slony-i-with-pgadmin.html
http://www.nabble.com/No-replication-on-Windows-Vista-td19488860.html

Friday, February 20, 2009

How to build a Web Host on your computer that is accessible from the outside world?

I assumed that your followed my previous post for your webhost (http://ccstone.blogspot.com/2008/09/how-to-build-web-host-on-your-computer.html) and it is now the time make it accessible from outside world.

1) Change Default Port of your WebHost

1a) Open \mowes\apache2\conf\httpd.conf
1b) Goto line 127 (or somewhere around there
1c) Search for "Listen 80"
1d) Change 80 into any number between 2000 to 8000 (as you like), e.g. 2337
1e) Save and exit
1f) Restart mowes
1g) Give your homepage a test: originally you access a page like http://localhost/mingpao.php , now you have to access the same page with http://localhost:2337/mingpao.php

Notes: the reason behind changing default port is that most ISP blocked port 80, and it is not an easy (noob) task to redirect traffic

2) Setup Port forward in your router (if you have one)

There is no detailed instruction for this setting, as everyone use different brand and model of router. But in general you can follow procedure

2a) Open Command Prompt ("Run..." then enter "cmd")
2b) Type ipconfig and press enter. This is an example of result:

Windows IP Configuration

Ethernet adapter Local Area Connection:

Connection-specific DNS Suffix . :
IP Address. . . . . . . . . . . . : 192.168.1.222
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.1.1

2c) Start browser and browse the page indicated by "Default Gateway". E.g. http://192.168.1.1/ . You are now in router setting.
2d) Enter password as needed. Those login information can be retrieved from your router manual.
2e) Look for Firewall setting / Port Forward / DMZ / etc.. Every brand and model differ from each other, so you have to navigate yourself to the destination.
2f) Setup a Port Forward to your computer. E.g. map port "2337" to "192.168.1.222" as IP Address in your ipconfig result.
2g) Save setting and retrieve your EXTERNAL IP address from your router setting. It can usually be found in "Connection Status" page or "Internet Setting" page. E.g. "202.169.200.234"
2h) Test if your setting is correct. Instead of using http://localhost:2337/mingpao.php, now try http://202.169.200.234:2337/mingpao.php

3) Setup a free domain name for your webserver

Some new router include this function internally. If so, follow the instruction in router manual. For those routers without such setting, or if you do not use a router at all, follow this instruction for easy domain setup.

3a) Visit http://www.no-ip.com/services/managed_dns/free_dynamic_dns.html
3b) Sign up the FREE DNS redirect service
3c) Register a Domain of your choice. E.g. helloworld.myvnc.com
3d) Download and install the No-IP DUC client.
3e) Add in the Hosts your newly registered domain
3f) Test your new host. Instead of http://202.169.200.234:2337/mingpao.php, try http://helloworld.myvnc.com:2337/mingpao.php
3g) (Optional) Setup the "Port 80 redirect" function at NO-IP DUC, so that you don't have to enter the port 2337 (I didn't do it personally)

-------------

After all those setup, you can now access your webhost from anywhere in the world, including your Laptop, iPhone, PDA, etc.

Enjoy

Sunday, September 14, 2008

How to build a Web Host on your computer

Here is the procedure for setting up a web host at home and have PHP/MySQL ready with php_curl library support.

Step 1) Download MoWeS

Goto http://www.chsoftware.net/en/useware/mowes/mowes.htm
Choose "Download" from the left menu.
Click "No thanks, take me to the download"
Click "I Do not have a MoWeS Protable II... "
When choosing the component, check
Apache2
MySQL5
PHP5
PHPMyAdmin
and press "Download now".

Notes: I do not know why people keep asking if they should download the SE version of the modules. Just follow exactly the component named above.

Step 2) Extract & start up
Extract the content to C:\mowes. At this moment you should have mowes.exe as C:\mowes\mowes.exe
Run mowes.exe, it prompt you for setup those components you have chosen.
Start the installation and press "OK" several times.
The webhost will start automatically and your browser will start and goto the hosted website.

Step 3) Get ready with CURL library
In the MoWes Portable II windows, press options.
ON: Start Apache and MySQL automatically when MoWeS starts
ON: Hide the Apache window
OFF: Delete xx logfiles after stooping
OFF: Start Webbrowser
OFF: Start External App
other setting left as default.
Press Accept and OK
Press "End" in your Mowes window. The webhost will be down.

Goto C:\mowes\php5, copy the 2 files:
libeay32.dll
ssleay32.dll
to your windows system32 folder ( usually c:\windows\system32).

Open C:\mowes\php5\php.ini in text editor. Modify the following lines:
205:
implicit_flush = On
321:
max_execution_time = 120
674:
extension=php_curl.dll
696:
extension=php_openssl.dll
727:
date.timezone = Asia/Hong_Kong
Save and exit text editor.

Notes: Some of the changes include removal of a ';' (semi-colon). Be really really careful and the line should be completely like the one written above. No exception for any of the above lines, each line contain changes to be made.

Notes 2: for line 727, it specifies the default time zone so that any time displayed will be under that time zone. If you are not from Hong Kong, take a look at HERE for more cities of your choice.

Step 4) Testing
Start mowes.exe. The webhost will be ready.
Put any file C:\mowes\www , for example a.zip
Open your web browser, URL = http://localhost/a.zip
If successful, you will be prompted to download a.zip
Put mingpao.php to C:\mowes\www
If successful, you will be reading today's mingpao

Notes: mingpao.php is the ultimate testing script for your configuration. It can be obtained here.

Friday, May 23, 2008

BlackBerry Bold 9000 Hands on

As some of you may know, I work for a company that concentrate on developing Input Method for various platform. I am the one responsible for development on Blackberry and today I received the newest product from RIM, the Bold 9000.

If you have ever played with any of the 8xxx series and now holding a brandnew 9000, you may be impressed by its new interface, new functionality and new plastic/metal case.

However, to make things short, I conclude with one sentence: It looks well, it plays badly.

If you are looking for tech details for Bold, read CNET news, or if you want some screenshot, visit crackberry. The one on my hands looks exactly the same. And it's awesome. (Don't compare it with iPhone though)

As a Chinese Input Method developer, what I concern most, as any Chinese would do, is the ability to display and enter Chinese characters. When comparing with the Font on Pearl, which looks like Chinese Char in CWin31 or just eTen, the ones on Bold are clean and clear, and are as good as those on newest Windows Mobile. Bold support HKSCS natively, so there should be no problem reading Hong Kong News. Moreover, Bold has built-in support for multi-lingual User Interface. One can choose its favorite Interface language right from SetupWizard.

The true power of BB handheld relied on its ability on messaging. Actually I didn't see many Hong Kong Blackberry users typing in Chinese (I see users on MTR mostly). Maybe it's because its poor support on Chinese Char Typing. (Visit here for details on Chinese Support).

Bold supports quite a number of Chinese Input Method (natively, like UI) , like 粵拼, 注音 and 倉頡. However, after the "first taste" of using the input method, I lost my chance to test further... THE INPUT METHOD BROKE MY BLACKBERRY!!!

I started up my Blackberry in English Interface with 粵拼 input method. I went to "message" and compose my first email. Everything is the same as in 8xxx machines, nothing surprising. As I have no real experience using 粵拼, I tried some common strokes like "lee" "sin" "sang" and something like that. The control is really bad as if it is programmed by a guy without any experience on Chinese typing. I tried to type a little faster, but it seems to be extremely slow (on searching output?). The speed is unacceptable and I started losing temper. I tried to hit the keyboard in random and finally it showed me "Input Method Error, restarting..." (not exact phrase). That message popup since then whenever I typed fast enough, which is like 3 key per second (not 3 char per second), which is common for any "normal" Chinese typer. I lost my interest and go on testing something else, like Web Browser.

About half an hour later I switch the machine into Chinese Trad Interface, hoping a native Chinese Interface would gain better performance working in Chinese. I start the message editor again and try typing like before. I sense strangely as i switched between input method. The BB gave me 注音 chars when the active input method is 倉頡, wtf. I keep typing until the proper 倉頡 show up, then after like 2 sec... my BB hanged up. There were no response whatever I do, including pressing and holding the POWER button. It showed me the BB is shutting down and is actually hanged.

There's nothing I can do, and I remove the battery and put it back again (a normal hard reset for BB). I think it will be fine since I do reset like 50-60 times a day when i worked on it (install/uninstall require restarts). But no luck, it gave me "SYSTEM PROCESS ERROR" and force me to reset. And of course, no luck after reset. And i am now holding a bricked Blackberry Bold.

To be lucky is that the machine is a sample from the network provider and I can wait for an exchange a few day later. I should not have great hop as it's just a demo machine anyway. I hope that the stability can be improved before its release. The release date is not announced, and they should have lots of things to do. (and Thx to those sloppy programmers, my state in my company is as firm as it can be)

Sunday, May 4, 2008

Charles Wilson's War

昨天看完DVD版的Charles Wilson's War,很失望。

失望的不是Tom Hans的表現、不是Julia Roberts的表現、不是故事、不是整個制作的任何一個部份。反之,在這幾個方面我也很喜歡,很欣賞。

使我失望的唯一一個地方,是香港方面的宣傳。

Charles Wilson's War是人物傳記,講述的是一名美國的Congressman,利用他的影響力,以協助阿富汗於八十年代被蘇聯入侵時作出反擊,說白一點,是反戰電影。

此片是2007的制作,香港於2008年3月左右上映。當初第一次接觸到此片,是在RoadShow上看到的宣傳片,中文名好像叫做「韋氏風雲」。宣傳片上最令我留意的,當然是Tom Hans。(自Forrest Gump開始,只要是Tom Hans我也會很留意)也就是說,我看此片的主因,就是看角色,怎樣看此片也是「大卡士」之作吧。

因那是RoadShow的宣傳片,聲音差點聽不到,所以只能用眼去判斷那是什麼的片。宣傳片的內容除了各人的對手戲外,最特出的就是Julia Roberts從水裡走出來的一段,占時間肯定是最長的。看了不只一次,就是看不出這是什麼主題的作品,沒有所謂,反正都會看。

看罷,不能說是非常出色,但也找不到大問題(除了歷史的部份,有關蘇聯撤兵原因的一段過份簡單)。可能因為此片太「美國」,有些背景未能理解,這是自身認識不足的問題,說不上是什麼缺點。

但這港版的宣傳片,也實在是太爛了。其實片中有很多出色的鏡頭,例如在巴基斯坦難民的一段,又或者是Soviet Copters等,都值得加入宣傳片中。至少,從宣傳片中我看不出阿富汗,甚至看不出戰爭。

想深一層,宣傳片的爛,責任何在?是不是發行商的錯?我想也未必。

爛的不是宣傳,而是香港觀眾。如果打正旗號是反戰片,有多少港人會去看?關心國際事務的人不斷減少,炒股的人數則不斷提升,「關心國際大事」就即是「關心會影響股價的事」的意思。縱使Julia Roberts的角色有如「夏蕙姨」,不打緊,找一段載黑超的就無問題。沒有「美色」的西片,怎樣騙你們入去看?

Tuesday, April 29, 2008

php_curl + cookies

curl_setopt($ch, CURLOPT_COOKIEJAR, "d:\Web\PHP\cookieFileName");

很多script會使用cookies,但cookies path最好是full path