Friday 19 November 2004

Maiden Voyage - Engineer's Journal 1

After years of preparation, the Maiden Voyage of Fablusi v2 has departed. Crew on board - Captain: Marie; First Officer: Albert and another 15 very experienced sailors. Destination: Sexual Harassment in the workplace (Bencon Industries).

It took about 2 weeks to load all the resources, the role information, configure the iSpaces, set up the tasks and look and feel. We then started enrolment. All the recruits were assigned roles.

We sailed into a storm - literally. The first was the Melbourne storm - several sudden flickers of power. I thought our server was protected by UPS (uninterruptable power supply), but obviously that did not protect us. So mysteriously, only one role profile was left in the database. Some of our sailors had to re-submit.

Then came the "random auto-self destruction" of the database. Every now and then, but not every time, the database created about 300,000 records of the same sim-mail reference. After a few of those uncontrolled, unplanned and unintended creations of a few million records later, the database started to slow down. The connection time-out was set for high speed venture and fast response from the database server. Hence time-outs kept occurring. It took me the better part of two days trying to figure out what has happened. You will now see a larger area on my head - hair being torn out during frustrations.

At the same time, I was trying to install another database server. [In heavy production environment, it is absolutely necessary, I know!, But I thought I was at the engineer room and thing should be OK, so there was no backup database!] The web server I am running is still a NT box. The new server is a window 2000 server. After I installed everything and moved a copy of the database to the new server, I found out that NT and window 2000 databases do not speak the same language. The ODBC on NT is speaking 8-bit (single byte) ASCII and window 2000 speaks 16-bit (double byte) uni-code! I have to go into the window 2000 database and change each field to single byte. It was done after two nights of 2:30 am bed time.

Then I found out that my window 2000 does not like to be a database server too. It happily serves the data if it is called by the web-server it is hosting, but refused to pass any data to my NT box. (User management problem...)

By now, the party has heated up. Rumours and harassment are flying everywhere. We almost reached the port of Sexual Harassment in the workplace. Marie decided to declare emergency and evacuated the ship and continue the journey on foot.

That gave me a chance to reboot the NT server. You know what, after reboot, the database no longer messed up. I was trying the whole day yesterday trying to bombard the database with all sort of data. It stood up and refused to misbehave!

I reckon, during the Melbourne storm, some memory might have been damaged and causing the database random misbehaviour. After reboot, the bad memory may have been blocked, or it was a soft-damage that it has recovered. Anyway, the database is running beautifully at the moment.

No comments: