Archive for August, 2011

I might have temporarily solved the issue

Friday, August 26th, 2011

The issue of the crashing server wasn’t an as easy fix as I thought. The crashing database table wasn’t the culprit at all. The CPU kept hanging at irregular intervals anyway. I was running on two CPU cores and decided, just because I ran out of options, to downgrade it to run on just one CPU core. Speed wise we’re not missing out on all that much currently – it’s still very snappy and the system load is at 0.42 when the old system was going between 10 and 20 the last weeks. Much to my surprise the system has now been up for 17 hours without an issue. Not really ruling out that it might go down at any moment, but it does seem to be behaving now.

If it stays up during the weekend I’ll send over the error messages from the kernel when we ran two CPU cores to the hosting company and see if they have come across it before. It’s an issue that needs to be fixed as I’m currently running the biggest package for 1 CPU they’ve got so if I want to crank it up I have to go for at least two CPU cores.

I will now try to get all the member functions going so that we can start reporting all the results. It’s a busy motorsports weekend as always! :)

The server that was awesome – at one hour intervals

Thursday, August 25th, 2011

FrustrationThe DNS should have propagated for all of you now which means you can all see Driver Database on the new server. I got it working last night and it was blindingly quick! Googlebot (the spider Google uses to index the webpages) and Yahoo crawler all went crazy with happiness because now they could index the site a lot quicker. Googlebot was downloading a site every 0.15 seconds and the server wasn’t even breaking a sweat. Happy happy joy joy!

But then it just broke down. All of a sudden I couldn’t connect. I could log in to the system and saw that it wasn’t running at 100% like I thought it would be, it was idle. 0% of the CPU’s were occupied and it just sat there twiddling its virtual thumbs with nothing to do. Odd, to say the least. I rebooted the machine and it was back like it should like nothing ever happened. I started looking at the system logs and started to find worrying things. Both the httpd (web server) and the mysql (database server) were being blocked according to the kernel. I started googling, looking for an answer only to find that it was A) A bug in the kernel, or B) a hard drive going corrupt. I wasn’t satisfied with any of these answers.

After an hour or so the server went back again. Same procedure as last hour. Rebooted, pulled my hair and cursed a bit.

Then I suddenly noticed that I was getting strange PHP errors on the site suggesting to me that I had forgot to import one of the tables. I found it odd as I was pretty sure it had worked before. I had a look in the database only to find that the table responsible for adding pageviews to the profile pages (for the popularity and most buzz lists) was corrupt. I repaired it and the errors went away. After an hour the server went down again.

I rebooted, looked in the database and saw that the very same table was corrupt again. I had a look in the error log from the database server and found out that some 10 minutes before the server stops responding that table goes corrupt. The server then tries to write to it a few times for each viewed page on the site (which at Googlebot’s and Yahoo’s rate were more than ten pages per second) spitting out error messages all over the logs. And then the shit goes down. A coincidence? I was hoping it wasn’t.

I removed the code that calls that table and decided to head for bed (it was 4 in the morning by then).

It looks like the server’s been running for over five hours now. I’m hoping the problem’s fixed now, eventhough I won’t be betting on it.

The small price to pay to keep it running might have to be that the most buzz and DriverDB popularity will be a thing of the past. Worse things have happened.

Driver Database on the new server

Wednesday, August 24th, 2011

I’ve just pointed the DNS records for the main site to the new server. It may take a few hours for it to start working due to DNS caching but hopefully it will start to work for most of you shortly. I have no idea how the site will behave on the new server hardware but I’m starting out at two virtual CPUs at 2.8 GHz each and 2 GB of RAM. I’m hoping that will do for now, but there’s always the possibility of cranking it up (it will cost more, of course).

If you get to the new server you’ll see a “Driver Database is now located on a new server!” below the Members heading. I haven’t yet had time to check all the member functions so logging in and editing stuff is still offline.

If you find anything weird – PHP errors or noticies please comment on this blog post and paste the URL you got the message on. :)

Database in place

Wednesday, August 24th, 2011

Things have moved a bit slow but finally now everything’s in place on the new server and I can start to test that everything’s working like it should. So far it isn’t, but it’s just a matter of changing a few lines here and a few lines there in the code. Nothing major, but it will probably take a few hours as there are lots of files to go through.

Downloading / Uploading

Tuesday, August 23rd, 2011

Moving of the files is taking a little bit longer than expected. I had forgot just how big Driver Database has become lately. With the old server going on its knees downloading a compressed 3 GB archive of the site was a job for the early morning while I caught up with some sleep. Now I’m uploading the archive to the new server and this time it’s my pathetic upstream bandwidth that’s the bottleneck as I’ve only got 0.5 Mbit upstream.

6.1% complete at the cruising speed of 115.8 kB/s. I guess my ISP doesn’t think there are any legal reasons to why people need decent upstream speeds. :(

That’s where we are at the moment. :)

Press play on tape

Monday, August 22nd, 2011

Press play on tapeThe files containing the code have now been transferred to the new server. Up next are the image folders. Then it’s time for the database. Progress…

Change of plans!

Monday, August 22nd, 2011

The old server is really struggling. The MySQL database server is crashing like there’s no tomorrow and everything’s more or less horrible. So there’s a change of plan. Originally I was going to do a lot of work on the site on the new server before moving the database over; fixing things, optimizing stuff and rewriting a bunch of sloppy code. That will take time and with the old server having more and more problems we’re really running out of time. The new plan is to move the site as it is – and that work will start later tonight.

In a few hours time I will shut down the member functions in order to freeze the database before the move. This basically means that you won’t be able to log in and submit any data or stuff – but it’s kind of hard to do that anyway when the server’s going down every minute. I will then transfer the code and all the files to the new server and work on getting it running there. Once it seems to be running I’ll point the DNS entries to the new server and hopefully things should go better from there. I have no idea of the time frame on this but I’ll try to get it done during the night.

New server status update #5

Friday, August 19th, 2011

server upgradeAll of Driver Database’s blogs (driver blogs, news and updates) have now been running for a few days on the new server. After a little boost in the memory department they’ve been running reliably as well. The speed is quite decent and at this point I see no need for more memory or more CPU cores for this server (to keep costs down, of course).

The second new server has been set up and it’s the one that will host the main site. All focus will now go to optimizing the main site on the new server. I’ll rerwite a whole lot of the code as it’s rather sloppy in some places and will try to make it as quick as possible to give it the best conditions possible in its new home. :)

More to come!

New server status update #4

Monday, August 15th, 2011

LinuxThe updates and driver blogs have now been running on the new server for a few days. During the night the server refused to co-operate and I had to reboot it. Looking at the logs it appeared to have ran out of memory. One of the upsides of being on a cloud server is that there’s no need to physically add more RAM to the server so after a few clicks of the mouse I added another 512 MB RAM to the system.

Will monitor it closely to see if that’s enough. I think it ought to be, unless there’s a memory leak somewhere.

On another note I guess you may have seen that we suffered some data loss last Thursday. While trying to optimize a slow database table it in some weird way imploded and delete two thirds of the data. Not what I have intended, eventhough it did speed up things. ;)

The latest backup (which was four days old) was in place a few hours later and I spent all of Friday reconstructing lost connections in the database. We lost a few championship points and positions as well as team info that was added in the last couple of days but fortunately there was no major harm done.

Next up is to move the news.driverdb.com blog to the new server – hopefully without having to upgrade RAM again.

New server status update #3

Thursday, August 11th, 2011

the speeeeedAll the driver blogs are now located on the new server and are therefor way faster than on the old server. There should also be a bit less load for the main site that is still located on the old server (that move is a bit more complicated).

In the coming weeks I’ll open up the driver blogs for new registrations again – something which I removed over a year ago due to the extreme amounts of spam bloggers signing up (I had over 2000 spam blogs signed up in less than a week). Got a new anti-splog plugin that hopefully will take care of things.

Now work will continue on making the fastest Driver Database ever seen (shouldn’t be too hard, right?) on the new server. There will be some layout changes as well…