ARGH MySQL / PHP (again?)
I don't know if I've ranted on this last time, but it happened again with another site, so I shall rant again. For context, I'm the technical admin of a certain large site, which I won't even name due to NSFWness -- I'm just mentioning it here so that visitors from that site know who I am :) A brief timeline of events:
- 2 days ago, I move the database to a separate server, as MySQL grinds to a halt when the database is too big to fit in RAM.
- At around 11pm yesterday, the site suddenly dies, printing "500 Internal Server Error" in response to all requests
- Restarting the web server fixes it, but the old PHP processes don't die, and the new ones lock up pretty quickly too
- PHP isn't actually printing any error messages, it's just locking up; LigHTTPD only prints "All PHP processes are busy, try again later"; MySQL logs show no error messages.
- It seems that the site is fine for as long as nobody tries to post new content.
- I try stopping and restarting the database, it says "Can't stop database"
- I finally realise what happened -- the disk with the database tables filled up, then MySQL crashed and died horribly, corrupting chunks of the tables. Then when PHP asks the database what's up, and the database says "error", it doesn't print "there's a database error", it just dies horribly and locks everything up, causing ISE's and using up all the ram until everything around it dies too.
In contrast, my preferred database server, Postgres, ran fine throughout the incident -- read-only requests were served as normal, and if anything tried to add data it would be told "the disk is full"; no crashing, no corrupt data, no problem. Also, it seems to scale better, currently dealing with a database of 8 * the size of RAM quite happily.
It's 7am, so I'm going to bed now; first thing I do when I wake up will be start work on giving this app Postgres support; then I can uninstall mysql since nothing else needs it \o/
2008-01-06 18:17:40 -0600