February 23, 2005 - 23:30 UTC
A sudden, unexpected power outage due to a blown breaker shut the whole BOINC project down for several hours (along with all the other projects in the lab). The cause is still unknown, so there will be a scheduled power outage in the near future to hunt for electrical problems. We do know this: we just can't seem to catch a break around here.
We are checking all tables in the database before restarting the data servers. We were able to gracefully shut down many servers on battery backup (UPS) before the batteries drained, but not all of them.
Earlier this morning the project was off for some routine maintenance (tweaking the BIOS on the database server to get rid of spurious error messages and snapshotting for database backups). An hour after we brought everything back up the power went off.
We were able to gracefully shut down many servers on battery backup (UPS) before the batteries drained, but not all of them, including the new BOINC database server. So the data is scrambled, and mysql refuses to start. Our last backup to tape is a week old. This week's tape backup was about 60% finished when the power went out (Murphy's law in a nutshell).
The good news is we have a replica database which should be up to date. The bad news is that this had disk errors upon booting up and its drives are still resync'ing. After that, we'll have to check the table integrity on the replica - if we're lucky and mysql is able to start, we can then dump the data from the replica back onto the master and continue right where we left off.
February 24, 2005
UPDATE: Yesterday afternoon a breaker blew and power for the entire lab went down. We came back on-line three hours later, but due to servers shutting down ungracefully we had to check every table in every database. This process took all evening and still continues as of now (17:30 UTC). When these checks complete we can start the data server, web server stats, etc.
February 24, 2005 - 23:30 UTC
Update on yesterday's outage: We are still dealing with some database fallout. Most of the classic SETI@home systems are up - enough that we can serve workunits to users. However, BOINC is dead in the water until we get at least one database server up and running.
With the master database corrupted beyond repair, we turned all our attention to the replica. Its disks finished sync'ing last night, and after some file system checks the machine booted and mysql started just fine. A battery of tests revealed no corruption.. until we got to the result table. Of course, that's by far the biggest and most important table in the database. We are attempting to repair it now.
Assuming we can repair it with little or no data loss, we will then dump all the data from the replica back onto the master. If we're lucky, this will be done by tomorrow morning and we can start revving all the engines back up.
Please note that since it was a slower machine than the master, the data on the replica database server was about 30 minutes behind real time. We did try to limp both systems along to sync the replica data up even further but no dice. So, when we do get back on line it will be as if there was a half-hour hole in time during which all uploaded results were lost (and any user profile updates, message board postings, etc.). We sincerely apologize to all our users for this loss.
Court brought in a UPS from his personal server collection. So the master database will be protected while we scramble to purchase another. The database server was unprotected yesterday because is was in our lab, not in the data closet where all of our UPS's are. We were/are just weeks away from a data closet reorganization designed to make room for the DB server.
Originally posted by: amdxborg
Wow what an operation! Seems like classic is dead again... Hope they get everything up soooon!!!
Originally posted by: MDE
Seems like there's a new SETI outage every week...
Originally posted by: DaFinn
Dayum, hurryy... running out of WUs real quick...
February 25, 2005
The database has been restored and we are bringing up services. More info in Technical News
February 25, 2005 - 20:00 UTC
The database has been restored with a loss of the most recent 1/2 hour of processing just before the crash. Credit gained during that short period is lost and some folks may see transient download problems.
February 25, 2005
The database has been restored and all services are back up. The data server is very busy right now. There will be upload and download problems until the load normalizes.