- Feb 5, 2001
- 2,989
- 18
- 81
December 15th is the current target date. Have a big outage soon for a big step in the master migration database, and then when we're back up start sending out the mass e-mail saying it's time to start moving over.
- Matt (Lebofsky)
Big outage: master migration (merging old SETI@home classic science database with BOINC). Everything needs to stop for this to happen, and may take 24-76 hours. Don't want new users coming over to BOINC and finding it down for several days. Of course we'll post news items about this when we have a better idea when this will happen.
Remember this is only the current planned cutoff date, and may slip a week or two, but we're really trying to stick to this one.
- Matt (Lebofsky)
Yet another post, this from the "classic"-forums:
Regarding posts about the "December 15" shutdown date. Sorry I didn't post that
over here - bear in mind that nothing is really official until it makes it to
the news on the front page (or technical news). Though we really really really
want to stick to this schedule, it may slip a week or two. Anyway, expect a
mass e-mail soon, funny warnings from your classic clients, a note on the classic
home page, etc.
- Matt (Lebofsky)
Actually, we might be able to get by without an outage - we're working that out now. You can see why I don't post things to the front page until we are a lot more certain about events/dates/times...
- Matt
3 new messages, updated 12.11.2005
This 1st. is answering another forum-post, 10.11.2005:
Just checking in and reading the last post - I figured I'd chime in and clear up some points.
So currently the Sun v40z simular to jocelyn is now $6,695.
This is true, but remember it needs some disk drives. Currently attached to jocelyn is a 3510. I think the one we have (12x73GB drives) runs around $25K
(before discount). The 3510 was donated by Sun, we bought the v40z.
Look at the Obsolete and Dying Servers that support UCB.
Reminder: the data recorder, which has been taking data 24/7/365 at Arecibo observatory since 1998 (except for various telescope related outages), is running on a Sun IPX (about 10-12 years old) with two SCSI drive enclosures (one with a 1GB drive, the other with a 2GB drive). Works great. Just want to point this out - it may be obsolete, but if it works why spend $$ to replace it?
In a month we'll start "repurposing" (I hate that word) SETI@home classic servers for BOINC, or retire them. Of course, there are some new hardware needs, like a replica database server, a better web server, etc. that we can't really get out of our old hardware.
Also their main storage device is a rather expensive SNAP appliance. (don't have details for it) SNAP is also listed on the sponsors page.
The Snap 18000 (donated by SNAP) is being used for workunit storage, and data archiving. We have an nStor FC JBOD attached directly to kryten for result storage (using software raid), and a similar nStor attached to castelli for our master science database. As well we have a NetApp used for everything else (home accounts, classic SETI@home databases, SERENDIP data, and more). So the Snap is an integral part, but not the main storage device.
- Matt
2nd. is from 11.11.2005:
Note, Seti_Enhanced is heavily influenced by Angle-Range, there expected 15x difference between VHAR and AR 0.226. "Normal" angle-range is roughly 10x crunch-time compared to "classic", so the server-load will likely be even lower... atleast till highly-optimized versions is released...I haven't checked the "active" user count BOINC, but there are more SETI/BOINC users than active classic users at this point in time, and the numbers are growing/shrinking, respectively, every day without any effort on our part.
If everybody migrates over from classic to BOINC, that would double the current number of BOINC users. Assuming all of them are active, we'd go from 700K results processed per day to about 1.4M.
Of course, we only create 1M results per day, as that is our data collection rate, so we will be short 400K results/day. So, in essence, our load can only grow another 300K/700K= 42%.
Before you freak out, we won't be cutting anybody off. Remember that we will soon be shipping out the new clients with more science analysis, which will take approximately 4 times as long. So this will effectively cut our result load by 75%. So instead of a demand for 1.4M results, it'll only be about 350K/day, which is half our current load.
As for a flood, we shall see. I imagine a lot of classic users will not be swayed by an e-mail and will hang on until the last possible second, and then either quit or migrate, so there will be a spike on the December 15th (or whatever the cutoff date ends up being). Hopefully by then we'll have the new science code in place, and the load will then be half of what it is now and a spike on new users won't cause any harm.
A lot of other users probably aren't paying attention, and won't get an e-mail (due to a stale e-mail address), or check their clients for warning messages, or check the SETI@home page... and won't even notice anything is wrong until months after the switchover. Not much we can do about that.
- Matt
So, the 3rd. forum-post, 12.11.2005, 00:15:54 GMT:
Stocking up on WUs to send during our science database merge next week. If all goes well, there won't be a "real" outage - just a "behind the scenes" outage. A backlog of 2M results to send should last a few days.
And once we're done, mass e-mail time! We're still on for the December 15th classic shut down date, even though the e-mail won't be going out until the 18th of this month (or later). There will be postings on the web site and the classic clients will all be popping up windows warning of the impending shutdown before the mass e-mail starts going out.
- Matt
(Fixed a typing error, it's a huge difference between AR 0.225 and 0.226...)
Updated 15.11.2005:
November 14, 2005
Tomorrow we are going to start merging the SETI@home classic master science database with SETI@home/BOINC. This will take several days, during which some back-end servers will be offline (splitters and assimilators). We stored up a large queue of work - we are hoping this will keep all BOINC clients occupied during the length of this outage. If we run out of work, we will have to stop the rest of the back-end servers. After we begin the outage, progress will be detailed in Technical News.
Technical News:
November 15, 2005 - 21:00 UTC
Today we started the big master database merge. This step is simple in essence: we are combining all the scientific data from SETI@home classic and SETI@home/BOINC into one big database. However, this is the culmination of many months of effort.
What happened during those months? Among other things, we had to migrate all the data off of one server onto another, find and remove redundant data, add new fields to old records and populate them, write and test software to merge databases while keeping all relational constraints intact... Basically a lot of cleanup, a lot of testing, and backing up the entire set of databases between every major step.
While this merge is happening, nothing can be updating either of the master databases. We shut off the splitters (which input new workunits into the database) and the assimilators (which input new signals). Over the weekend we created a backlog of about 2 million results, so this should keep the clients well-fed for most of this outage. The assimilator queue, of course, will grow significantly. When everything but the signals themselves have been merged, we may turn the splitters back on (at this point they won't screw up any relational constraints by adding new work to the mix). When the merge is completely done, everything will be turned back on, and the assimilator queue should quickly drain.
Right now science being done in SETI@home classic is redundant to the science in SETI@home/BOINC, so this will be the last of the big science merges. The classic project will be shut down before the end of the year.
We do hope to eventually place the master science database on a faster machine with bigger/faster disks. This will mean another outage, but it will be a simple unload/reload of the data (as opposed to a unload/reload/correct/merge).
Technical news:
Addendum (22:45 UTC): We hit a major snag when trying to merge rows which had columns of user-defined type. We'll back out of the merge so far and turn everything on for the evening, and probably try again from the beginning tomorrow morning.
Hmm, a small forum-post , answering a question:
Looks like Classic and Boinc are doing the SAME exact wus? Is this how anyone else is reading this? That any Cpu cycle spent on Classic from this point forward is really just for credit?
Yes, yes, and yes.
Just hit a show-stopper and we're halting the merge. See updated tech news note. Bummer. But we'll try again tomorrow.
- Matt (Lebofsky)
SETI@Home "classic" web-page:
SETI@Home/BOINC web-page:November 15, 2005
On December 15, 2005, after 6 years of operation, this project will shut down. But SETI@home is not going away. We're alive and well and running on the BOINC distributed computing platform. It's easy to switch over - just follow the instructions on the new SETI@home web site. We hope to see everyone there!
At the time of shutdown, the stats for this project will be frozen but will continue to be available on the web.
Thanks to all original SETI@home participants for their tremendous dedication to the project. You made SETI@home into something of lasting significance.
November 15, 2005
The old SETI@home Classic project will stop issuing work on December 15, 2005. Anyone still running SETI@home Classic should deactivate it and install SETI@home/BOINC (see instructions under 'Getting started' on this page).
Technical news:
November 16, 2005 - 23:00 UTC
Today we had our usual Wednesday outage to back up the database and upload directories, but also took the opportunity to move some equipment around.
We work closely (and more or less share the same staff) with a seperately funded project that does a survey of hydrogen in our galaxy using SETI@home data. They recently bought a new 3U server (a dual-opteron with 4GB ram and 3TB of SATA drives) which we are in the process of incorporating into our server closet. To do so, we had to move our BOINC upload/download server out of the way. In fact, it had to move out of the cramped closet altogether.
As always, this was no small task, as this server needs a network connection to our private ISP, as well as a connection to the gigabit switch to communicate with the workunit storage server. But our only choice was to move it into an office which had regular old LAN ports and nothing else. In short, we needed to invest in a bunch of long ethernet cables and move some plugs around on the lab's main switches. But the move went smoothly and everything worked after we powered back up. It's great when that happens.
Meanwhile, we're still dealing with the master merge woes from yesterday (see previous post below for more information).
There is a major shell game when merging these two databases, as there is a chain of relational constraints that tie all signals (spikes, Guassians, etc.) to their result, which is tied to a workunit, which is tied to a workunit group, which is tied to a tape. These constraints must be kept intact, even though merging two databases means the ids of all the rows change in the process.
We developed and tested a whole bunch of SQL which did the job, but never tested it on the two tables that contain rows of user-defined type, which in turn contain lists of indefinite size. The Informix SQL engine balked at these, as it should.
Since we have a bunch of C++ code which already does inserting/updating into these tables, Jeff has been busily working today on a fix using C++ instead of SQL. We hope to have this finished and tested and perhaps try again with the master merge tomorrow (after we catch up from today's outage).
November 18, 2005 - 00:30 UTC
Regarding the master database merge (see posts below), it looks like it is going to be postponed again - at least until next week, and probably sometime after that (since next week is short due to the holiday). We are continuing to develop C++ tools to move data around, and we don't want to rush into anything (and potentially screw up the database) before the software is fully tested.
As well, we need to coordinate the merge with the mass e-mail asking all Classic users to move to BOINC. We were hoping to start that this week, but the merge delays have postponed everything (since we may require a long outage and don't want a flood of new users finding the project inaccessible upon first try). It's likely we'll start the mass e-mail early next week and do the merge much later on.
And now the email is on it's way. Haven't got it yet, but AFAIK it looks like this below:
SETI@home completes the switch to BOINC
----------------------------------------------------
SETI@home has completed its transition to BOINC (Berkeley Open Infrastructure
for Network Computing). BOINC, developed at U.C. Berkeley, is a framework for
volunteer computing projects like SETI@home.
Switching your computer to the new "SETI@home/BOINC" is easy. Visit
http://setiathome.berkeley.edu for instructions. We'll be shutting down the
"SETI@home Classic" project on December 15. The workunit totals of users and
teams will be frozen at that point, and the final totals will be available on
the web.
Scientists at other universities are using BOINC to create volunteer computing
projects in areas like molecular biology, high-energy physics, and climate
change study. See http://boinc.berkeley.edu for more information. BOINC lets
you donate computer time to multiple causes, divided up however you want. For
example, your PC could spend 40% of its time searching for extraterrestrial
life (SETI@home), 30% studying climate change (http://climateprediction.net),
and 30% studying protein folding, design and docking (Rosetta@home:
http://boinc.bakerlab.org/rosetta/).
You can run SETI@home 100% of the time if you want. But we strongly encourage
you to participate in other projects. Doing so ensures that when SETI@home is
down or has no work, your computer stays busy helping other researchers.
Support SETI@home
----------------------------------------------------
SETI@home is increasingly reliant on individual donations. Without the support
of individuals like you the project will not be able to continue. We urge all
SETI@home participants to make a yearly donation to keep the project going:
http://setiathome.berkeley.edu/donate.php
----------------------------------------------------
Thanks again for your participation in SETI@home.
The SETI@home Team
http://setiathome.berkeley.edu
Technical news:
November 22, 2005 - 21:30 UTC
We began sending out the mass e-mail yesterday warning SETI@home classic users that we are going to close down the old project on December 15th. It was sent to all 200,000 of the active classic users by this morning. Inactive classic users are being e-mailed at this point.
Due to the influx of new BOINC users (and the unfortunate timing of some googlebots and other web spiders) the load on our web server was extremely high for the past 12 hours. To fix this, we finally deployed a second web server to split the load. As DNS updates spread throughout the internet, the load on klaatu (the original single web server) decreases while the load on penguin (the new secondary web server) increases. Both are Sun D220R's (2 x 440MHz Sparc, 2 GB RAM).
Part of the problem was that the web servers were configured to spawn more many clients than actually necessary, which left lingering, unused threads open on the database, which in turn lead to the database running out of connections. Some users saw messages to this effect when the load on the web servers was at its highest.
This looked like a database problem, when in fact we are currently enjoying a 10% performance boost on the database. Last week we moved some memory off the myisam tables (which contain web forum info and not much else) and slated it for the innodb tables (which contain user, host, result, workunit, etc. tables). The myisam tables didn't need the excess memory.
By the way, the master database merge (see below) is currently on for the beginning of next week.
Also, they've just re-configured the "classic"-server, so the client will now complain each time you download more work. This from the cli:
You are running a client from the original SETI@home project. This project will be shutting down on 12/15/2005. Please join the new SETI@Home project at http://setiathome.ssl.berkeley.edu/
November 30, 2005 - 22:30 UTC
So the master database merge is at a complete standstill. Unless everything suddenly works, we probably won't embark on this adventure until after the December 15th cutoff date for SETI@home Classic. It has become a programming/database nightmare where each fix or workaround brings forth another unexpected show-stopper.
Our server closet is in flux. The SETHI project (which uses SETI@home raw data to study hydrogen in our galaxy) recently bought a new dual opteron system (4GB RAM, 3TB drives) which we wanted to rack up in our closet, but kryten (the BOINC upload/download server) was actually in the way. So we rolled kryten into our secondary lab. But first we had to route a Cogent connection and a link to our internal gigabit switch into that lab. Also in this lab are isaac (the boinc.berkeley.edu web server among other things) and jocelyn (the BOINC database server), which we hope to move in the closet shortly after Classic is shut down.
When this happens, we'll be able to turn off sagan (the Classic data server) and get it out of the way, so we can remove a set of four A5000 (disk arrays attached to galileo which hold the now-defunct Classic master science database). And all this is just the beginning of what is shaping up to be a large-scale shell game.
We also updated DNS maps and URLs to continue balancing the web load as well as move the BOINC core client downloads off isaac and onto kryten. With the warning e-mails still being sent, the new core client downloads have been peaking out at 40 Mbit/sec. Since isaac, which handles these downloads, is on the campus network, this was adversely affecting others. So we moved all that traffic onto our Cogent link, which is now close to topping out at 100 Mbit/sec at any given time. All BOINC core client downloads, SETI@home science client downloads, SETI@home/BOINC workunits and SETI@home Classic workunits are all going out over our single Cogent connection. Of course, Classic activity will ramp down significantly over the coming weeks, so bandwidth constraints shouldn't be an issue.
A forum-post, 01.12.2005:
More Technical News:How about.. crashed classic science database? I just rebooted it, stats scripts have been restarted, usually take days to generate all the pages that everyone wants to see, etc.
- Matt
December 6, 2005 - 04:30 UTC
With the influx of new users, bottlenecks were bound to happen. A couple nights ago we started dropping connections on the upload/download server (kryten). This server was also serving the new BOINC core client downloads. We immediately moved the client downloads onto the campus network which was ugly, as this added about 20 Mbit/sec of traffic onto the regular campus network.
On Monday morning we fixed this by making the secondary web server (penguin) the BOINC client download server. In its former life penguin was the BOINC upload/download server so it already had the plumbing and hardware to be on the Cogent network. So without much ado, we were able to move the core client downloads off the campus net. But what about the secondary web server? Well, another Sun D220R (kosh) wasn't doing very much at the time, so we plopped apache/php on that and made it the backup web server. Some people might be getting failed connections to our home page as DNS maps need a while to propogate throughout the internet.
Meanwhile, we were still dropping connections on kryten. At first we thought this was due to the upload directories (physically attached to kryten) getting too large, as the assimilators were backing up (and they only read files in the upload dirs). Upon checking half the files in upload were "antiques," still leftover from server issues way back in August. We will delete these files in good time. We increased the ufs directory cache parameters on kryten but this didn't help at all. So our current woes must lie in the download directories (kept on a separate server) or some other bottleneck further down the pike we haven't discovered yet.
And while all this was being diagnosed and treated we actually started the master science database merge. This is why most of the backend services are disabled, and will remain off until the first half of the merge is done (about 2 days from now). We hope the results-to-send queue lasts us through this first part. Having these back-end services off is actually helping kryten catch up on its backlog of work to upload/results to download.
More to come as we discover more about current server issues and progress further with the database merge...