Scientific Workstation (~$10000)

[disjtr1]

Hello,

I am not sure whether this is the most appropriate forum for this but I can not find a better one.

I have just started a new job as a computational scientist. For the first time in my career I actually have quite a lot of money to spend on a workstation. Of the order of $10000 each year for three years. What machine should I buy? I run CPU intensive simulations that are 'embarassingly parrallelizable'. I will almost certainly run Linux but possibly dual boot Windows.

My last workstation was a dual Opteron that I got built fairly cheaply by a local shop. I was very happy with it. My new employers want me to go for a big company like HP, IBM, Sun or Dell for warranty reasons. I am tempted to go for dual Xeon quad core X5355 processors. That would allow me to run eight simulations at once. Is that a better choice than the current Opteron set-ups? Or should I get a workstation based on an IBM power processor? I know nothing about those. I guess I could even get my own Beowulf cluster and add a few nodes each year. Any opinions?

Many Thanks,
Chris

 

[MichaelD]

Hi Chris, and welcome to the AT Forums.

You've already stated that your company is paying for it and understandably, they want you to BUY the workstation from a big OEM for warranty reasons; ergo, we won't talk about building a workstation.

With that kind of money, you have a lot of options. If you were definitely running Windows as your main operating system, I could easily make many recommendations. But, I'm unfamiliar w/Linux and hardware compatibilites...you'd have to check with the company you buy it from.

I'll make one recommendation for a Windows system, though. SuperMicro makes very good motherboards and cases, and they sell some really nice workstations.

A dual, dual core Xeon system with at least 8GB of RAM running Windows Server 2003, with a nice SAS/SCSI RAID 5 or a RAID 1 array for the OS and a RAID 5 array for the data crunching programs would be really fast and really reliable.

Something like this

LINK

or this would be a really good start.

Again, for Linux you'd have to do the research and make sure that whatever system you buy is compatible with it.

$10K is a really nice chunk of change; you will be able to get what you need for that money. Additionally, the forums at 2CPU.com will be an excellent place to get recommendations for a high end workstation like you're looking for...a little better than here, quite honestly...but AT has way more traffic. Good luck. :thumbsup:

 

[disjtr1]

Thanks for replying,

I will check out SuperMicro as a vendor. I am based in the UK and are considering these people since they seem to have experience with Linux workstations:

digital networks UK

Has anybody any feedback on them?

I am pretty confident on my choice of CPUs, the current Xeon chips seem to be the fastest. Regarding the disk drives, you mention RAID 1 for the OS and RAID 5 for the numerics. That mirrors the OS which makes sense and then RAID 5 is fast with some integrity checking, right? How many disk drives would I need for that set up? What rpm should I be looking at?

Thanks,
Chris

 

[MichaelD]

Some basic info on hard drives and RAID arrays:

AFA drive speeds go, SCSI drives run at either 10K or 15K and will have slightly faster access times than SATA drives which spin at 7.2K. The biggest difference is the price; a SCSI raid setup can be a lot more expensive than a SATA raid setup. SCSI drives almost all have a 5 year warranty and these days, most SATA drives do as well. The big price justifier that SCSI used to have was the warranty/reliability. SATA has closed that gap considerably though. For the same amount of storage (GB) a SCSI array will be about 2x the price, but you'll get about 33% faster IO (10K rpm vs. 7.2K rpm).

IMO, unless you are running a huge array that will see a lot of IO, you'll be just fine with a SAS/SATA array. You'll save a nice chunk of change that can be used to buy more ram; 2GB sticks are stratospherically priced. :Q You'll need the money.

RAID 1 = mirroring. You need a minimum of two drives. If one dies, you lose nothing.

RAID 5 = striping with parity...speed with security, so to speak. You need a minimum of three drives. You can lose one drive and lose nothing. More drives=more input/output, and the ability to have a hot spare.

A RAID 1/5 setup is a classic server/workstation setup. The OS is safely mirrored on a small array, and your databases and storage run on a fast, large array. The advantage to splitting them up is that they are independent of each other. If your OS gets hosed, it wont' affect your data (assuming you don't have a virus, of course!). If your database crashes, your OS is fine.

AFA how many drives to use goes...assuming you go with a Supermicro setup that has 8 drive bays, I would go with a 2 drive RAID 1 mirror for the OS and a 6 drive RAID 5 array for the database/storage. You could designate one drive as a hot-spare (it gets put into action automatically if one of the drives in any of the arrays dies), and that would leave you with a 5 drive RAID 5.

That's just one of many possible setups, but one that would work well. Most of my servers at work are setup this way.

The nice thing about going with a workstation/server grade setup (i.e. $$$$) is that you have those options available to you.

 

[disjtr1]

That is really a very comprehensive reply. Thank you for explaining those hard drive issues to me.

 

[Sunner]

Well, not knowing a whole lot about the software you expect to have running, I'll drop a few more general tips

Do make sure to profile your applications before you buy so much as a keyboard.
For example, getting an expensive disk system would be a complete waste if you run a simulation that ends up with data that will fit in a spreadsheet.

Also, do you have any directions about what Linux distribution you'll be running?
Most shops tend to have settled for something, more often than not in mid-big businesses it's Redhat or SuSE for support reasons.
Redhat has an official HCL on their site(here, I presume SuSE has something similar over at their site.
Of course, that's not to say something that's not on the list won't work, most any piece of good hardware will work just fine, those lists can be helpful in making sure though, and sure is a nice word when spending a fair amount of cash

And lastly, you could always have a look at IA-64.
In some cases IA-64 boxes can be extremely fast, most of those tend to lean towards the HPC direction, so it might be worth looking into.
Also, Linux/IA-64 will probably be a less cumbersome combo than for example Linux/POWER.
Downside being that HP is pretty much the IA-64 vendor for anything but extremely large HPC setup where SGI sits.

Oh and by the way, as far as disk speeds go, if your application turns out to be disk intensive, do make sure to figure out if it's just doing loads upon loads of sequential I/O ala video recording, or if it's more random I/O.
S-ATA disks can manage just fine for sequential I/O for a cheap penny, while SCSI/SAS disks excel in the latter(well, actually they're generally ahead in the first department too, only the difference is nowhere near as big, and sequential speed is cheap and easy to increase with more drives).

 

[ForumMaster]

if your work is massively multi-threaded (as it sounds) then getting a dual quad core machine is a great idea. is it available already? i know that dual dual core computers (such as the mac pro) are available but are 8 core computers available?

as for RAID, if you're thinking about it, how critical is your work?

RAID 0: Striped Set
RAID 1: Mirrored Set
RAID 0+1: Striped Set + Mirrored Set (4 disk minimum)
RAID 3/4: Striped with Dedicated Parity
RAID 5: Striped Set with Distributed Parity
RAID 6: Striped Set with Dual Distributed Parity

RAID 6 would be the absolute safest but it has to write a lot of data.

as for your last question, PowerPC processors were used

you'll excuse me for this link as i don't mean to advertise apple's offering but they did use PowerPC chips so this review of the G5 or the 970FX is appropriate. but forget running windows on these cause it's going to be slow.

link

 

[Steve]

You could also look at a Sun workstation, such as the Ultra 40: "Runs Solaris 10 Operating System, Red Hat Enterprise Linux, SuSE Linux Enterprise Server, and Windows XP Professional" (32 and 64 bit XP)

 

[jkresh]

if you could describe a little bit more about your uses that might help with coming up with a system. Ie your code is highly threadable so an dual quad core machine or maybe a quad or oct dual core might make sense. How much data do you work with (as a raid 5 array with a bunch of 15,000rpm sas drives can get pretty expensive) also how memory intsenive is your work? If the memory load is light and independent threads makes a lot of sense (and network bandwidth is not an inssue) then spending 2,000 on a quad core main box an the other 8 on a bunch of cheap dual core machines to cluster together (either a beowolf type setup or you could even use windows clustered server (free for now)) . The clustered aproach would get you the most cpu power but be much lower on memory and drive bandwidth then 1 10,000 workstation. One benefit to going with a 4 or 8 way opteron setup over a dual quad core zeon is depending on who you buy it from you might have a drop in quad core upgrade for next year (allowing you to relatively cheaply double your number of cores (8 to 16 for 4way 16 to 32 for 8 way).

 

[tribbles]

Sun may very well be the way to go for you, since most of their workstations are designed to run on multiple operating systems. If you're going to be running an enterprise Linux distro, I wouldn't have any worries about purchasing any of Sun's workstations.

 

[Sunner]

Originally posted by: tribbles
Sun may very well be the way to go for you, since most of their workstations are designed to run on multiple operating systems. If you're going to be running an enterprise Linux distro, I wouldn't have any worries about purchasing any of Sun's workstations.

They don't really run anything that any other tier1 workstation won't.
Solaris will probably be better supported since that would make a nice Sun/Sun combo, but for one, the OP didn't mention Solaris, and secondly, Solaris runs on a lot of non-Sun hardware.
The only workstations that Sun offer that are "truly Sun" are the Blades, and those are based on UltraSPARC-IIIi's, and those aren't exactly speed demons.
The rest is rebadged Intel/AMD stuff, just like HP/Dell/IBM/Whoever.

 

[Steve]

Originally posted by: Sunner

Originally posted by: tribbles
Sun may very well be the way to go for you, since most of their workstations are designed to run on multiple operating systems. If you're going to be running an enterprise Linux distro, I wouldn't have any worries about purchasing any of Sun's workstations.

They don't really run anything that any other tier1 workstation won't.
Solaris will probably be better supported since that would make a nice Sun/Sun combo, but for one, the OP didn't mention Solaris, and secondly, Solaris runs on a lot of non-Sun hardware.
The only workstations that Sun offer that are "truly Sun" are the Blades, and those are based on UltraSPARC-IIIi's, and those aren't exactly speed demons.
The rest is rebadged Intel/AMD stuff, just like HP/Dell/IBM/Whoever.

The funny part is your screen name

 

[sparks]

Dell's workstations are capable of dual quad cores so if your application benefits, then an 8 core Linux machine would be awesome.

 

[disjtr1]

Thank you for all the replies. They contain some thought provoking ideas.

It seems that the key is that I think carefully about exactly what I will be doing. My current work essentially splits into two parts:

a) Monte-Carlo simulations. These are complex biological simulations. Lots of interacting individuals. They are CPU intensive. I do write quite a lot to disk but it is the calculations that are the bottle neck. Because you typically want say 100 replicates at a single set of parameter values then these are trivially easy to parrellelize. You simply run each replicate independently. I have done this on clusters and because each replicate is essentially an independent program it is very efficient.

b) Solving large sparse matrices. This requires a lot of calculations but I think the bottle neck here is more to do with how much RAM you have. These matrices are BIG and need to be stored in memory. This task could be parrallelized but I have not done that yet.


Regarding Operating System. Most code I run is written by myself so it does not matter too much. I do use some free libraries and applications though. Esentially it will be some flavour of Unix. I run Windows on my laptop. On my last workstation I ran Fedora and had no problems. I was able to compile any software or library that I needed. I may look at SUSE this time. Would using a professional unix distro i.e. Solaris enable me to buy faster compilers (all my code is C or Fortran) than gcc?

I think that my money would be best spent on CPU rather disk speed. I liked the dual quad core Xeons (with 8GB RAM) as giving me potentially eight threads allowing me to tackle 8 replicates at a time for a) plus each core being fast enough so that b) could be solved quickly (without parallezing too). However, jkresh's suggestion of a 4 or 8 way Opteron having more expandability in future is worth thinking about. Especially since I will have $10000 every year for the next three years. From this respect the cluster idea is nice too. I am a little worried though that with a cluster I may spend too much time on my own support (the IT guy here is not going to be that helpful) whereas just running a Linux workstation and installing apps is something I am confident to do myself.

Some of you mentioned alternatives to the x86-64 architecture in particular IA64, is that faster?

Many Thanks,
Chris

 

[disjtr1]

Sorry as an afterthought, what would be really helpful would be benchmarks (probably just MFlops would be good) comparing alternative architectures and processors. Does anything like that exist? I have had a look at the Java Scimark benchmarks:

Scimarks

From this the Suns look good as does Core2Duo in general.

 

[TriggerHappy101]

Try http://www.falcon-nw.com/

They make crazy powerful systems. I wouldnt go through their online builder but I would call them up and explain your postion, they will custom make you a system.

 

[Slammy1]

Ah, the heady days of grad school doing computational chemistry on a Silicon Graphics workstation. It's been a while, but I remember a computational firm came in for a presentation and they ran everything on Apples for graphics. I'd talk to colleagues, see what they run. You'll probably put most of the money into a display, unless things have changed that much.

 

[DrMrLordX]

Originally posted by: disjtr1

I think that my money would be best spent on CPU rather disk speed. I liked the dual quad core Xeons (with 8GB RAM) as giving me potentially eight threads allowing me to tackle 8 replicates at a time for a) plus each core being fast enough so that b) could be solved quickly (without parallezing too). However, jkresh's suggestion of a 4 or 8 way Opteron having more expandability in future is worth thinking about. Especially since I will have $10000 every year for the next three years.

Task 'a' that you listed will probably be helped most by an efficient, multi-proc/multi-core system. If you can get an 8-way dual-core Opteron, that's 16 cores, and support for an enormous amount of memory for task 'b'. However, so long as task 'b' remains single-threaded, it will probably be faster on an Intel Woodcrest/Clovertown (Xeon) system so long as there is sufficient RAM available.

Some server reviews, such as this one:

http://www.anandtech.com/IT/showdoc.aspx?i=2897&p=1

show that Opteron scales better than Woodcrest/Clovertown when going from 4 to 8 cores (and presumably this trend would continue when scaling from 8 cores to 16). However, a single isolated core from a Woodcrest or Clovertown Xeon will outperform a single core from an Opteron.

Some of you mentioned alternatives to the x86-64 architecture in particular IA64, is that faster?

IA64 = Itanium. It's . . . uh . . . I shouldn't try to explain it. My only piece of advice is to stick with x86/x86-64 if you're already familiar with it. Here's Itanium in a nutshell:

http://en.wikipedia.org/wiki/Itanium

and . . .

http://en.wikipedia.org/wiki/IA64

. . . some info on IA64.

 

[Cat]

You might think about GPGPU through the new Radeon or GeForce cards, if your work is really that easy to parallelize.

 

[AirForceElite]

i am just curious though
what is it that you do, that requires $10000 computer
could you please list the task that need to be run?
(just trying to understand the need for such expensive systems)

 

[Steve]

Originally posted by: AirForceElite
i am just curious though
what is it that you do, that requires $10000 computer
could you please list the task that need to be run?
(just trying to understand the need for such expensive systems)

Originally posted by: disjtr1
My current work essentially splits into two parts:

a) Monte-Carlo simulations. These are complex biological simulations. Lots of interacting individuals. They are CPU intensive. I do write quite a lot to disk but it is the calculations that are the bottle neck. Because you typically want say 100 replicates at a single set of parameter values then these are trivially easy to parrellelize. You simply run each replicate independently. I have done this on clusters and because each replicate is essentially an independent program it is very efficient.

b) Solving large sparse matrices. This requires a lot of calculations but I think the bottle neck here is more to do with how much RAM you have. These matrices are BIG and need to be stored in memory. This task could be parrallelized but I have not done that yet.

 

[Beatnik]

Looks doable. The Quad Intels are about as good as it gets right now.

I just did a quick check on the Dell website. You can config a Dell Precision 690 like:

Quad X5355
Quad X5355
Redhat EL 64bit
8GB 667 FBDIMMS
3 x 146GB 10KRPM SAS Drives
Raid 5 SAS

That got me to $7700. I suspect that you really want to be 64-bit OS. If you have a single app that is going to exceed 2GB or 4GB, then you get pushed into wanting to do 64-bit. You could cut cost on the above by skipping the RAID5 SAS, but if you care about your data, then you probably want it at least under RAID5.

 

[Mday]

Originally posted by: AirForceElite
i am just curious though
what is it that you do, that requires $10000 computer
could you please list the task that need to be run?
(just trying to understand the need for such expensive systems)

He mentioned running simulations. Those require fairly beefy systems. It's a matter of high speed processing with a lot of data. So that's what's different that consumer and thus require more on everything from CPU power, memory and storage (typically). Remember, if you want something a little faster, you gotta pay a lot (relatively), so that's why it will run so high.

He's also going to need FB memory with ECC, and those are a mint.

 

[DrMrLordX]

He only needs FB-DIMMs if he goes with a Woodcrest or Clovertown system. Socket F is registered ECC DDR2 as I recall.

 

[disjtr1]

quote:

just trying to understand the need for such expensive systems

------------------------------------------------------------------------------------------------

Well as explained earlier, I do need such an expensive system. I sometimes wonder though why someone would give me the money for it. These simulations should help with things like setting fishing quotas. Applications are a long way off though.

------------------------------------------------------------------------------------------------

quote:

You might think about GPGPU through the new Radeon or GeForce cards, if your work is really that easy to parallelize.

--------------------------------------------------------------------------------------------------

That is rather a novel suggestion. I just looked at GPGPU on wiki. I think that most of my work would be a bit tricky to implement like this, but there are some simulations I do, that may lend themselves to it. Would the standard graphics card that comes with a Dell:

256MB nVidia Quadro FX3500 Graphics Card

be suitable for this?

---------------------------------------------------------------------------------------------------

quote: I just did a quick check on the Dell website. You can config a Dell Precision 690 like:

Quad X5355
Quad X5355
Redhat EL 64bit
8GB 667 FBDIMMS
3 x 146GB 10KRPM SAS Drives
Raid 5 SAS

---------------------------------------------------------------------------------------------------

I will likely end up getting something very similar to that. I will think about whether I need RAID 5 or not. Regarding Dell as a supplier, are their components - motheboards, RAM etc. all top quality? The alternative would be HP and they would install Linux for me.

--------------------------------------------------------------------------------------------------

Thank you for all the replys. This is proving to be a very useful exercise for me.