Ideas for Putting Together a $70000 Midrange Server

[MikeK614]

This isn't the typical thread about how to troubleshoot the installation of a 6600GT, but I figured I would try this forum because I know there are a lot of very intelligent and knowledgeable who frequent this site. I work for an engineering company and we are currently looking for additional processing capacity for a flow modeling (CFD) software package. The models we run can be very large and we require several processors to run them in a reasonable amount of time (It takes about 3 days to complete one run on 7 Itanium 2 processors to give you an idea). I received a quote recently from a major server/supercomputer manufacturer for $150,000 for a 16 processor system Itanium 2 system with 3TB of storage, or $100,000 for 8 processors and less RAM. That is simply more than we are looking to spend on this current project. Much of that cost was for an ultra high end proprietary networking interface. A system like the one they were quoting might be great if you felt you needed to add hundreds of additional processors at some point, but we really only need probably 32 at most.

Looking at the benchmarks published for the program we are using, processor for processor, Opteron based systems were quite a bit faster than the Itanium 2 solutions I became very intrigued by this and started looking at all the other available options. Some of the options from IBM looked pretty good and seemed to be a fraction of what the original system would have been. Now I am just wondering if there is anything else out there that I am not thinking about.

Does anyone have suggestions for a 16+ processor system, 2GB per processor, that runs Linux and is less than $70000. Processor speed and communication between the processors are the biggest performance drivers, we don't need a disk storage system that is anything special. Would it be worth looking at trying to build our own custom system if we were able to support it ourselves? Are there other processors that might be as good of or a better value for the dollar than an Opteron system like a 64 bit Xeon based system or something else I haven't even though of? I'm just trying to get alternate ideas at this point because I really don't think (or at least I hope) paying $100,000+ for a medium sized system is our only option. Thanks in advance.

 

[InlineFour]

definately do not build it yourself, especially at this price point. i don't think $70,000 is considered a "midrange" server. i don't even think dell's top of the line server has 16 CPUs.

 

[phisrow]

You pretty much need to talk to a pro at this level, unfortunately. With dual core AMD chips, you can get as many as 16 cores in a moderately standard, single image, box. Any larger than that is deep into low end supercomputer territory.

Give the Cray XD1 system a look. http://cray.com/products/xd1/index.html

There may well be other systems in a similar vein; but it looks to be more or less what you might want.

 

[Missing Ghost]

I would take a look at sun's SPARC systems. Your task looks cpu intensive so that would be my #1 choice.

 

[Inspector Jihad]

i dunno, i think my computer can run hl2 as good as that one...so whats really the point?

 

[theMan]

its not for gaming, its a server

 

[Inspector Jihad]

Originally posted by: theman
its not for gaming, its a server

really...

 

[feelingshorter]

I dont think most of the people here know anything about that type of stuff. But gezzes...70k computer!

 

[Fullmetal Chocobo]

IMHO, your best best would be to get some blade servers, and create a cluster for the work you need. My machine supercedes most, but a $70k machine is something that you want a warranty and service staff on.

 

[Greenman]

How about 8 daul core cpu's? Bottom of this page.
http://www.tyan.com/products/html/barebone.html
Comes with a 1350 watt PSU.
I'd call Tyan just to see what they have. Can't hurt to ask.

 

[Powermoloch]

wow

 

[MikeK614]

I think our best bet might be Opteron blades. But the real unknown for me is the interconnect you would use to actually implement something like that. Gigabit ethernet is the worst, but you might be able to get more processors to offset its slower speed, and it might work sufficiently in that role. Something like NUMAlink might be the best, but it is probably overkill for this size application, plus it is extremely expensive. Is it better to pay $20000 for a Numalink architechture just to squeeze every last possible bit of performance out of each processor, or just use that money to buy additional processors. I wonder what sort of scaling a Opteron based system could handle using Infiniband or Myrinet. Like I said, 32 processors is probably the max we would ever need. Power5 and SPARC chips might be great just because they have an architechture that can handle being incorporated into machines that have hundreds of processors, but they usually dont offer the best performance per processor. And $70,000 is actually an el-cheapo solution considering the $150,000 quote we were originally looking at.

 

[MikeK614]

Greenman, that actually looks very interesting. That is the type of alternate solution I am looking for. The Dual core part of the equation really makes it look attractive. I will take a look at that. Keep them coming.

 

[NoStateofMind]

Man! What would I do with $70,000 to build a gaming BEAST! errr...I mean a server..yeah..a....server. *clears throat*

 

[imported_goku]

Originally posted by: Fullmetal Chocobo
IMHO, your best best would be to get some blade servers, and create a cluster for the work you need. My machine supercedes most, but a $70k machine is something that you want a warranty and service staff on.

QFT, yea I believe a network cluster would be the best best for price to performance ratio. Having 16 processors in one system would only be required if you had like 10 of those as well....

 

[zephyrprime]

Try HP, egenera, tyan, linux networx, rackable systems, or hpc systems.

The nice thing about the opteron is that it is so much cheaper than the Itanium. Soooooo much cheaper. The Itanium is a little faster but only becuase it has that humongeous cache afforded by it's huge die size.

Unfortunately, the Opteron only supports up to 8 processors without extra glue or networking.

 

[Some1ne]

Does anyone have suggestions for a 16+ processor system, 2GB per processor, that runs Linux and is less than $70000.

No, the best I could do with what knowledge I have is an 8-way server with dual-core Opterons (which would give you 16 cores) and 4 GB of RAM per processor (so 2 GB per core). Looking around though, I can't even find a retail outlet that offers any 8-way mainboards. If you need 16 processors, you're firmly in the realm of specialized professional-level systems, although couldn't you do a cluster of lesser systems and get essentially the same results? Regardless, if you went the DIY route, a quick search implies your costs would be:

Case/PSU for system: $3000
Mainboard: at least $2000, if you can even get one at retail
CPU's: 8 Opteron 865's @ ~$2000 each = $16000
RAM: Assuming whatever board you get takes 184-pin registered ECC RAM at DDR400 speed, 32 1 GB modules @ $115 apeice = ~$4000
HDD's: No idea what HDD interface your hypothetical mainboard will support. Let's assume it comes with 8x SATA ports (which it very well may not), 8 250GB SATA150 drives w/ 16 MB cache (in RAID-5, for 1.75 TB of storage) @ $110 apeice = ~$1000

So I'd say you're looking at ~$25,000 to $30,000 for a DIY 16-core system, which means that with your budget you could get two of these if you wanted (almost 3 really)...although I sure wouldn't want to have to troubleshoot that damn thing, let alone a room with multiple iterations of it inside.

 

[BlacKJesuS]

canu loan me...like.....2k?

 

[fire400]

http://www.oracle.com/corporate/press/2...062105_bull_oracle_tpch_finalsite.html

This website offers contact information on finding a system that may suit your needs.

The following article is an example of one of their groundbreaking technologies, which features 16x CPU's on a Windows Server 2003 based system, which in turn, will not hesitate to run any Linux OS's.

You probably don't need any lecturing about how to maintain servers, but please know you that are not limited to a specific company or manufacturer when it comes to high end servers.

 

[CpuDesigner]

MikeK614
Your problem needs to be defined a bit *precisely* - parameters such as the level of inter-processor and/or inter-cluster communication scaling have a very significant impact on your performance and thus your system cost.

A farm of blade servers with 1GbE ports and high performance switches can work well if the inter-processor/cluster communication rate is relatively low compared to the computation rate. While I suspect that this is not the case for CFD problems if you scale to a larger number of processors, perhaps 16 processors may be sufficiently small to live with GbE inter processor/cluster communication (if this is the bandwidth you are living with the 7 processor itanium system).

On the other hand, a 32 processor system may require a 10Gbit interconnection switch fabric?which will cost more than the processors themselves.

Assuming you can easily host your software on another platform, you also need to consider the availability required (e.g., handling more frequent failures when less-expensive less-reliable equipment is utilized).

I am not surprised to see Opteron (or Power) systems performing significantly better than Itanium. I have been using x86 server farms (100s of processors) for processor/chip verification for several years (inter-cluster communication is low due to parallel simulations of independent data-sets).

Given you already have started with some benchmark information from various systems, I would work with your software vendor and data from your previous simulations to get a clear understanding on what performance you can achieve with your budget (number of processors, inter-processor/cluster communication, memory, etc requirements).

Select a system that meets your budget ? and then benchmark on that exact configuration as part of your acceptance requirements.

 

[pkme2]

http://www.cacr.caltech.edu/beowulf/tutorial/building.html

How about this?

http://www.pcmech.com/byos/index.htm

Building with multiple CPUs, you say?

Naturally, if you ever put one together, you're be the envy of the Maui R&T Park.

http://www.htdc.org/incubation/MRTC/default.asp

 

[Greenman]

You should prolly call AMD as well. I'd guess they have someone there who could give you a couple pointers on what to use and where to get it. It seems to me that if you can use "off the shelf" components you're way better off, even if you give up a little performance, because it looks like you can buy 4 or 5 "standard" systems for the same price as 1 16 cpu setup. Bear in mind I know almost nothing about hardware at this level, I'm just telling you how I'd approach the problem.
I'd also see if the software you run will work as a distributed process, like folding or seti, I know it's clutching at straws, but the only stupid question is the one you don't ask.

 

[MikeK614]

I will try to see where I can get with these smaller opteron systems. 8 dual core opterons like in one tower seems like a really good setup. If I do jobs that are less than 16 processors, I would just be using the Hypertransport between the processors within the box in which case I would not need to worry about a switch, Infiniband, Numalink etc. For some background info, the way this program works is by solving a system of 5 or more fluid dynamics equations simultaneously. The only way to do this is using an iterative numerical solution technique, and it normally takes thousands of iterations to "converge" on a correct solution. When you break the model up to be run on multiple processors, each processor simply gets its own set of the finite volume cells that make up the flow domain. Each processor will compute the same set of cells for every iteration, and it will communicate its results back to the host node which then figures out what the starting conditions throughout the model should be for each of the compute nodes, sends those out to all the various nodes, and the process repeats or "iterates" again. So the major source of bottleneck in a parallel problem like this is the time it takes all of the parallel nodes to communicate back to the host. It might take 30 seconds for the processor to actually finish its computation for that iteration, but if it takes another 10 seconds for the data to go back to the host before it can send out the info for the next iteration, the processors are just sitting around doing absolutely nothing, and obviously this is highly inefficient. So basically, real time high speed communication between the various processors is absolutely critical for good performance. It is best to avoid communication through any type of external network if at all possible.

 

[pkme2]

What are you planning to use the server for?

Why would you need a complex multiprocessor unit like you're suggesting, when you can economically do the same for less money. Is it practical or is it dreamworks?

I recommended the cluster because it was the most efficient method to go without having to expend a whole lot of dollars.

It's like trying to use a D9 Catepillar, when a Bobcat can do the job.

We have a Beowulf cluster at our college and I can't envision why you would spend $70,000. Since we are are a small entity, it only made sense (spelled 'cents') because monies are always limited, that we stick to something reasonable and affordable.

 

[MikeK614]

???

As I said, CFD.