OT: High end linux clusters

[MereMortal]

Does anyone have any experience with high end linux clusters? I am in charge of getting quotes for a turnkey system at work. If anyone has any tales of dual Intel Xeon vs. dual Athlon MP, or Gigabit interconnect vs. Myrinet vs. Dolphin I would appreciate your input.

The cluster will be used to do adaptive mesh MHD simulations, so unfortunately it needs as fast an interconnect as fiscally possible. Myrinet may be too expensive, however.

We are leaning toward a Xeon solution at the moment, due to heat/failure concerns with Athlon systems, and the fact the Xeons have a larger L2 cache and thus get higher MFLOPs for larger sized arrays.


Anyway, just thought I would take a shot while I'm waiting on vendor quotes.

 

[ProviaFan]

Sorry I can't help much, but thought I'd give you a bump so your thread doesn't get buried under all the MTR threads that are going on right now.

Hopefully someone more knowledgable will see this and come in and enlighten you.

 

[daweasel]

ok are you doing this just to make me jealous or are you really getting something

 

[daweasel]

any chance of the person/lab waiting for the opterons?

I've heard of several big purchases of hammer arrays already.

 

[Lord Demios]

About three years ago when I was doing this stuff for a University, these were the things that we found. Not sure how much has changed, but it's food for thought.

Dual CPU had a tendency to fill up your system bus, and you start having a bottle neck.before you can use both CPU's to their max.

PCI bus truely has trouble handling the I/O of a gigabit ethernet card.

How many nodes are you looking at. Some of the clusters I got to work on were setup so that every machine had a 5 port 100 mbit network card, and would connect to 5 other nodes. It worked well with up to 16 nodes, but as I said, a lot of what I have worked with has changed since my day.

www.beowulf.org is a good place to start looking at clusters. I would also recommend looking at this page for www.chpc.utah.edu.

LD

 

[MereMortal]

Thanks for the info. I've been to all the major beowulf related sites. Now I'm just looking for the pitfalls that don't get mentioned anywhere. The cluster we will build will most likely be a 16 node, 32 processor system. Just got a quote! Myrinet may be possible.

Weaz, I'd love to wait for opterons. Unfortunately, we will get funding through a unnamed goverment ageny that is trying to spend its alloted budget. So the money must be spent soon or it will disappear. And to get the funding we need realistic quotes.

 

[networkman]

Xeons rock! I still have my dual 400s with their 2meg L2 caches.

 

[Shuxclams]

Microway has out of the box configurations.














SHUX

 

[daweasel]

MM know how that goes, got to spend the budget so they get the same allotment the next year. Yeah for the stupid government.

Good luck on the search and send pic's when the computers arrive, so I can drool.

 

[lobadobadingdong]

pci bus does limit the ethernet gigabit connection, but there are a few 66Mhz gigabit cards out there, or you may think about going w/gigabit fibre channel setup, is much, much more reliable and gets rid of any/all speed fluctuations do to the crosstalk enherent to wired data transfers. Gb fibre nics are also easier to find in the 66Mhz pci cards than the ethernet. Fibre channel is what Dell uses for their clustered rigs, so that might be a good indication of the speeds capable of with that setup.