Memory rank, and Nehalem

[hymy]

I used to hang out here, and keep up with computing technologies. So I know you guys are the best. Unfortunately for my current situation I grew up and became a microbiologist, and lost touch with computers.

I currently work with a group of chemical/biological engineers who are basically doing alot of computational biology. For the curious the link below is what we're using.

http://www.csb.ethz.ch/tools/efmtool

So the engineers are not necessarily very hardware savvy either.

We set out to build a dual nehalem xeon workstation, because the models we need to run are going to be large with output on the order of 100's of millions of efms, which means matlab matrices on the scope of 100,000,000 x 100 and bigger. We need tons of RAM and computational power for this.

But the IT guys who designed the system picked quad rank ram:
http://www.newegg.com/Product/...x?Item=N82E16820139024

I found out a little too late that nehalems only support 8 ranks per channel, which means we only get 48 gigs of ram with those dimms. So I need to send them back to newegg and eat the restocking fee.

So the question has been could we use 6 of the quad rank dimms (allowing us to only send back 12 instead of 18) and use 12 dual rank dimms. If so what would be the performance hit if any? Total Ram is 72 gigs.

I'm thinking about these dimms as an alternative:
http://www.newegg.com/Product/...x?Item=N82E16820220423

Motherboard/cpu:
http://www.newegg.com/Product/...x?Item=N82E16813131379
http://www.newegg.com/Product/...x?Item=N82E16819117184

 

[BTRY B 529th FA BN]

What are you talking about, ranks? What are ranks and per channel?

 

[Emulex]

with dual nehalem you need to be concerned about numa.

each cpu eats about 20% penalty when it must ask the other cpu for ram.

best speed (1333) is running 6 dimms per cpu (dual rank 4gb).

keep in mind this is 24gb per quad core for best performance. if objects that are located on cpu #1 need to access more than 24gb or any of the ram via the cpu #2 that has to be taken into consideration. numa aware o/s and poorly built numa applications can still do bad stuff if they eat up all of the ram causing thrashing.

12 x 4gb 1333 = optimal performance for now.
18 x (any gb) = 800mhz 25% performance hit

12 x 8gb 1333 (or 1066) = maximum 72gb but cost prohibitive for most folks.

that pretty much leaves most people with 48gb 1333 optimal with 24gb per numa node (cpu) unless you are super rich or something. 8gb sticks cost like 3-4x what 4gb sticks cost.

just keep in mind that your software should be considerate of the location of ram and processor affinity. vmware is an example of a very numa-aware application but even so it is possible to break the rules and run your machine less efficient if you desire to do so.

 

[Rubycon]

Your board is dual rank DIMM compatible. The second kit you specified will work.

I'd recommend Crucial for this board honestly.

 

[Emulex]

http://www.dell.com/downloads/...b/DDR3_white_paper.pdf

there is a good paper to read. it is outdated or dell sucks. hp can run 8gb (12) or 4gb (12) at full 1333mhz bus speed. check your motherboard manufacturer for bios update.. (originally the intel 5500/5520 chipset did have these limits ibm and hp seemed to have gotten around it with a bios patch)

here's a better pdf to read:
ftp://ftp.software.ibm.com/com...5usen/XSW03025USEN.PDF

 

[hymy]

Well unfortunately we ain't wealthy so no 8gig dimms. 48 gigs total is not enough RAM, so we have reconciled ourselves to the reality of 800 mhz. The software we are going to run is JAVA based, so the amount of ram we can allocate to the JAVA heap is orders of magnitude more important than how fast that RAM is. Also we need enough ram to hold the output and manipulate it in Matlab, again the amount is more important. This isn't going to be used as a server of any kind.

So the question I'm asking is can I keep 6 of the 4 rank ram, and buy 12 dual rank dimms. This would limit the restocking fee, a little. It seems that it should work, that would be 8 ranks per channel and it would be balanced I believe. But I like to be sure given the amount of money in play.

Rubycon, do you know where I can get some reasonably price crucial 4gig 2 rank dimms.

BTRY B 529th FA BN, youl'll need to google it, I can't really explain it well enough that it would make sense.

 

[VirtualLarry]

Originally posted by: BTRY B 529th FA BN
What are you talking about, ranks? What are ranks and per channel?

RAS banks. Basically, the number of RAS strobe lines per DIMM. "Rank" is sort of an abbreviation for RAS Bank. It was Intel who coined the term, I believe.

 

[Emulex]

seems like it should work - always check your manual of course. you do lose some interleaving with unequal dimms

 

[hymy]

Originally posted by: Emulex
seems like it should work - always check your manual of course. you do lose some interleaving with unequal dimms

So in order to interleave, they all need to be the same, not just match within the "subchannel"? I was hoping for better news. Thanks for clearing it up.

:edited after I got what you were saying:

 

[hymy]

@Emulex,

Thanks for the literature, I went through it. I also did some googling. It haven't seen any definitive evidence that it will work. But what I've read so far is that I should be able to put 1 quad and 2 doubles per channel.

Does anyone know for a fact without speculating, regarding this matter?

 

[tcsenter]

Originally posted by: hymy
Does anyone know for a fact without speculating, regarding this matter?

Not according to Intel (see: Table 3-2):

http://www.intel.com/Assets/PDF/datasheet/321322.pdf

You could populate one QR DIMM + one DR DIMM per channel (2DPC config), leaving the third DIMM on each channel empty. You can use the excellent documentation that Intel provides for its server boards as a reference to fill-in all the black holes that will be present in ASUS' documentation.

And I would recommend Intel, TYAN, or SuperMicro over ASUS for server boards. You'll get way better BIOS refinement from Intel than ASUS, not to mention Intel's documentation, driver, utility, and software support is stellar. e.g.

ASUS KFSN5-D = 3 BIOS updates
ASUS L1N64-SLI WS = 4 BIOS updates
ASUS L1N64-SLI WS/B = 3 BIOS updates
ASUS Z7S WS = 3 BIOS updates

The Z7S WS and L1N64-SLI WS were nightmarish for many users due to BIOS immaturity.

Originally posted by: hymy
The software we are going to run is JAVA based...

How awful! I never knew scientific computing and JAVA could be uttered in the same sentence. At least, I hoped it would never be.

 

[pcslookout]

Originally posted by: tcsenter

Originally posted by: hymy
Does anyone know for a fact without speculating, regarding this matter?

Not according to Intel (see: Table 3-2):

http://www.intel.com/Assets/PDF/datasheet/321322.pdf

You could populate one QR DIMM + one DR DIMM per channel (2DPC config), leaving the third DIMM on each channel empty. You can use the excellent documentation that Intel provides for its server boards as a reference to fill-in all the black holes that will be present in ASUS' documentation.

And I would recommend Intel, TYAN, or SuperMicro over ASUS for server boards. You'll get way better BIOS refinement from Intel than ASUS, not to mention Intel's documentation, driver, utility, and software support is stellar. e.g.

ASUS KFSN5-D = 3 BIOS updates
ASUS L1N64-SLI WS = 4 BIOS updates
ASUS L1N64-SLI WS/B = 3 BIOS updates
ASUS Z7S WS = 3 BIOS updates

The Z7S WS and L1N64-SLI WS were nightmarish for many users due to BIOS immaturity.

Originally posted by: hymy
The software we are going to run is JAVA based...

How awful! I never knew scientific computing and JAVA could be uttered in the same sentence. At least, I hoped it would never be.

I hate java as well. If I didn't need to install it on my pc I wouldn't. Unfortunately I need it.

Wish there was a alternative for java like there is for a pdf opener.

 

[hymy]

@tcsenter

You are right sir, in fact in section 3.2.1 it states that quad rank dimms cannot be used in 3 dimm configurations.

Unfortunately we have all the parts, and can't ditch the motherboard. That Asus board was DOA though, and I'm waiting on the replacement.

Well in regard to the java. It's developed by a guy in switzerland, and apparently when he started the development he used java. It was for is Ph. D. I believe, so I assume he used what he knew. I asked him about porting it and he said it had just gotten to large and he didn't have time do it. It's unfortunate, but the algorithm that he developed is something like 100x faster than anything else we could use. Of course it is open source, so it shouldn't be long before it get's ported.

 

[hymy]

There are things you can do with java that help. I've found that the java switch -XX:+UseAdaptiveSizePolicy goes along towards making java's memory management bearable. It's supposed to be default when using parallel garbage collectors, but I've never seen it used unless the switch is actually set.

 

[Emulex]

just compile the java

 

[hymy]

I don't follow, can you elaborate

 

[taltamir]

call newegg support, explain the situation, ask if they will wave the restocking fee IF you buy the new ram from them. (aka, wave restocking fee, but give you store credit instead of cash back).

also, what is "rank" in terms of RAM? do you mean 4GB dimms instead of 8GB?
there is a programming rank:

In computer programming, rank with no further specifications is usually a synonym for (or refers to) "number of dimensions"; thus, for instance, a bi-dimensional array has rank two, a three-dimensional array has rank three and so on. Strictly, no formal definition can be provided which applies to every programming language, since each of them has its own concepts, semantics and terminology; the term may not even be applicable or, to the contrary, applied with a very specific meaning in the context of a given language.

But that has nothing to do with ram

 

[tcsenter]

To better understand what is meant by a DIMM rank, it may be helpful to remember that the PC memory bus is 64-bits wide. Since there are many millions of bits across many different chips on a module, the module (and chips) must be wired-up in such a way so that only a certain combination of chips will be 'actively' connected to the memory controller at any given time in order to create this 64-bit wide area (a.k.a. row).

The term DIMM side, or "rank" as it is now called by JEDEC, refers to this particular combination of chips that will create a full 64-bit read or write over the memory bus at any given time. Each DIMM can have one or two ranks (modules designed for servers can have four ranks). Each 64-bit rank is connected to a circuit in the memory controller called a Chip Select (CS). The number of independent CS signals supported by the memory controller defines how many ranks are supported. Current memory controllers typically have a total of eight CS signals, supporting a total of eight 64-bit ranks (server chipsets are coming with more).

A single sided module has one 64-bit rank. A double sided module has two 64-bit ranks. Logical DIMM side and rank are the same thing. It so happens that a single rank module often has chips on one physical side of the module, while double rank modules often have chips on both physical sides of the module. However, this is simply a coincidence resulting from practical design issues limiting the ways that memory modules and memory slots can optimally be layed-out and implemented on the system board. There are modules with chips on both physical sides but have just one 64-bit rank (4-bit wide or x4 DRAM). It is possible to construct a module with chips on only one physical side but actually has two 64-bit ranks (though I don't think any are produced).

 

[hymy]

@tcsenter I'm guessing you left a "not" out of "Logical DIMM side and rank are the same thing". That is perhaps the best explanation of rank I've read. Though I can't say that I yet understand it all that well. I do however feel more confident that I finally ordered the right stuff. It's amazing how confusing it is, with different terminologies, and some companies not even specifying the rank. I Ended up requesting an RMA today for all the quad rank stuff, and ordering 12 2 rank patriot (which is all that newegg had). That way we stay below the $5000 dollar mark (which keeps another level of university bureaucracy off our backs). I going to watch what happens with the memory market over the next year while we evaluate whether the 48 will get the job done.

@taltimir I did put that very request regarding the restocking fee in RMA comments field. I didn't really consider that there was any real chance they would cut us a break, but since you bring it up I'll contact them in the morning. If they cut us a break, I can probably get the remaining 6 dimms. But Now I'm wondering if 8gig dimms will be feasible in a years time.

 

[Emulex]

why would you order from university to newegg?

for the most part university(or any large customer) get:
1. free shipping (both ways, including if you rma)
2. mad discounts
3. someone who knows their isht to select a proper product FOR YOU on call 9-5

 

[taltamir]

you were supposed to call them BEFORE placing an RMA.
tc... thanks for clarifying. I didn't realise JEDEC started calling dimm sides ranks. and apperantly it is mentioned in wikipedia's dimm article (not the wikipedia is a good source): http://en.wikipedia.org/wiki/Dimm

Consumer DIMM vendors have recently begun to distinguish between single and dual ranked DIMMs. JEDEC decided that the terms "dual-sided," "double-sided," or "dual-banked" were not correct when applied to registered DIMMs.

hymy, why do you actually need to return it again? you seem to be asking if you could mix different ranked ram but you admit to not knowing for sure. Yet you already initiated the RMA while accusing the guy who bought the ram of not knowing what he was doing.
Are you sure you want to be the one to take responsibility for such decisions?

Also, am I correct in understanding that you want to replace 2 ranked 4GB dimms with 4 ranked 4GB dimms? why?

 

[Emulex]

i think he wants to ditch the 4rank for all 2rank.

dual rank interleaves the best on the intel 5500/5520 chipset. you get some advantage of having the exact same ram in the channels to boot. that might help offset the 25%+ loss of bandwidth by having the chipset clock down to 800mhz from 1333

 

[hymy]

@Emulex, Regarding university purchasing: I hear ya, and I'm right there with ya. But I could go on and on about why it isn't simple, and how it ought to be. It's a system that needs to be seen to believe. I would rather not get into why things were done the way they were. But I assure this was the best that could be done given that.

@taltimir just to catch you up (and a summary for everyone else) .

*Newegg has this wonderful fully online RMA process, that doesn't (would I really want to know why "doe snot" is a spelling suggestion in firefox) require a phone.

*There was no accusing of anyone, and the RMA was initiated after a group discussion, where many people smacked their forehead. Because nobody involved knew anything about memory rank and nehalems. I figured this out purely by chance, because I'm overly curious about little details. It is a purely understandable situation, just look at how many people including yourself were unaware of this little technical detail.

* The problem was that when you have 18 dimm slots on a dual nehalem motherboard and 18 quad rank dimms in your sweaty fists, you can only put 12 of those dimms on that motherboard. I could have just sent 6 back, but that would mean future memory expansion would require that the remaining 12 be discarded. Because the only way to put 18 dimms on there is to buy dual rank ram. Since I was the only one that vaguely understood the problem at the time, and I didn't know enough to decide what to do, I came here. The information that I had dug up implied that you could get 8 ranks of dimms per channel, but I wasn't going to make a decision based on implication. The hope was that we could put 6 quad rank dimms and 12 dual rank dimms on the board because that's 8 ranks per channel.

*The answer to the question (thanks, tscenter) was that, no it won't work because the presence of any quad rank dimms on a channel limits you to only 2 dims per channel. So the quad rank had to go back. I also assume that the limit is actually six ranks per channel.

 

[Emulex]

sometimes paying a few more bucks for the right answer - the first time is worth it yeah i know how university purchasing works very intimately

universities pay so little its not funny. the ram you bought (assuming you paid screen price) cost more than your university to purchase say branded hp or ibm ram madd discounts they get..

good luck - let us know how that java app works out.

 

[hymy]

Do you really believe I'm that ignorant, EMULEX?

I do envy your liberal university. But, It's not like that here. It's a Dell only world here. The only way you can buy a computer easily (huge sarcasm) is to walk over to to IT and buy a DELL. If your lucky they'll deliver that Dell in a few months. If you don't want a DELL, you must fill out paperwork and turn it in, and listen to a lecture about how you really want a DELL instead. They will sign off on it eventually though. If your needs are such that a pre-fab computer doesn't economically fit your needs (in our case IT would have preferred we bought a $10,000 DELL server, we of course don't need that or the hassle of getting a large purchase approved). The building of computers from parts by departments is not something that is supported. IT doesn't not buy parts other than what they need for repair and upgrade of existing DELLS, which they buy from DELL. They therefore don't have any purchasing agreements with anyone else, other than I think a smaller arrangment with HP. So no "madd discounts". It's not a perfect world for everyone. And as a matter of fact the paper work was filed, and they didn't catch the error either.