Best way to plumb 7 GPUs

[nrobidoux]

Took me a little while to get to this point, (1.7 years) long story that I'll skip. But the day is almost here to press the "on" button and run my computer to it's full potential.

I have 7 HIS 7950 IceQ w/ Boost that I water cooled. I was thinking about plumbing them serially but I think that would create a problem putting 7 cards in simultaneously.... each one rotating this way or that if I couldn't manage to handle them perfectly.

So I hooked them up in parallel. I have three pumps. I hope that is enough flow and pressure to overcome the division by 7 and supply a backup pump.

I have a lot of cleaning to do before I even have enough space for this monster build but I was wondering your thoughts so far. And what prompted this post is does it matter if the inlet / outlet are on the same or opposite sides of the GPU stack? I have them on opposite because I believed in theory that putting them on the same side would allow the 1st GPU to have the least "resistance" for the water.

 

[Micrornd]

Very interesting experiment.
I can't help with the water cooling, but do have questions.

What motherboard will you be using?
And what is up with the cards?
As bridged, you have quadfire and 3 independent cards.
AFAIK quadfire is the max for AMD.
How will you be using them?
Is this for a flight sim application?

 

[aigomorla]

there are 2 types of parallel setups for gpus.

One is the standard both inlet and outlet at top, and bridged down below.
This setup works great when your only using 2 gpu's tops.
When using 3+ gpu's however, the middle card gets bypassed sometimes, causing temps to be very uneven. Water likes to flow in the least restrictive path, so if by some fluke one of your gpu's doesn't have equal flow, it will partially bypass the gpu and hence starve it of fresh water.

The next parallel setup is something we call an F parallel.
That involves the inlet at top card, outlet on the opposite side bottom card.
Think of 2 x F's coming together where the second F is inverted.
This setup works a bit better but suffers from the same rule above.
Parallel setups will travel down the path with the least amount of resistance.

Lastly is the serial method.
The limiting factor in this will be flow.
Having 7 GPU's will most definitely form a heat gradient inside your gpu loop.
Which translates to the top cards being colder then the bottom cards because the water which travels though it will increase in temps.. think of a tankless water heater.

The best and easiest way to mitigate this is by increasing flow in general.
General Rule of thermodynamics states water traveling @ 1.5gpm will effectively have a holding potential of 300-350W of heat b4 it goes up 1 C.

Assumption that each gpu you have will put out about 190W for gaming scenarios, however folding scenarios i would probably give about 30-50W extra for a total of 220-230W.
http://www.anandtech.com/show/5476/amd-radeon-7950-review

So lets take the max @ 230 x 7 = 1610W of theoretical heat value when all 7 cards are churning @ 100%

1610W / 350w => 4.6

That means as water enters the top card and exits the bottom card the temp of the water will increase a whooping 4.6C in a serial setup.

Best way to mitigate this is again via increased flow.


How would i do this?
i would most likely serial it, and put no less then 2 dedicated pumps on its own loop.
You said you had 3 pumps, so that would probably be your best choice.
Each pump will double head pressure but only give u 1/3rd more flow.
So by having 3 pumps you will effectively triple the head pressure @ an increase of 2/3rds more flow.

I would keep the loop as short as possible, use 2 very large rads with very low resistance in flow rate, and experiment from there.

However there isn't much you can do unless you want to use risers cards and split the gpu's apart.

 

[nrobidoux]

What motherboard will you be using?

ASUS Z9PE-D8 WS... older version. Bought it... January-ish 2015. Wow. I say older version lol. Anywho I say that because it looks a little different than before and believe supports additional... Oh wait. They changed the socket. It's now LGA 2011 v3, mine is LGA 2011.

As bridged, you have quadfire and 3 independent cards.
AFAIK quadfire is the max for AMD.

Yes. I learned things the hard way. It wasn't until a few months after I attempted to get this thing working that I learned the Linux and Windows limitations were driver related.... although that doesn't really relate to your comment above.

But had I known that initially I would've got 4 7990's and Xfired 2.

And what is up with the cards?
How will you be using them?
Is this for a flight sim application?

I continued to pour money into this project for the sole reason that no computer will defeat me. Built all my previous "normal" rigs with no issues. This rig, issues every step. I felt like such a noob.

Literally even the links between GPUs were too big!!!! 11mm was the smallest I could find. 2mm too big according to multiple caliper measurements. What's 2mm? Doable with 2 GPUs in adjacent slots not 7. (2mm * 7 = 14mm == ~0.5"). I had them machined down 2+mm.

Initially for mining but GPU mining is dead. Wouldn't be defeated, now folding.... periodically, so I don't upset anyone who pays the electric bill.

I'm all for leaving it on 24/7 churning on something... but only if I can monetize it somehow. Every time I put some thought into how I don't come up with much.

... Also play Elite Dangerous. Kinda overkill since I can max settings with 1 7950.

 

[MrTeal]

If you're worried about the cards rotating, you could stick a rod through one of the holes to keep them all aligned. If you pick the one next to the power connectors it would be easy enough to remove, though you could just leave it in there.

What pumps do you have, and what are the specs on them?

 

[nrobidoux]

...however folding scenarios i would probably give about 30-50W extra for a total of 220-230W.

Pretty much agrees with what I saw enabling/disabling individual GPUs while folding.

...230 x 7 = 1610W of theoretical heat value when all 7 cards are churning @ 100%

Mining = 100%. Folding I don't think quite gets there but that is okay because I want to stay within the sweet spot of my PSUs and not pop any breakers.

...the water will increase a whooping 4.6C in a serial setup.

All things considered I think that is quite good.

Best way to mitigate this is again via increased flow.
How would i do this?
i would most likely serial it, and put no less then 2 dedicated pumps on its own loop.

I will probably put a ball of some sort in the connectors to block off alternating sides. Handling (I need to weigh them to get a real number) 25+ lbs of PCB and metal with 6 articulating points is not fun LOL. I only did it with 3 and thought it was for the birds.

I would keep the loop as short as possible, use 2 very large rads with very low resistance in flow rate, and experiment from there.

2 MORA 420s 9x140mm fans on each. Went for quiet so only 700 rpms.

 

[nrobidoux]

If you're worried about the cards rotating, you could stick a rod through one of the holes to keep them all aligned. If you pick the one next to the power connectors it would be easy enough to remove, though you could just leave it in there.

What pumps do you have, and what are the specs on them?

See if I can find a brazing rod somewhere... hopefully...

Pumps?
Alphacool HF D5 Top - Black Acetal w/ Swiftech MCP655-PWM Pump Installed

 

[nrobidoux]

How will you be using them?

With a few assumptions on the interconnection between neurons I believe I can almost simulate a brain of a fruitfly in realtime!!!

That's actually something I use to tell my coworkers lol.

Possible in theory maybe but I doubt I'll be the one to do it.