My quad-GPU F@H crunching rig

[VirtualLarry]

I know that a crossfire mobo cannot run SLI. However, can you install two Nvidia graphics cards, (not in SLI mode), and simply use them as a dual-card solution to run F@H on both GPUs?
Edit: Apparently the answer is yes.

This thread will be used for recording my quad-GPU cruncher adventures.

I ordered and recieved a K9A2 Plat. mobo, which is an AM2+ mobo with FOUR, yes, FOUR PCI-E 2.0 x8 slots on it. I also have acquired four 9600GSO video cards. I am still waiting on acquiring some 75ohm resistors to build some VGA dummy loads with.

Edit: Completed build, now with pictures!
Outside
Inside
R.I.P. Videocard

 

[Denithor]

Look at my sig.

Definitely doable, I have 3 GS/GSO cards crunching right now on an X38 board. And a fourth added next time I visit my mother & retrieve my other GS.

 

[VirtualLarry]

I'd like to expand my question. What is the cheapest rig for multi-slot PCI-E crunching. Two, three, four slots, etc.? Was pricing out those P5E64 WS boards, they seem too expensive for me. AMD or Intel solution, either one works for me. (I have acquired two EVGA 9600GSOs, and have two more ASUS 9600GSOs enroute.) Willing to consider a pair of two-slot systems, or possibly a single multi-slot system. Want to build a F@H rig (of course).


Text
Text

MSI K9A2 open box $115

If I got one of the four-slot AMD mobos, could I run it off of a BE-2400 CPU, or does having four cards crunching REQUIRE a four-core CPU to keep it busy? I'm not sure I want to splash out for a 65nm phenom, what with the Deneb chips coming out so soon.

Edit: Would running Vista cut down on the CPU requirements, since I hear the "one CPU core per GPU" rule is mainly applied to XP, and that Vista has lower CPU requirements. If so, would Vista 32-bit or Vista 64-bit be better? Is F@H (multi-GPU) easier to run on one rather than the other?

AM2+ quad-core 1.8Ghz 65W $139

4-slot PCI-E mobo - $115
quad-core AM2+ CPU - $140
4x 9600GSO - $240
2x2GB DDR2-800 - $55
Corsair 750TX PSU - $90

$640, no case, no HD, no DVD burner. I have an Antec 300 that could be used, and I could probably scrounge a HD somewhere.

Assuming PPD of around 5K per card, that's 20K per day. (Wow, even with a rig like that, things like Mark's 1M point scores still look huge.)

 

[Markfw]

MSI P7N Diamond LGA 775 NVIDIA nForce 780i SLI ATX on newegg (open box) is $172, and allows for 4 cards ! I priced out that, with 4 9800GT's , A Q6600 and Vista (I already have all the other components) and it was $942. But 22-24k PPD out of one box would be sweet !

Edit, that am2+ motherboard looks cool for $115. A quad in there@stock should be enough HP to run the 4 cards..

 

[Denithor]

With Vista you don't need a quad to push 4 cards. My farm averages 15% use across the two cores while crunching on three cards.

 

[VirtualLarry]

Originally posted by: Denithor
With Vista you don't need a quad to push 4 cards. My farm averages 15% use across the two cores while crunching on three cards.

That's good news. I already have the BE-2400, which I could overclock. That would save me $140.

This could be a problem: link
"If you run Windows Vista, you need to have each GPU connected to a monitor (a KVM can also work but not all models of KVM will provide a signal when it's not switched as current monitor) or a VGA dummy plug. "
How important is that? I don't have anything to plug into the four GPUs, other than the primary monitor. Do I need to order some dummy plugs or something?

Found this
and this

Edit: Ok, I ordered a brand-new K9A2 Platinum mobo off of NewEgg. Decided not to go for the open-box one, mainly for the warranty. Will be using my BE-2400 for now.

Went to RS to pick up some connectors and resistors, they were mostly out of all of their resistors, only one drawer had 68Ohm ones, but they were all gone. Does anyone know of any good online sites to order some 75Ohm resistors off of?

Found this , 75Ohm, 10 for $0.55. Not sure what shipping is.

 

[Denithor]

Yes, you will have to have the DVI 'terminated' somehow for this to work in Vista. I fortunately have a 2709W with enough inputs to satisfy this need (2 DVI, 2 HDMI, VGA). I did find that I could make it work with just two inputs, basically plug in one card to an input on the monitor and activate that client, then unplug and repeat for the other cards/clients.

The resistor idea is cool, I may have to swing by my local RS tomorrow to see if they have what's needed from those threads. Every time I reboot this computer I'm going to have to go back through the startup for each client, it will be much easier if each card is semi-permanently terminated correctly.

 

[VirtualLarry]

So, with the news that ForceWare 180.70 supposedly allows GPU folding with much less CPU time in XP (comparable to Vista?), I decided to build my rig.

Antec 300
EarthWatts 650W (may be replaced with a Corsair TX750 in the future, if I need the extra juice)
4 x 9600GSO (Two Asus "Top" cards, factory overclocked, one EVGA superclocked, and one EVGA regular)
MSI K9A2 Platinum (with 4x PCI-E x16 slots)
AMD BE-2400 dual-core
AMD 4-heatpipe cooler
WD 6400AAKS
Lite-On 20x IDE DVD burner
a couple additional Antec DBB 120mm fans for the front of the Antec 300

As soon as my Vista x64 DVD finished burning, I'm going to try and hook it up.

To start with, I'm only running two 9600GSOs, as the PSU only has two 6-pin PCI-E connectors. I do have some PCI-E 6-pin splitters on order from NewEgg though.

 

[VirtualLarry]

Ok, I got it running. Strange problem, Windows Update doesn't work, after a fresh install and chipset drivers/ethernet drivers/sound drivers/video drivers. Don't know what's going on there.

I created shortcuts for up to four GPUs, and I configured the first two and got them running.

Showing 2812 PPD for project 5750 on first GPU, and 2830 PPD for project 5756.
CPU usage is totally non-existant, I get spikes of 1% every now and then, but generally it stays at 0% all the time. CPU is a dual-core AMD BE-2400 2.3Ghz, CoolNQuiet disabled.

Is this a lower than normal PPD score for 9600GSO cards running at 550/800? Or is this just because more recent projects seem to be causing lower PPD for NV users?

Edit: Checking on the machine in the morning, it shows GPU1 doing project 5015, showing 4459 PPD. That's a lot closer to what I expected.

Edit: Windows Update works now. Must have been a glitch with MS's web site.
Edit: According to RivaTuner 2.20, clocks are actually 600/900.

 

[theAnimal]

That PSU should have plenty of juice for 4 9600GSO.

Some of the newer WU are giving lower PPD.

 

[Insidious]

Originally posted by: VirtualLarry
.
.
.
Edit: Checking on the machine in the morning, it shows GPU1 doing project 5015, showing 4459 PPD. That's a lot closer to what I expected.

Those sound like pretty good numbers.

-Sid

 

[VirtualLarry]

Well, I got in my two PCI-E 6-pin power splitters from NewEgg today. I installed the other two graphics cards into the rig. Top two graphics cards are Asus "top" 9600GSO, third card is EVGA 9600GSO, and fourth card is EVGA 9600GSO superclocked.

I don't know what I'm going to do. Temps are absolutely insane. The Asus cards, even though the heatsink makes the card take up two slots, doesn't seem to be very effective. It idles at 60C, and before, with only the two Asus cards in the first and third slots, they were running at 80C and 70C. Now, sandwiched all togther, I watched as the temps for the top card went from 98C to 103C!!! Not good. The heatsink is too hot to touch. The EVGA cards with the longer cooler, possibly the reference cooler, are only 60C under load. Then again, there is some space next to their fans, since they are only single-slot.

So I'm not sure what to do. I'm going to get a 120mm side fan to mount on the Antec 300, but I don't think that will be enough.

I suppose I should make an open offer to any owners of single-slot EVGA 9600GSO GPUs, if you want to swap with an Asus "top" card, be my guest. Then again, possibly the temps are because the Asus cards run at 600/1700/900, and the EVGA stock card is 550/1350/800. But it does seem that the Asus's cooling method is less efficient. So if you're not running multi-GPU, and want a free slight upgrade, contact me. I want EVGA single-slot cards. (Even at stock speeds)

 

[Insidious]

It sounds like you need to use RivaTuner to set the fans to direct control @ 100% on each of the video cards.

The stock fan control is useless and seem to think we like burning our fingers on heat sinks.

-Sid

Get RivaTuner Here

note: on my PCs, RivaTuner doesn't get all the settings saved for multiple video cards. Anytime you have to re-start your computer, re-check your settings in RivaTuner and adjust back to the 100% fans on each if necessary.

 

[VirtualLarry]

I have RivaTuner 2.2.0 installed. No Fan speed control option shows up for those two Asus cards. It does show up for the EVGA cards.

I installed an Antec TriCool 120mm fan on the side panel on high, and now my temps are reaching 91.5C, instead of 103C and rising.

I don't know what the temperature spec is on these cards, but for some reason I thought it was 80C.

Edit: Now the top two GPUs are doing different workunits, and it's showing 95-96C.

 

[Insidious]

You probably already tried this, but did you look at both the low level, and the driver tweak tabs?

Another possibility might be to try using different drivers. I know my RivaTuner behavior changed when I changed from 178.24 to 180.60 driver. Which ever one you have installed, maybe try the other one.

I have seen posts that say a NVidia GPU is still OK at 100C. That's too aggressive for me. I'd hate to see them above 85C or so if I could help it.

-Sid

edit: I don't know how it works, but you might try the tweaking tool from NVidia. NVidia System Tools

 

[VirtualLarry]

Well, I had to shut 'er down. The cables unwound a bit and started rubbing against the 120mm front fans inside. I need to pick up some 120mm fan grills to put inside to prevent that. Otherwise, I'm going to have to take the machine apart every other day to move the cables around.

Edit: I gave in, and took the machine apart again. Don't want my WUs to get stale on me.

 

[Markfw]

BTW, all 8 of my 8800 GT's run less than 70c on 65% fan. I used the nvidia software to set them at that all the time (the performance tool). They are all dual-slot solutions though.

 

[petrusbroder]

Very interesting thread: I am considering to get myself a folding rig ... one mobo-4-core-cpu-600w-psu-RAM-HDD-DVD-rom and 4 GPUs - that should give some 14 - 16K in folding? Or am I calculating wrong?

 

[biodoc]

With 8800 GTs you are in the right range although the high end could be 20K with overclocking and "friendly WUs". I doubt if a 600W power supply would be enough though. You need 4x6 pin 12V (24V?) for the cards and you need a case that has great cooling too. Consider a high end Antec case. I have an Antec 900: 4x120s and 1x 200 mm on the top. Lots of blue leds too!:evil:

Have fun Peter!

 

[Markfw]

One Q6600@3.5 and 2 8800 GT's@ 740 under winxp = 12.000 PPD., 1 SMP and 2 GPU.

 

[VirtualLarry]

Well, at a chilly 58F ambient temp, my top video card temp is 84C.

Edit: Can someone using multiple NV GPUs test something for me? Download EVGA PrecisionTool here , and see if it offers fanspeed control for multiple GPUs, individually.

Edit: PrecisionTool works for more than just EVGA cards, btw.

Edit: My PPD has dropped to 10,473. Darn slow WUs.

 

[Insidious]

Sorry VL, my cards are ASUS

try the 178.24 drivers for your video cards. I've read several posts where the 180 series ferks up the fan control. (some models only)

-Sid

 

[VirtualLarry]

Well, my quad-GPU rig just randomly restarted for the first time, a few minutes ago.

My other C2D rig that was stable running tests for 24hrs (any stability test, memtest86+, etc.), and had uptimes at times of up to a month, also had random restart problems.

Both computers were plugged into the same UPS, I think it's the UPS. I had to replace one of them because it was making my computer restart once a day. Once I plugged the computer directly into the wall, the restarts stopped. So I replaced the UPS. But it still does it now, just once in a great while. Very annoying.

Edit: It seems as though I cooked my video card. I was wondering how long it would hold out at 96C. I get no display through it.

 

[VirtualLarry]

Bump. Now with pictures. (See first post)

Edit: Compare my rig with Fastra.
Note their GPU temps, 100C!!!

 

[Insidious]

three days ago, I noticed one of my computers had re-started itself.... ??

Then yesterday it did it again....

I looked at my temperatures once it was running again and WOAH.... my GTX+ was HOT (nearly 90C)

On a closer look, I found my case exhaust fan was running really slowly. I replaced it... temps are back to normal and no more re-boots.

-Sid