Fried multiple GPU's, running out of clues.

[NedStar]

This is a problem I've been having for the last few months and It's driving me crazy. I've debugged everything I could think of but maybe you have a few ideas.

Here's the story.

I've fried a grand total of 6 GPU's in the last year. Five in the last few months.
They fry at random but all fry the same. PC shuts down and refuses to boot without removing the GPU. There are no pre warnings, it just goes POOF.
GPU's have been rma'd and replaced thus far, also tested in another pc and deemed broken.

There is no OC in place nor do temps exceed 70 degrees in case of 6x series and below 60 degrees for the 7x series.

In the case of crossfire, the top card with the cables would be the first to fry. Upon switching cables the other card would ultimately fry. Time-frame varies.

3 x 6970's fried.
2 x 7970's fried
2 x 7950's fried.

The power outlet is grounded and other pc's run on it with no problems. There is also a surge protector in place along with an UPS with AVR covering all the monitors, pc and router.

What I've done.

Countless stress testing and other checks, I'd mention them but seeing as a lot of parts are replaced I'll only mention the parts. Parts not mentioned are not tested in any way.

PSU replaced thrice, coolermaster 850 backup (fried 1 GPU) and two corsair 750AX (fried 6 GPU's).
Entire case and all the hardware has been replaced. All but the monitor and mouse/keyboard have been swapped.
With and without UPS with AVR between power socket and PC.

What has not yet been replaced.


Monitors. (testing this now)

Since I've fried another one today in the exact same manner after having replaced so many parts. I've decided to remove the soundcard and the removal of the HDMI, DVI cables from the last good GPU.

I really have no clue what else to do and I would greatly appreciate any help you might have on this. I'm also not entirely sure how plausible it would be for me to even throw it at the soundcard or hdmi, dvi cables.

Many thanks in advance.

 

[Synthet1X]

Are you plugged into a surge protector? You should have gone with NVIDIA .

 

[AnonymouseUser]

I would suspect dirty power. A Back UPS with AVR would be a wise investment even if it didn't solve this problem.

 

[NedStar]

I would suspect dirty power. A Back UPS with AVR would be a wise investment even if it didn't solve this problem.

I'm sorry but what is an AVR? I did a quick google but couldn't find anything useful.

Regarding dirty power, that's what crossed my mind as well. However there is a surge protector in place for all the pc devices. The only thing that is not grounded or hooked to a surge protector is the HDMI cable which is coming from another room that doesn't offer grounding.

Also what symptoms would one expect with dirty power? All the devices in the house are working fine with the exception of GPU's frying like crazy; only in this pc though, all the other pc's in the house are fine. One located in the same room the problem pc is in and one located in the room the HDMI cable is coming from.

 

[AnonymouseUser]

I'm sorry but what is an AVR? I did a quick google but couldn't find anything useful.

Automatic Voltage Regulation. Without this, the power would be unmodified.

Regarding dirty power, that's what crossed my mind as well. However there is a surge protector in place for all the pc devices. The only thing that is not grounded or hooked to a surge protector is the HDMI cable which is coming from another room that doesn't offer grounding.

Surge protectors only help with surges. The dirty power needs to be fixed or you need some sort of voltage regulation to protect against it.

Also what symptoms would one expect with dirty power? All the devices in the house are working fine with the exception of GPU's frying like crazy; only in this pc though, all the other pc's in the house are fine. One located in the same room the problem pc is in and one located in the room the HDMI cable is coming from.

What are the symptoms of dirty power? While it doesn't mention killing electronics, it does.

 

[Ike0069]

It does sound like "dirty" power could be the culprit, but not sure why it would only take out the VC and nothing else.

 

[Steltek]

It does sound like "dirty" power could be the culprit, but not sure why it would only take out the VC and nothing else.

+1 - start your investigation here. I'd invest in a good quality high capacity APC UPS.

The video card has by far the highest voltage draw of any device in your system. It makes sense that a dirty power source would cause such a failure first. Long term, I wouldn't be surprised if you began to see motherboard issues as well.

Any good quality UPS should include monitoring software and will let you know if you are having severe voltage fluctuations.

It might also be a good idea to get a circuit tester and check the outlet - there could be something majorly wrong with the wiring on that circuit. You definitely need to know if that is the case as it could be a fire hazard if something isn't wired correctly, the outlet is defective, or if the circuit is severely overloaded.

 

[NedStar]

Would something like this suffice?
Price is a bit steep but seeing as I have a 750W power supply I reckon I need something like this cause the cheaper ones show a max output of below that wattage or will that max output only be used in the case of a power outage?

I'd like to hook up all my monitors to it as well and the 865W output might be cutting it short under full load.

Furthermore I'll have a go at testing the outlet as well.

I'm sorry for asking any possible obvious questions but my knowledge in the electrical department is not that high.

Many thanks again.

 

[AnonymouseUser]

Are you overclocking? What monitors do you use?

I would think the 1200 Pro (720W) should be more than sufficient. You don't really need more than one monitor on battery backup, and even a 7970 Crossfire setup will only pull ~640W max. And, unless you do get frequent power outages, you'll probably never be gaming during one.

I would definitely get the outlet checked as well.

 

[AnonymouseUser]

Also, if at all possible, you should reconsider this:

The power outlet is grounded and other pc's run on it with no problems.

You're pulling a lot of power on that one outlet/circuit. It could very well be overloaded and be the cause of the dirty power conditions.

 

[sm625]

With the sort of luck you have you should just start with all new components. It could be something like your case causing it.

 

[NedStar]

Are you overclocking? What monitors do you use?

I would think the 1200 Pro (720W) should be more than sufficient. You don't really need more than one monitor on battery backup, and even a 7970 Crossfire setup will only pull ~640W max. And, unless you do get frequent power outages, you'll probably never be gaming during one.

I would definitely get the outlet checked as well.

I'm indeed overclocking (just cpu) and I use a 7950 crossfire setup. I've measured my monitors a while back and they are roughly 30-40 a piece. Using 3 monitors so that's roughly 90-120.

Power outages are pretty rare down here so I'm more interested in the AVR and security it provides for my system. I've gone ahead and order the 1500pro as the price difference wasn't too great.

I'll get the power outlet properly checked by someone more knowledgeable then me.

Thanks for the advice, much appreciated.

Also, if at all possible, you should reconsider this:

You're pulling a lot of power on that one outlet/circuit. It could very well be overloaded and be the cause of the dirty power conditions.

Cheers, I didn't think to do this. I've now split up my pc from the rest of the applications and will try and split up more when the UPS arrives.

With the sort of luck you have you should just start with all new components. It could be something like your case causing it.

I sadly am thinking of doing this. It sort of became a witch hunt and looking with wary eyes at any part not yet replaced at this point.

 

[NedStar]

Well here I am again..

My freshly replaced GPU for my previous came in Saturday and was installed today. After about 30-60min of usage the pc shut itself down and another GPU was fried. The GPU is once again entirely unresponsive and pc refuses to boot with the GPU powered.

I've replaced everything in the pc since the first GPU's started malfunctioning.
With everything I do mean everything with the exception of my keyboard, mouse and two monitors. Both cables on the monitors have been replaced.

The UPS with AVR is in place and shows no power malfunctioning of any kind. The base house voltage is around 214-225v (EU).
No power spikes of any kind were recorded by the UPS and the voltage during the GPU frying showed 218v.
I also hooked the UPS to another power outlet and moved the other pc's to a separate power group.

I recently replaced the case and every part not yet replaced, this means fans, cpu cooler, hard drivers etc. I've also only plugged in both display port monitors this time around and kept my HDMI cable and audio card out.

I've measured the HDMI, DVI and Display Port cables through but I find nothing out of the ordinary. HDMI was at 0.2v (TV is always off when I game, thus the low voltage), 3-4v on the DVI and 2-3V on the Display Port cables which drops to 0v after a few sec.

I really am desperate at this point and would greatly appreciate any further clues anyone might have.

 

[AnonymouseUser]

Well d*mn. :\

Honestly, outside of software, or a combination of software, I don't know what it could possibly be. Unless somehow the case is the culprit, but even that seems highly unlikely.

 

[Steltek]

Well d*mn. :\

Seconded.

The only other thing I can think of are your monitors, as they have remained the only other constant throughout your entire ordeal. I'm beginning to wonder if you have a defective monitor which is somehow responsible for damaging the video card. This is absolutely the only other thing I can think of. I mean, you've replaced every other single thing else that could be possibly causing the problem. There is literally nothing left that it could be.

I presume you are using standard length and not extended length video cables?

 

[NedStar]

Normal length cables are used. I've just removed two out of my three monitors. I only hooked up two of my three monitors during the last GPU so I reckon it's safe to use the third monitor.

Which monitor might be bust out of the other 2 remaining is beyond me and I have no clue if I could possibly test or get this tested?

Well d*mn. :\

Honestly, outside of software, or a combination of software, I don't know what it could possibly be. Unless somehow the case is the culprit, but even that seems highly unlikely.

The case has been replaced, furthermore I'm using a fresh windows install with bare essentials. Steam, AMD drivers and that's about it.

Edit*

Also I measured the hdmi, dvi and display port cables by hooking up the com to the ground socket and the red (v) to the cable itself. This is with the monitors being on and not plugged in to the GPU as I need something to jam the meter in.
Is this a correct way to measure the output of the cables?

 

[aj654987]

With the sort of luck you have you should just start with all new components. It could be something like your case causing it.

Sounds like something funky is going on. There could be a short somewhere with the case to the motherboard that is causing power issues going to the pci-e slot.

He could try onboard video if he has it or an old pci card if he has one of those and his motherboard has a slot.

First thing I would have said is replace the motherboard in addition to the psu but he says he already did that.

I had a motherboard a while ago that was killing AGP video cards, the psu has been replaced but it was an "open box" motherboard and after a little bit of time it would damage the agp video cards to where the 3d acceleration crashed and no longer worked but 2d desktop mode still worked. Took a while to figure out, new mobo and it didnt happen anymore.

Only other thing I can think is maybe the other PC's in the house are older and thus less sensitive to the "dirty power" on the lines and its something that only bothers high end current generation GPU's. That, a short with the case or two bad motherboards in a row.

 

[aj654987]

Have you moved recently? If you are up in the north east during winter it can get ridiculously dry and I was getting shocked by static electricity all the time in my room until I placed a bowl of water beneath the AC unit to raise the humidity.

 

[aj654987]

The monitors seem like a possibility. It would explain why the other computers dont have issues, and its the video cards repeatedly dying without being anything else and the monitors have been the constant.

There could be some kind of short in the monitor that is sending current where it shouldnt be going to the video card. I mean the monitors shouldnt be sending anything out to the video cards but the video cards shouldnt be dying one after another so something is happening. Figure the video cards put out a low current signal over hdmi/displayport/vga and any of those wires crossing or any current going the wrong way coming from the monitor end would be enough to destroy sensitive electronics.

 

[AnonymouseUser]

At this point the best thing to do is to install a new GPU and try it in another home/building for a while, or run a known working similar PC from another home/building in his home. Even then it may take many hours or even days to get confirmation, but I am thinking it may still be a problem caused by the home electricity. Either that, or NedStar is the worst PC builder ever! (j/k )

 

[NedStar]

Only other thing I can think is maybe the other PC's in the house are older and thus less sensitive to the "dirty power" on the lines and its something that only bothers high end current generation GPU's. That, a short with the case or two bad motherboards in a row.

I'm still not sure about the whole dirty power thing. To my understanding the UPS with AVR should be taking care of that. Not so sure about the harmonics though and whether or not the UPS protects or can protect against it.
That being said I have asked my neighbor who is an electrician to come and investigate the power for me. This should happen somewhere this week.

Regarding mobo or case; I find it too unlikely that after having replaced both, it would still be the problem.

Have you moved recently? If you are up in the north east during winter it can get ridiculously dry and I was getting shocked by static electricity all the time in my room until I placed a bowl of water beneath the AC unit to raise the humidity.

I'm from the Netherlands, if anything it's usually damp over here. Comparable to the UK.
Furthermore I've been living in the same house for the last 15 years and running pc's of the room I'm in for at least 10 years. This is the first time I'm having stuff (gpu's) consistently breaking down.

I used to run three pc's in this room with heavy cards for it's time. 2900xt, 4870x2 and all these never had these kind of issues.

I now run only two pc's in the room my game pc is in and a home server upstairs which is hooked to a none grounded plug which has been running flawless 24/7 for a few years now.
Same can be said of any other application in the house, no malfunction or unexpected behavior of any kind.

There could be some kind of short in the monitor that is sending current where it shouldnt be going to the video card. I mean the monitors shouldnt be sending anything out to the video cards but the video cards shouldnt be dying one after another so something is happening. Figure the video cards put out a low current signal over hdmi/displayport/vga and any of those wires crossing or any current going the wrong way coming from the monitor end would be enough to destroy sensitive electronics.

It's to my understanding that the cable is powered by the monitor in the case of display port.
I'm not entirely sure but I do know there's active and passive display port connections. I know my connections are active. In the case of the monitor, shouldn't that be powering up the display port cable thus also feeding back power?

According to the Displayport faqthis does seem to be the case. Feel free to correct me if I'm wrong.

When I hooked up a multi meter I would measure power going over the cable while only being hooked up to the monitor. All my cables did this and only when coming in contact with the pins so I believe there is some power being fed back from the monitors.

One thing that struck me as odd was that there's always 2.20v going down one monitor which is a different monitor then the other two I got, but the other two have a max of 0.5v going over the cable coming from the monitor.
I'm not sure if this is within acceptable range. The wiki does state the max voltage is 16.

All the GND pins and negative pins have some voltage going through them coming from the monitors. The voltage would be the same across all pins but differ from one monitor versus the other two.

At this point the best thing to do is to install a new GPU and try it in another home/building for a while, or run a known working similar PC from another home/building in his home.)

That's exactly my plan. The current 7950 that is still alive in my pc and lasted a month so far will stay in the pc, albeit with only one monitor. The hopeful refund card will be going into another pc but in the same room. I really couldn't care less about crossfire at this point as long as the pc's actually stay alive for more then a year :X
All round this should give me some useful information if it does tend to break in the same manner which I deem unlikely.

Once again thanks for all the replies and if anyone has some more information regarding the voltages of these cables or how to do further testing that be great.

 

[bononos]

.......
I had a motherboard a while ago that was killing AGP video cards, the psu has been replaced but it was an "open box" motherboard and after a little bit of time it would damage the agp video cards to where the 3d acceleration crashed and no longer worked but 2d desktop mode still worked. Took a while to figure out, new mobo and it didnt happen anymore.

Only other thing I can think is maybe the other PC's in the house are older and thus less sensitive to the "dirty power" on the lines and its something that only bothers high end current generation GPU's. That, a short with the case or two bad motherboards in a row.

If it was back in the agp era, it could have been due to bad caps.

 

[power_hour]

That is a crap load of failures.

I would test it with a completely different system. That will tell you everything you need to know in 30 days. Either its you, your place or your supplier. Pick one or more.

 

[Yeek]

Digging up an old thread as I've just fried 2 units of 290x Lightning GPUs....

NedStar, did you finally find the cause of your fried GPUs?

 

[inachu]

Let's pretend everything is ok and they just get up and fried like that?

I place blame solely on the pc case with cause of death. NOT ENOUGH AIR FLOW.