Hospital had generator and UPS failure

[Mark R]

Got an interesting story today. A local hospital recently had a power outage, but its generator failed to start. Leading to a lot of difficulty on the ICU, with nurses and assistants having to do manual ventilation for 1 hour, difficulties in surgery, etc.

Realising the generator was hosed, they arranged for a rental. A couple of days later a 1 MVA generator on a flatbed was delivered, and after about 2 days the electricians eventually get it wired in.

Things worked well, until the next weekend, when the power went out again. The generator did start, but power to the UPS protected circuits was lost. When it eventually came back a whole bunch of equipment connected to the UPS was toast. Not much medical equipment was damaged, as it tended to have internal UPSs, so was not on the enterprise UPS. However, a whole bunch of servers, switches and SANs were totally fried. It took a few days to get most of the critical apps back on line on new servers after restoring from tape.

The UPS is also toast, so is currently bypassed.

Apparently, the problem was that the UPS had not been correctly connected to the temp generator. When the generator started, the dual-conversion UPS failed to charge. An electrician then manually switched it to bypass mode. Unforutnately, this was a UPS bypass switch, which is make-before-break; normally this would only be used with the UPS correctly synchonized with the mains frequency, but due to the manual rewiring the UPS couldn't sync with the generator. As a result, there was an unsynchronized parallel connection between inverter and generator.

The result: a lot of very expensive magic smoke.

 

[Red Squirrel]

Ouch! That sucks.

I worked in a hospital for a few years, I can't imagine the panic in a situation like this. Our hospital is setup on two hydro grids, each grid has it's own generator and everything is fairly evenly split off. When we lose hydro half the lights and outlets go off, then they come back on once the generator kicks in.

One time the generator did not kick in for one side of the grid. They managed to switch that side to the hydro grid that was still in service though.

I also like how anything power related always somehow falls on IT. "Fix it! This is critical!".

 

[Brian Stirling]

My guess is they don't regularly maintain and test there generators. I'm pretty sure they should be doing that about every two weeks...


Brian

 

[Mark R]

They used to test the generator on a 3 monthly schedule. But I think that stopped a couple of years ago.

 

[werepossum]

Interesting story indeed. Whomever is in charge of maintenance should have had proper signage detailing the procedures for manually switching power from normal to bypass.

 

[Ns1]

What hospital is this so I know never to go there.

 

[MiniDoom]

What hospital is this so I know never to go there.

no shit. i work for a small company and we test our generators weekly. you would think that a hospital would have their shit together.

 

[unokitty]

Why Do Hospital Generators Keep Failing?

It happened in New Orleans after Hurricane Katrina, when hospital staff were on their own when electricity and water cut out. Some died.

It happened last year in San Diego, when generators at two hospitals failed during a blackout.

And it happened last year in Connecticut, when a hospital had to be evacuated during Hurricane Irene when its generator failed.

It also happened in Houston during Tropical Storm Allison when hospitals got flooded and found out that their generators didn't work well when they were under water.

Apparently generators failing in hospitals are not as unusual as one might expect.

Uno

 

[Mark R]

no shit. i work for a small company and we test our generators weekly. you would think that a hospital would have their shit together.

There used to be 2-weekly generator tests but they stopped. It turned out that the testing was too disruptive.

The generator was not powerful enough for normal hospital business, only capable of supplying about 20% of the load.

Even though circuits had been partitioned into generator backed, and non-backup groups, the generator was only capable of supplying about 70% of the "essential" loads connected to the generator supply. The idea was that during generator use, work would be curtailed and equipment shut off to stay within the load rating. Most of this equipment like elevators, office and clinic PCs, etc. did not have UPSs.

As a result, for a generator test to proceed, all planned clinics and office work had to be cancelled. All PCs and non-emergency medical equipment (e.g. MRI and CT scanners) would have to be shut down well in time for the switch over to generator supply.

The generator would then run for an hour. There would then have to be shut down of any PCs, equipment that had been used during that time, prior to restoration of mains power.

The regular tests caused so much disruption, effectively planned work, clinics, etc. had to be cancelled for 2 hours on the day of a test, that management chose to abandon the regular testing as impractical.

 

[Gunbuster]

The generator would then run for an hour. There would then have to be shut down of any PCs, equipment that had been used during that time, prior to restoration of mains power.

This seems pretty silly. Would you not use a simulated load on the genset and not bother anyone in the hospital? I mean yeah do a test of the circuit and switching when you set it up but after that... It's not like your point of failure is going to be a wire in the hospital wall.

 

[Red Squirrel]

Sounds about right. The issue with this mentality is when something DOES go wrong they arn't prepared.

When I worked at the hospital we were not allowed to do any server or workstation updates because "it might break something". Then we got the sasser worm, SQL slammer, Conflicker etc.... and even then they did not want us to do the update, just remove the virus. I had a bunch of stuff in the login script to run virus cleanup tools for the common ones. Come to think of it, I'm not sure why the antivirus was not picking these up. We were actually allowed to update that and we did.

To this day, they're most likely still on windows XP service pack 1.

 

[pcslookout]

Got an interesting story today. A local hospital recently had a power outage, but its generator failed to start. Leading to a lot of difficulty on the ICU, with nurses and assistants having to do manual ventilation for 1 hour, difficulties in surgery, etc.

Realising the generator was hosed, they arranged for a rental. A couple of days later a 1 MVA generator on a flatbed was delivered, and after about 2 days the electricians eventually get it wired in.

Things worked well, until the next weekend, when the power went out again. The generator did start, but power to the UPS protected circuits was lost. When it eventually came back a whole bunch of equipment connected to the UPS was toast. Not much medical equipment was damaged, as it tended to have internal UPSs, so was not on the enterprise UPS. However, a whole bunch of servers, switches and SANs were totally fried. It took a few days to get most of the critical apps back on line on new servers after restoring from tape.

The UPS is also toast, so is currently bypassed.

Apparently, the problem was that the UPS had not been correctly connected to the temp generator. When the generator started, the dual-conversion UPS failed to charge. An electrician then manually switched it to bypass mode. Unforutnately, this was a UPS bypass switch, which is make-before-break; normally this would only be used with the UPS correctly synchonized with the mains frequency, but due to the manual rewiring the UPS couldn't sync with the generator. As a result, there was an unsynchronized parallel connection between inverter and generator.

The result: a lot of very expensive magic smoke.

We had this happen here during a hurricane.

 

[DrPizza]

I think my wife's hospital still tests their generators VERY regularly, because of the fear of what happens when the generator doesn't start, but is needed.

I'm surprised the generator took so long to arrive. My son does that every day for his company - they get a call & they're usually there asap with up to a few megawatts. He sets them up & helps get all the connections jumped over.

Then again, you said, lots of expensive smoke - gee, maybe it WAS my son. (I doubt it) But, he has plenty of horror stories - generally someone else's blunder. Some maintenance idiot seeing that a generator alarm was going off & needed oil added, so poured multiple gallons of oil into something that wasn't where oil goes. Though, he did leave a tool on top of some area of a large generator. They fired the generator up, his screwdriver fell down, into the fan blade, and propelled into a very large radiator.

 

[Red Squirrel]

I work in a NOC so when there are mass power outages (Quebec likes to do that any time it snows. LOL) we have techs running all over with big generators topping up all the batteries. Ever wonder why your land line still works even weeks into a big outage? Yep, it's those guys. We even had them working on Christmas last year. While there are the CO buildings, there are also cabinets on the side of streets, sometimes that act as a "mini CO". Those need generators too. Last Christmas we had like 30 or so sites without power and about 6-7 generators on the go. These guys are working through a blizzard too, some roads are even closed and nothing is plowed, and they're pulling generators through that.

 

[Kaido]

Wow, that would be incredibly stressful, holy cow.

 

[Gillbot]

My guess is they don't regularly maintain and test there generators. I'm pretty sure they should be doing that about every two weeks...


Brian

I was present doing some other related work during our local hospitals testing. If anything didn't work right, it was replaced.

 

[Vic Vega]

...

 

[nageov3t]

the worst night I ever had on the job was when I got the 11 pm call that the generator failed at our data center and we had 3000 servers offline.

turned out that right as grid power failed, lightning struck the generator.

 

[thecoolnessrune]

That's why our datacenters test their generators every single week.

 

[thecoolnessrune]

the worst night I ever had on the job was when I got the 11 pm call that the generator failed at our data center and we had 3000 servers offline.

turned out that right as grid power failed, lightning struck the generator.

Oh goodness the SLA on that but have been horrendous

 

[SKORPI0]

What hospital is this so I know never to go there.

Probably this one.

Power outage hit neighorhoods, hospital

A power outage near downtown Albuquerque left a hospital partially in the dark Saturday and disrupted homes, businesses and traffic signals.Around 10 a.m. the power partially went out at Presbyterian Hospital on Central Avenue SE near Interstate 25.

Can the hospital sue that "electrician" who didn't know that he was doing for damage to hospital equipment? :\

 

[Vic Vega]

...

 

[Vic Vega]

...

 

[rpanic]

This sort of happened at my work. UPS tech put system in bypass for maintenance and forgot to put it back. Power went out, generator kicked in and fried some switches and a server. Cisco made some money that Sunday.

 

[nageov3t]

I doubt a company that stupid has internal SLAs.

I don't think we're stupid

(that said, SLA's are above my paygrade but my understanding is that we only have serious SLA contracts with a couple major clients)

this was years ago, so my memory is foggy, but I think downtime for most servers was limited a couple hours until the generator was repaired.