Hospital had generator and UPS failure

Page 3 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

OverVolt

Lifer
Aug 31, 2002
14,278
89
91
I'm not surprised hospitals aren't doing that great financially. Its like shit you'd expect you hear happening in 3rd world countries.
 

Mark R

Diamond Member
Oct 9, 1999
8,513
14
81
Another update: Uupgrade work is well underway.

New transformers are being delivered next month. Then in another 4 weeks, they're switching over to the new supply. There will be a 24 hour period with no mains power, and the hospital will be powered by their generator.

The big issue is that a lot of medical equipment has never been tested on the generator, because it had always been assumed that mains supply interruptions would be short. For example, prior to routine generator testing, all big power hogs, like MRI scanners would be shut down and powered off for the duration.

A bunch of people have gone round contacting the various equipment manufacturers asking about generators. The MRI scanner manufacturers have come back saying that the scanners should not be operated on generator power, as the voltage quality is unlikely to be good enough for their sensitive equipment. However, continuous generator power must be provided to the superconductor support equipment.

The CT manufacturer says that the voltage input to the scanner must be within +/- 5% of nominal during a scan. They say the scanner may work if the voltage fluctuates more than this, but the images may be blurred or distorted, or the scanner may abort the scan. The electricians aren't sure that our 700 kVA generator will handle the CT scanner going from 1 kW to 200 kW (it's nominal maximum rating) instantly without the voltage going out of range. They're planning to get a guy with a power analyser in to see what the voltages/currents actually do during "normal" use, and if the numbers look reasonable, they'll test it on the generator

If it doesn't work, then we'll have time to arrange rental of a bigger generator with better voltage regulatotion on a temporary basis - although the main electrician did say that he wasn't sure if any generator supplier could guarantee voltage regulation under "max power" scanning conditions.
 

NetWareHead

THAT guy
Aug 10, 2002
5,854
154
106
Another update: Uupgrade work is well underway.

New transformers are being delivered next month. Then in another 4 weeks, they're switching over to the new supply. There will be a 24 hour period with no mains power, and the hospital will be powered by their generator.

The big issue is that a lot of medical equipment has never been tested on the generator, because it had always been assumed that mains supply interruptions would be short. For example, prior to routine generator testing, all big power hogs, like MRI scanners would be shut down and powered off for the duration.

A bunch of people have gone round contacting the various equipment manufacturers asking about generators. The MRI scanner manufacturers have come back saying that the scanners should not be operated on generator power, as the voltage quality is unlikely to be good enough for their sensitive equipment. However, continuous generator power must be provided to the superconductor support equipment.

The CT manufacturer says that the voltage input to the scanner must be within +/- 5% of nominal during a scan. They say the scanner may work if the voltage fluctuates more than this, but the images may be blurred or distorted, or the scanner may abort the scan. The electricians aren't sure that our 700 kVA generator will handle the CT scanner going from 1 kW to 200 kW (it's nominal maximum rating) instantly without the voltage going out of range. They're planning to get a guy with a power analyser in to see what the voltages/currents actually do during "normal" use, and if the numbers look reasonable, they'll test it on the generator

If it doesn't work, then we'll have time to arrange rental of a bigger generator with better voltage regulatotion on a temporary basis - although the main electrician did say that he wasn't sure if any generator supplier could guarantee voltage regulation under "max power" scanning conditions.

wow, one appliance with a 200 kW startup load!!!!!!

Is this a diesel generator? Usually most diesels won't have that much of a problem adjusting rapidly to a huge new load. We have a Cummins generator that is natural gas fueled and we were advised that diesel is better handling varying and heavy loads than gas. We still went gas because we dont have that sort of equipment here (mostly datacenter type electrical loads rather than machinery) as well as to take advantage of cheaper fuel cost and no need for fuel delivery.
 

OutHouse

Lifer
Jun 5, 2000
36,413
616
126
If it doesn't work, then we'll have time to arrange rental of a bigger generator with better voltage regulatotion on a temporary basis -


last year we rented a generator when we did a power shift when we add a third diesel generator which required a total disconnect from city power for just about 30 hours. the rental came in on a lowboy and was wired ready to go for a hot standby in case our other two failed. but i didnt have to deal with all the crazy power hogs like you do, its just a multi story office building and we did it on a weekend. It was expensive but far cheaper than if we lost total power.
 

monkeydelmagico

Diamond Member
Nov 16, 2011
3,961
145
106
What kind of micky mouse operation is this hospital? Centralized line interactive ups has been the standard for hospital power backup for well over a decade.
 

Red Squirrel

No Lifer
May 24, 2003
67,898
12,365
126
www.anyf.ca
What kind of micky mouse operation is this hospital? Centralized line interactive ups has been the standard for hospital power backup for well over a decade.

I brought that idea up at my hospital since they got power outages all the time, really short blips but it was enough to cause computers to shut down or freeze, and was probably hard on medical equipment too.

They had some weird political reason as to why they could not do it. It had to be hospital grade or something. I find that's silly, it's best to have a non hospital grade system that can be bypassed, than to not have it at all.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
I brought that idea up at my hospital since they got power outages all the time, really short blips but it was enough to cause computers to shut down or freeze, and was probably hard on medical equipment too.

They had some weird political reason as to why they could not do it. It had to be hospital grade or something. I find that's silly, it's best to have a non hospital grade system that can be bypassed, than to not have it at all.

The grade was likely something about reliability and run time combined with how clean the power is. The equipment might not be tolerant of poor power and since failure can mean people die, they tend to be picky about it.
 

MarkXIX

Platinum Member
Jan 3, 2010
2,642
1
71
What kind of micky mouse operation is this hospital? Centralized line interactive ups has been the standard for hospital power backup for well over a decade.

I agree, seems odd that there appears to be a complete lack of centralized or even decentralized large UPS/battery capability on critical equipment in a hospital.

I'm not an electrician by any means, but as for the MRI I would think it would have its own battery room to "smooth out" the typical power fluctuations a generator puts into the system.

Its like I said before, critical systems should have batteries in between them and whatever the power source is. Critical computers should either have an UPS under the desk or should be a laptop in a dock.
 

Mark R

Diamond Member
Oct 9, 1999
8,513
14
81
wow, one appliance with a 200 kW startup load!!!!!!

That's not the startup load, that's operating load - although it doesn't operate for very long. Usually only 5 seconds at a time, but occasionally up to 30 seconds.

Unlike a motor load, this is an electronic load - so more akin to a server cluster which was previously idling, suddenly going to 100% CPU/GPU.

Obviously, the scanner has an internal SMPS to convert mains voltage (400V AC) to DC (140 kV, 1000 mA) but the question is whether the PSU will be able to properly regulate the voltage on its output rail if the generator voltage sags when the load comes on.

Is this a diesel generator? Usually most diesels won't have that much of a problem adjusting rapidly to a huge new load.

Let's hope so. We hope that "real life" scanner operation isn't so heavy a load. I wouldn't be that surprised if while maximum load is 200 kW, a "normal" peak load is about 50 kW. We won't know for sure, until someone gets a power analyser hooked up.
 

Mark R

Diamond Member
Oct 9, 1999
8,513
14
81
I agree, seems odd that there appears to be a complete lack of centralized or even decentralized large UPS/battery capability on critical equipment in a hospital.
A hospital is a big load, and not all equipment is "critical". More importantly, a lot of equipment has very "difficult" load characteristics.

E.g. a surgical laser might be designed for a pulse rate of 10 Hz, so must be able to charge its capacitors is 0.1 s, at a charge current of 100A. But a realistic use, would be 1 Hz, so that every time the laser fires, the lights dim for a fraction of a second as the caps recharge. This type of load is ill suited to UPSs. I know of one hospital that was testing one, and even though the OR suite had a big central UPS, the laser actually caused the UPS to trip-off (it didn't actually cut off the power, but it switched from dual-conversion mode to "fault bypass" mode, where it switched to unprotected mains power having tripped on an "inverter fault").

Similarly, a high end MRI scanner might take 0.5s current pulses of 300A @ 400V to operate its control magnets, but for 90% of the time, the load will be near zero. Again, this sort of load is poorly suited to a UPS, and the manufactures specifically recommended that the MRI scanners NOT go on a UPS, with the exception of the control/data acquisition computer, for reasons of excessive cost, maintenance requirement and space requirement.

Its like I said before, critical systems should have batteries in between them and whatever the power source is. Critical computers should either have an UPS under the desk or should be a laptop in a dock.

The hospital does have a large central UPS for critical areas - the main problem in the original incident was that it drained when the generator failed to start.
 

MarkXIX

Platinum Member
Jan 3, 2010
2,642
1
71
A hospital is a big load, and not all equipment is "critical". More importantly, a lot of equipment has very "difficult" load characteristics.

E.g. a surgical laser might be designed for a pulse rate of 10 Hz, so must be able to charge its capacitors is 0.1 s, at a charge current of 100A. But a realistic use, would be 1 Hz, so that every time the laser fires, the lights dim for a fraction of a second as the caps recharge. This type of load is ill suited to UPSs. I know of one hospital that was testing one, and even though the OR suite had a big central UPS, the laser actually caused the UPS to trip-off (it didn't actually cut off the power, but it switched from dual-conversion mode to "fault bypass" mode, where it switched to unprotected mains power having tripped on an "inverter fault").

Similarly, a high end MRI scanner might take 0.5s current pulses of 300A @ 400V to operate its control magnets, but for 90% of the time, the load will be near zero. Again, this sort of load is poorly suited to a UPS, and the manufactures specifically recommended that the MRI scanners NOT go on a UPS, with the exception of the control/data acquisition computer, for reasons of excessive cost, maintenance requirement and space requirement.



The hospital does have a large central UPS for critical areas - the main problem in the original incident was that it drained when the generator failed to start.

Learn something new every day. I will keep my IT experience in the Army, where most of the time our stuff is ruggedized and built for less than ideal conditions. No medical facility work for me anytime soon!
 

Red Squirrel

No Lifer
May 24, 2003
67,898
12,365
126
www.anyf.ca
Learn something new every day. I will keep my IT experience in the Army, where most of the time our stuff is ruggedized and built for less than ideal conditions. No medical facility work for me anytime soon!

Yeah I don't miss working in medical, shoe string budget and everything is go go go go and if something goes wrong they want to blame someone right away. I guess it's not just medical that's like that but I find it's worse because lives are possibly at stake, and they'll blame the lowest person on the totem pole when someone dies, which is IT. I got blamed for a death once. You learn to just take it even though you know 100% it's not your fault. Easier than trying to argue.
 

Brovane

Diamond Member
Dec 18, 2001
5,482
1,673
136
A hospital is a big load, and not all equipment is "critical". More importantly, a lot of equipment has very "difficult" load characteristics.

E.g. a surgical laser might be designed for a pulse rate of 10 Hz, so must be able to charge its capacitors is 0.1 s, at a charge current of 100A. But a realistic use, would be 1 Hz, so that every time the laser fires, the lights dim for a fraction of a second as the caps recharge. This type of load is ill suited to UPSs. I know of one hospital that was testing one, and even though the OR suite had a big central UPS, the laser actually caused the UPS to trip-off (it didn't actually cut off the power, but it switched from dual-conversion mode to "fault bypass" mode, where it switched to unprotected mains power having tripped on an "inverter fault").

Similarly, a high end MRI scanner might take 0.5s current pulses of 300A @ 400V to operate its control magnets, but for 90% of the time, the load will be near zero. Again, this sort of load is poorly suited to a UPS, and the manufactures specifically recommended that the MRI scanners NOT go on a UPS, with the exception of the control/data acquisition computer, for reasons of excessive cost, maintenance requirement and space requirement.



The hospital does have a large central UPS for critical areas - the main problem in the original incident was that it drained when the generator failed to start.

Any consideration to flywheel system to keep things up until Generators kick in? Looks like these Hospitals used Flywheels instead of Battery UPS's to keep there imaging systems up.

http://www.vyconenergy.com/pq/pdfs/130626_VYCON_BannerCaseStudy.pdf

http://www.vyconenergy.com/pq/pdfs/130802_VYCON_TexasScottishRiteCaseStudy.pdf
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
Any consideration to flywheel system to keep things up until Generators kick in? Looks like these Hospitals used Flywheels instead of Battery UPS's to keep there imaging systems up.

http://www.vyconenergy.com/pq/pdfs/130626_VYCON_BannerCaseStudy.pdf

http://www.vyconenergy.com/pq/pdfs/130802_VYCON_TexasScottishRiteCaseStudy.pdf

In (some of) the data centers we built, the big issue with fly wheels is they tended to max out at about 30 seconds of run time. In most cases this was fine for an outage and transition to generator except when a gen set decided to stumble to life and took a bit longer to stabilize especially if they took longer than expected to parallel properly. They ended up putting in both. Flywheels to handle transitions, UPS to handle a stalled start that gave them enough time to cross connect the load on to another on site gen group. The site had 16 gen sets onsite.

I would imagine that they looked at this. However if they can't seem to maintain their generators, who says they will maintain the flywheels or that the gen set will even start in the 15-30 window they have?
 

OverVolt

Lifer
Aug 31, 2002
14,278
89
91
are power outages seriously a common thing in the US?

We have mostly above ground power lines. Its a big country. Most of Europe has underground lines. So tress fall down and take the lines with them all the time in big storms.
 

mmntech

Lifer
Sep 20, 2007
17,504
12
0
They test the generators at work monthly. The station has a contract with the CRTC that has them handling emergency alerting for the country, so they absolutely need to be operational in times of a power outage.

It's surprising that hospitals don't have at least two generators for redundancy. Guess they're too cheap. Then something bad does happen and they end up paying more in the long run from all the lawsuits.
 

monkeydelmagico

Diamond Member
Nov 16, 2011
3,961
145
106
It's surprising that hospitals don't have at least two generators for redundancy. Guess they're too cheap. Then something bad does happen and they end up paying more in the long run from all the lawsuits.

The hospital I work for has 3 main and some mobile supplementals in case of line failures. We can run the entire place with 2 at around 80% of rated load during peak summer months.

There is something seriously wrong with the way the OP's hospital is, and has been, handling emergency power.
 

imagoon

Diamond Member
Feb 19, 2003
5,199
0
0
There is something seriously wrong with the way the OP's hospital is, and has been, handling emergency power.

Sadly this is "situation normal" for lots of orgs. Generators and their operations are a pretty large line item which makes some finance people target it as "low hanging fruit." I find that most people have serious difficulties evaluating risk which as a combo makes something like "Generator testing and maintenance" something they feel they can red line off.

Then people die. Lawsuits etc. But that is a whole different budget!! amirite? Those lawyers are already on the payroll and they are bored!
 

MarkXIX

Platinum Member
Jan 3, 2010
2,642
1
71
Sadly this is "situation normal" for lots of orgs. Generators and their operations are a pretty large line item which makes some finance people target it as "low hanging fruit." I find that most people have serious difficulties evaluating risk which as a combo makes something like "Generator testing and maintenance" something they feel they can red line off.

Then people die. Lawsuits etc. But that is a whole different budget!! amirite? Those lawyers are already on the payroll and they are bored!

This. Our public works folks are always trying to pawn off generator upkeep labor and costs, that is until there's a failure and they lose their IT operations during a crisis when they need it and then they're back on board doing it for another year or so until they forget again.

That, or we recommend they purchase the requisite battery systems for their mission supporting IT operation, then they come back around too.
 
sale-70-410-exam    | Exam-200-125-pdf    | we-sale-70-410-exam    | hot-sale-70-410-exam    | Latest-exam-700-603-Dumps    | Dumps-98-363-exams-date    | Certs-200-125-date    | Dumps-300-075-exams-date    | hot-sale-book-C8010-726-book    | Hot-Sale-200-310-Exam    | Exam-Description-200-310-dumps?    | hot-sale-book-200-125-book    | Latest-Updated-300-209-Exam    | Dumps-210-260-exams-date    | Download-200-125-Exam-PDF    | Exam-Description-300-101-dumps    | Certs-300-101-date    | Hot-Sale-300-075-Exam    | Latest-exam-200-125-Dumps    | Exam-Description-200-125-dumps    | Latest-Updated-300-075-Exam    | hot-sale-book-210-260-book    | Dumps-200-901-exams-date    | Certs-200-901-date    | Latest-exam-1Z0-062-Dumps    | Hot-Sale-1Z0-062-Exam    | Certs-CSSLP-date    | 100%-Pass-70-383-Exams    | Latest-JN0-360-real-exam-questions    | 100%-Pass-4A0-100-Real-Exam-Questions    | Dumps-300-135-exams-date    | Passed-200-105-Tech-Exams    | Latest-Updated-200-310-Exam    | Download-300-070-Exam-PDF    | Hot-Sale-JN0-360-Exam    | 100%-Pass-JN0-360-Exams    | 100%-Pass-JN0-360-Real-Exam-Questions    | Dumps-JN0-360-exams-date    | Exam-Description-1Z0-876-dumps    | Latest-exam-1Z0-876-Dumps    | Dumps-HPE0-Y53-exams-date    | 2017-Latest-HPE0-Y53-Exam    | 100%-Pass-HPE0-Y53-Real-Exam-Questions    | Pass-4A0-100-Exam    | Latest-4A0-100-Questions    | Dumps-98-365-exams-date    | 2017-Latest-98-365-Exam    | 100%-Pass-VCS-254-Exams    | 2017-Latest-VCS-273-Exam    | Dumps-200-355-exams-date    | 2017-Latest-300-320-Exam    | Pass-300-101-Exam    | 100%-Pass-300-115-Exams    |
http://www.portvapes.co.uk/    | http://www.portvapes.co.uk/    |