Remote shut down system for a server room..

[Danman]

Hi guys,

Not sure if this should be posted here but I'm gonna do it here anyways.

This past weekend, we took a hard hit at our office. We are in Melbourne, Florida where Hurricane Frances hit. Our generator failed on us after 24 hours so our UPS kicked in. Our UPS then couldn't handle the AC so that failed. I have Net Botz rack mount systems set up that takes the temperature, humidity, etc. readings and if it gets too hot, it will page all of the system admins. Well, it worked but our T1 provider line was down so it couldn't get through to the outside world.

So our 91 servers and a lot of Cisco equipment had no AC for three hours before my boss ventured in to make sure everything was okay. It was so hot that you couldn't touch the metal because you would of been burned.

Most of the servers will power back up but it's obvious we are gonna have a lot of problems, some of them are already flaking out. We got the go to gut everything and replace it, probably close to 350 to 400k in damages. We are gonna let the insurance battle it out later, we need to get this stuff replaced right away.

Lesson learned, but we need to set up some kind of system that will remotely shut down servers if it reaches a certain temperature. We have Windows, Solaris, Linux, and SGI servers so it would have to be able to shut down them all.

Does anyone have experience with this? I'm only the intern so this really isn't my responsibility, but it would really impress everyone if I could come up with a project that we could integrate in our server room.

Thanks guys!

Dan

 

[Matthias99]

Originally posted by: cashman
Lesson learned, but we need to set up some kind of system that will remotely shut down servers if it reaches a certain temperature. We have Windows, Solaris, Linux, and SGI servers so it would have to be able to shut down them all.

Does anyone have experience with this? I'm only the intern so this really isn't my responsibility, but it would really impress everyone if I could come up with a project that we could integrate in our server room.

Thanks guys!

Dan

I'm not sure of the best way to get the environmental data out of the monitoring system, but shutting down UNIX/Linux systems remotely is really, really easy. Just have a script telnet/ssh in (you can use something like Expect to automate the logins, or set it up so it requires no password from the monitoring/shutdown system) and run 'shutdown' as root. Windows is a little trickier, but I'm certain there are remote scripting/automation programs that could do what you want.

Alternatively, you could have a process running on each system that occasionally checks in with some server (which would be hooked up somehow to the temperature monitor), and have the process shut the system down if the room temp gets too high. I don't know of anything off-the-shelf that does this, but developing it would not be that hard.

Also, a lot of PC servers can have temperature-monitoring enabled in the BIOS, and can be set to cut the power if they are overheated (although this does not usually let the system do a clean shutdown).

 

[Danman]

Originally posted by: Matthias99

Originally posted by: cashman
Lesson learned, but we need to set up some kind of system that will remotely shut down servers if it reaches a certain temperature. We have Windows, Solaris, Linux, and SGI servers so it would have to be able to shut down them all.

Does anyone have experience with this? I'm only the intern so this really isn't my responsibility, but it would really impress everyone if I could come up with a project that we could integrate in our server room.

Thanks guys!

Dan

I'm not sure of the best way to get the environmental data out of the monitoring system, but shutting down UNIX/Linux systems remotely is really, really easy. Just have a script telnet/ssh in (you can use something like Expect to automate the logins, or set it up so it requires no password from the monitoring/shutdown system) and run 'shutdown' as root. Windows is a little trickier, but I'm certain there are remote scripting/automation programs that could do what you want.

Alternatively, you could have a process running on each system that occasionally checks in with some server (which would be hooked up somehow to the temperature monitor), and have the process shut the system down if the room temp gets too high. I don't know of anything off-the-shelf that does this, but developing it would not be that hard.

Also, a lot of PC servers can have temperature-monitoring enabled in the BIOS, and can be set to cut the power if they are overheated (although this does not usually let the system do a clean shutdown).

Of course we could all telnet in, but what if we do not have any power? This is what the situation was last weekend, all of our IT guys were out of power and we had no way of checking. We just went over to our office to see what was going on.

Do you know if Dell Poweredge servers have that BIOS option? I've never seen that before.....

Thanks.

Dan

 

[Carbonadium4]

try this..

http://www.netbotz.com/

You may have to chain either all ur servers to a serial cable if you want environment

for windows use

shutdown command
-r restart
-s shutdown
-f force
-t timer ie -t 60 - 60 second
-m \\computer name

shutdown -s -f -t 10 -m \\computername

only admin can shutdown remote pc

use shutdown -a -m \\computername to abort

 

[AluminumStudios]

Some of the APC rackmount UPS's we have have temperature monitors in them according to the web master. Using the APC software (which is available on a variety of platforms) you could probably have machines shut themselves down if the UPS connected to a machine via a serial cable says to shutdown because it's too hot.

I'm fairly certain that the APC software is availbale for Unix platforms and that you can have one machine connected to the UPS send out a network broadcast to have other machines running the APC software shutdown. I haven't worked hands-on with this very much but am pretty sure it can be done pretty easily and cheaply ...