to all sys admins

[rasczak]

ok i need some input here.

I'm putting together a daily checklist for routine maintenance on our windows servers/unix servers.

things i've got so far are review system logs, application logs, security logs.
verify disk space
verify daily backups
anti virus updates/scans

if you have any thing to add it would be greatly appreciated.

 

[Nothinman]

On Linux you should be using something like logcheck to email you only the things you don't want to see, if properly tuned for your environment it can save you a ton of time. And even better would be to have all of your servers sending their logs to a central server via syslog so that you only have one place to worry about.

 

[VinDSL]

Originally posted by: rasczak
if you have any thing to add it would be greatly appreciated.

Microsoft 'forgot' to include a syslog in their products. It's probably their single biggest mistake - and the reason I don't take MS seriously...

No problem. This is how I get around it (been running this combo for years):

http://www.kiwisyslog.com/kiwi-syslog-server-overview/ (Kiwi Enterprises | Kiwi Syslog Server)

This is an ESSENTIAL tool. If you're serious about admin'ing Windows machines... you'll get it!

As great as Kiwi Syslog is... that's only half the battle. You also need to install Snare:

http://www.intersectalliance.c...nareWindows/index.html (Intersect | Snare Agent for Windows)

This will increase the capabilities of Kiwi Syslog exponentially!

I find Kiwi Syslog and Snare indispensable for managing Windows machines.

LoL! Let's see how serious you are, and we'll go from there...

 

[Nothinman]

Microsoft 'forgot' to include a syslog in their products. It's probably their single biggest mistake - and the reason I don't take MS seriously...

MS didn't forget and if you really think that's their single biggest mistake you're not paying close enough attention.

 

[VinDSL]

Originally posted by: Nothinman

Microsoft 'forgot' to include a syslog in their products. It's probably their single biggest mistake - and the reason I don't take MS seriously...

MS didn't forget and if you really think that's their single biggest mistake you're not paying close enough attention.

You need to practice your reading comprehension!

Did you consider the context of this thread... not to mention, the OP types in all lower-case letters?

Dude, you're always coming to a gun fight with a knife! Give it up...

 

[Nothinman]

You need to practice your reading comprehension!

My reading comprehension is just fine, it's your writing comprehension that needs work. Go ahead and start a poll asking who's posts are more legible and on topic and see where that goes...

Did you consider the context of this thread... not to mention, the OP types in all lower-case letters?

I did consider the context of the thread, infact my first reply was more on topic than about 1/3 of your post. And I'm not even sure how the lack of proper capitalization even applies. Good presentation counts for something, but it's not all that important.

Dude, you're always coming to a gun fight with a knife! Give it up...

See? WTF are you even talking about?

 

[giverson]

Something like Nagios is invaluable for keeping an eye on things like disk space and services. Set the thresholds properly and you'll know about a disk space issue before any users notice.

Thanks for the tip on Snare, VinDSL. That'll fill one of the last gaps in our monitoring.

 

[seepy83]

These may or may not fall out-side of your Sys Admin responsibilities (depending on your organization/IT staff)

Check UPS's for fault lights.
Check Vendor-Specific hardware monitoring software, or lights on servers.

Personally, I like to do a quick review of a packet capture with Wireshark every once in a while to make sure there's nothing strange going on on-the-wire that no one is noticing.

While you're checking utilization (like disk space), it's usually helpful to put that data into a spreadsheet so you can predict when upgrades will be necessary.

 

[Nothinman]

Something like Nagios is invaluable for keeping an eye on things like disk space and services.

The last time I looked at Nagios it was fairly complicated to configure, something like ZenOSS might simpler.

While you're checking utilization (like disk space), it's usually helpful to put that data into a spreadsheet so you can predict when upgrades will be necessary.

Or if you use something that keeps historical graphs you can use that to trend growth.

 

[VinDSL]

Originally posted by: giverson
Thanks for the tip on Snare, VinDSL. That'll fill one of the last gaps in our monitoring.

Snare is the bomb!

Used along with a syslog daemon, it'll show you everything going on under the curtain... from bootup to shutdown. Matter of fact, it'll drive you crazy after a while.

Luckily, Intersect's "Snare Agent for Windows" allows you to configure it easily, through a slick online app, so you can tailor it to your specific needs (without a lot of hassle).

EDIT

Here's a screenie for you...

http://www.vindsl.com/images/K...log-Snare_Screenie.png (VinDSL.com | Kiwi Syslog Daemon showing Win Events via Snare)

Nothing 'exciting' going on right now, but it will give you the idea...

 

[imported_snikt]

One thing I haven't seen mentioned, if you have app. servers that are running proprietary apps and services, verify those proprietary apps/services are running or haven't hung up. We run a few proprietary apps which have specific services that need to be running and once in a while the services may hang or have stopped inexplicably and require the services to be restarted or the servers may have to be bounced altogether.

 

[rasczak]

Originally posted by: VinDSL

Originally posted by: Nothinman

Microsoft 'forgot' to include a syslog in their products. It's probably their single biggest mistake - and the reason I don't take MS seriously...

MS didn't forget and if you really think that's their single biggest mistake you're not paying close enough attention.

You need to practice your reading comprehension!

Did you consider the context of this thread... not to mention, the OP types in all lower-case letters?

Dude, you're always coming to a gun fight with a knife! Give it up...

Really not sure what capitalization has to do with the original question, but i've already looked into kiwi syslog, however, due to the "nature" of the project i am on, we're not allowed to use software not on an approved list (of which syslog isn't). i appreciate the input i am very serous about the work i'm doing and am still learning a lot.

Originally posted by: snikt
One thing I haven't seen mentioned, if you have app. servers that are running proprietary apps and services, verify those proprietary apps/services are running or haven't hung up. We run a few proprietary apps which have specific services that need to be running and once in a while the services may hang or have stopped inexplicably and require the services to be restarted or the servers may have to be bounced altogether.

Thanks. This is already a part of my morning checks. We too use proprietary apps.

 

[RebateMonger]

There's lots of software that will do these checks automatically and send you a daily report (or an alert if it's an emergency). Having a person do these checks is time-consuming and error-prone.

The stuff I use for Windows servers and networks is $15 a month per server and it will also monitor client PCs for the same issues.

 

[rasczak]

Originally posted by: RebateMonger
There's lots of software that will do these checks automatically and send you a daily report (or an alert if it's an emergency). Having a person do these checks is time-consuming and error-prone.

The stuff I use for Windows servers and networks is $15 a month per server and it will also monitor client PCs for the same issues.

i agree, however a lot of the "checks" are for standalone systems not connected to the domain for various reasons and as well we're only able to use software from an approved list. kiwi is a no go, however, if it was something microsoft related then that would probably be a different story. trust me, i would love nothing more than to have a simple report telling me my disk space is low on tppr0nserver.xxx.com

 

[Nothinman]

however, due to the "nature" of the project i am on, we're not allowed to use software not on an approved list (of which syslog isn't). i appreciate the input i am very serous about the work i'm doing and am still learning a lot.

Then you're going to have a hard time finding anything comprehensive that will accomplish your monitoring goals. Writing something yourself for some of it wouldn't be that difficult but people have already done that in the forms of the mentioned software. You should really see if you can convince them to let you use something.

 

[Reliant]

Originally posted by: giverson
Something like Nagios is invaluable for keeping an eye on things like disk space and services. Set the thresholds properly and you'll know about a disk space issue before any users notice.

Thanks for the tip on Snare, VinDSL. That'll fill one of the last gaps in our monitoring.

This. I'm responsible for several hundred servers and I would hate my life if I had to manually check things like disk space daily. Nagios is kind of a pain to get going, but once it's up and configured it's a really great tool.

 

[n0cmonkey]

Originally posted by: Nothinman

Something like Nagios is invaluable for keeping an eye on things like disk space and services.

The last time I looked at Nagios it was fairly complicated to configure, something like ZenOSS might simpler.

The initial setup is a PITA, but once you've got it running (and templated EVERYTHING) it's cake.

I think Zenoss is one of those products that I've heard is easy to setup but you need someone at it full time to get/keep it useful. Haven't tried it though.

Or if you use something that keeps historical graphs you can use that to trend growth.

That's the best idea. Spreadsheets are sooooo 1990's.

 

[n0cmonkey]

Originally posted by: Nothinman

however, due to the "nature" of the project i am on, we're not allowed to use software not on an approved list (of which syslog isn't). i appreciate the input i am very serous about the work i'm doing and am still learning a lot.

Then you're going to have a hard time finding anything comprehensive that will accomplish your monitoring goals. Writing something yourself for some of it wouldn't be that difficult but people have already done that in the forms of the mentioned software. You should really see if you can convince them to let you use something.

Nah, it means a call to the local CA or IBM reseller is in order. Unicenter or Tivoli should be on pretty much every list. Even if Nagios isn't (and you'd be surprised where you'll find Nagios ).

Now, there should also be a log centralizing agent for Windows of some sort on that list. Leaving something as critical as that off the approved software list for a serious "organization" doing serious "work" is a BIG mistake. Probably just have to know which vendor to buy it from.

 

[n0cmonkey]

Check to see if OSSEC HIDS is on your list. It can help out with logs. Haven't found a product to sift through and centralize logs that can beat it.

 

[sourceninja]

Originally posted by: rasczak

Originally posted by: VinDSL

Originally posted by: Nothinman

Microsoft 'forgot' to include a syslog in their products. It's probably their single biggest mistake - and the reason I don't take MS seriously...

MS didn't forget and if you really think that's their single biggest mistake you're not paying close enough attention.

You need to practice your reading comprehension!

Did you consider the context of this thread... not to mention, the OP types in all lower-case letters?

Dude, you're always coming to a gun fight with a knife! Give it up...

Really not sure what capitalization has to do with the original question, but i've already looked into kiwi syslog, however, due to the "nature" of the project i am on, we're not allowed to use software not on an approved list (of which syslog isn't). i appreciate the input i am very serous about the work i'm doing and am still learning a lot.

Originally posted by: snikt
One thing I haven't seen mentioned, if you have app. servers that are running proprietary apps and services, verify those proprietary apps/services are running or haven't hung up. We run a few proprietary apps which have specific services that need to be running and once in a while the services may hang or have stopped inexplicably and require the services to be restarted or the servers may have to be bounced altogether.

Thanks. This is already a part of my morning checks. We too use proprietary apps.

Umm, almost every single linux OS out there ships with syslog by default. Same with solaris....It's going to be tough not to use it.

Oh wait, I see you are refering to a windows syslog daemon. Well that sucks.

I have all my servers log to a single logging server here. Then a handful of cron job scripts check them and send me things of interest.

 

[Nothinman]

Oh wait, I see you are refering to a windows syslog daemon. Well that sucks.

And it would probably take some doing to get approval to have all of the servers send their logs over the network because they might contain senitive information and it opens up the possibility of a DoS attack.

 

[n0cmonkey]

Originally posted by: Nothinman

Oh wait, I see you are refering to a windows syslog daemon. Well that sucks.

And it would probably take some doing to get approval to have all of the servers send their logs over the network because they might contain senitive information and it opens up the possibility of a DoS attack.

That's why you have a separate logging network.

 

[seepy83]

Originally posted by: n0cmonkey

Originally posted by: Nothinman

Or if you use something that keeps historical graphs you can use that to trend growth.

That's the best idea. Spreadsheets are sooooo 1990's.

Wow...since when does Excel not do graphs/trending? I must be really out of the loop :roll:

 

[sonoma1993]

Originally posted by: rasczak

Originally posted by: RebateMonger
There's lots of software that will do these checks automatically and send you a daily report (or an alert if it's an emergency). Having a person do these checks is time-consuming and error-prone.

The stuff I use for Windows servers and networks is $15 a month per server and it will also monitor client PCs for the same issues.

i agree, however a lot of the "checks" are for standalone systems not connected to the domain for various reasons and as well we're only able to use software from an approved list. kiwi is a no go, however, if it was something microsoft related then that would probably be a different story. trust me, i would love nothing more than to have a simple report telling me my disk space is low on tppr0nserver.xxx.com

check out spiceworks, great tool, can be use to check for antivirus, event logs, disk space, and quite few other things.

 

[n0cmonkey]

Originally posted by: seepy83
Wow...since when does Excel not do graphs/trending? I must be really out of the loop :roll:

Is it a question of whether it does those things or whether it does them well? Yes, it does graphs (can't comment on trending, never really tried it).

Does it do them well? Well... No. It is too manual, and I assume (yes, with all implications of assume), it has trouble handling large data sets. It is also relatively static.

Input 2 million data points.
Create a graph.
Update it with 2 million more data points.
Does your graph change automagically?
In real time?
What kind of iron do you need to view that file?
Now, do it again for the other 300 things you want to track.
Now get started with the data on system 2.

Another thing a lot of these tools are good at is reporting. I want email/page/sms when something happens. Excel can't give me that.

I'm guessing excel is fine for small non-24/7/365 type shops, but I've never really worked in one of those.

EDIT: Here are 2 examples of applications that might just be appropriate for bigger installations:
Reconnoiter - it's a bit beta at the moment, but the idea (and people behind it) is great. I'm begging for an 8th day to the week (named monkeyday, a weekend day of course) so that I'd have time to play with it.
Splunk - this puts a lot of log tools to shame. The interface has got to be the prettiest I have ever played with. It's another monkeyday project though. Well, that and lack of spare hardware at the moment.