Server logs

[ultimatebob]

Yeah, a site like Amazon would be on clusters of servers scattered throughout every data center Amazon has. And they have a lot of them.

They probably generate Terabytes of logs daily, but the admins probably don't look at them all that often. Instead, they would probably have some automated process parse them for SEVERE/Critical type errors and send them to an incident management system.

 

[Red Squirrel]

I think OP was also talking more about apache logs, not application logs, which can very a ton based on what you choose to log and how verbose you make it.

My home automation/monitoring system has about 25GB of logs in it's log folder right now since about March 3 when I purged them. Another 4.2GB of output data, but that's going back to since the system was implemented. One of these days I'm going to actually implement graphing of that data, then it will just delete it as it makes the graphs. The logging is mostly in debug mode right now, I just never took it out of that mode as it still has quirks to work out and I'll resume development, eventually.

 

[repoman0]

Anyone have freeware suggestions for text editors to review large log files? A 150 meg HL7 log file can take notepad like 10 minutes to churn and open. I tried Ultra edit for a while and it was decent then the trial expired. Anything free that doesn't suck or riddled with malware?

SMS Trace is awesome for reviewing small log files. But it can't do big stuff.

Never dealt with logs like that, but notepad++ always worked well for me.

+1, notepad++ is much, much faster at opening and scrolling through large files compared to notepad.

 

[Red Squirrel]

Notepad is brutal for opening large files, though isn’t it limited to like 64K or something like that? Then it tries to open in Wordpad and that's just as bad.

Speaking of notepad, type "this app can break" in a new text file. Save it, and open it. Weird things happen.

 

[Markbnj]

If they're smart they ingest the log data into a database like elasticsearch, and use a visualization tool like kibana to find what they need.

 

[PliotronX]

Are you kidding me? At my company (Fairly well known internet site) we have servers that generate 15mbps of pure log data by themselves. I think EXTREMELY conservatively, we generate at least 10 TB of pure log files every single day, post compression.

A small part of my job is to make that data usable. A majority of the data is kept for about 2 years due to PCI compliance issues, majority of the rest of the data is kept for a year for long term archival. A small portion is then kept for 2 weeks at a time for short term analytics. Our 2 week archival cluster is at about 70TB of logs for those 14 days. uncompressed.

Jeepers! I thought our Cisco ISE doing 7-15GB a day was a lot! This thread is actually useful thanks guys

 

[NoTine42]

Notepad is brutal for opening large files, though isn’t it limited to like 64K or something like that? Then it tries to open in Wordpad and that's just as bad.

Speaking of notepad, type "this app can break" in a new text file. Save it, and open it. Weird things happen.

I'm not aware of a practical limit of Notepad in Windows 7, I often open 5-10mb text files. It's OK to find text, but scrolling around is very slow.

 

[John Connor]

Anyone have freeware suggestions for text editors to review large log files? A 150 meg HL7 log file can take notepad like 10 minutes to churn and open. I tried Ultra edit for a while and it was decent then the trial expired. Anything free that doesn't suck or riddled with malware?

SMS Trace is awesome for reviewing small log files. But it can't do big stuff.

I use Apache Logs Viewer. Not sure how it will handle a log file that large though. Would you believe I used to use Notepad ++? What a PITA!

 

[Carson Dyle]

They employ entire villages in India, for 6 to 11 cents per person per 12 hour shift, to read their logs. (Those earning 11 cents a day have upwards of eight years of experience.)

 

[JBT]

For sure they are using some big data type tool for logging. I work at a small utility and uncompressed we are in the tens of GB's per day for just our border devices.

 

[PliotronX]

I'm not aware of a practical limit of Notepad in Windows 7, I often open 5-10mb text files. It's OK to find text, but scrolling around is very slow.

Wordpad has always surprisingly handled large files better than Notepad for me :sneaky:

 

[slag]

Never dealt with logs like that, but notepad++ always worked well for me.

Thats what I use also.

 

[slag]

Notepad is brutal for opening large files, though isn’t it limited to like 64K or something like that? Then it tries to open in Wordpad and that's just as bad.

Speaking of notepad, type "this app can break" in a new text file. Save it, and open it. Weird things happen.

You got me. Nothing happens.

 

[purbeast0]

If they're smart they ingest the log data into a database like elasticsearch, and use a visualization tool like kibana to find what they need.

we recently started using kibana. i really like the interface of it.

 

[BarkingGhostar]

I used to work at Amazon. Their log files are not small. You get good at grep.

This is so true. I use grep multiple times a day in order to sift for the data I am looking for in log files that collectively run into the tens of MB each day.

A day without grep is a day I am not working.