Server logs

[John Connor]

So I was just thinking. I have a few websites and I read the server logs everyday. Their size is no more than about ~250KB in size. So I was thinking about how large the server logs for eBay or Amazon would be. I'm thinking, and I could be way off, that eBay produces a server log of about ~15MB in size per hour. Which means in one day that server log would be ~360 MB.

Then I was wondering. Do they read the log continuously or just keep them all archived until they need to look at something?

 

[Homerboy]

You are way off.

 

[KentState]

Seriously? Amazon probably has 5MB of log data per visitor session per day.

 

[slag]

I'm a sys admin at a smallish insurance company. Our server logs roll at 100 MB so they are manageable. We only have them set to info level, not debug unless we are actively researching a problem and are definitely not tracing the system calls to identify issues. Just one of the systems I look at regularly which isn't even a "busy" app server, rolls between every 15 and 20 minutes roughly. Imagine the app server for the front end of your business. It rolls every 6-10 minutes, based on load and what people are doing.

The web server logs just forward/proxy requests to the back end app server that does all the heavy lifting (which in turn talks to the various databases, mainframe IMS regions, CICS, etc).

 

[John Connor]

Seriously? Amazon probably has 5MB of log data per visitor session per day.

Are you sure? I could run a network sniffer and capture a season at Amazon and it probably won't be no more than 200KB minus the captured packets.

But like I stated, I could be way off. And I said eBay.

 

[Homerboy]

Are you sure? I could run a network sniffer and capture a season at Amazon and it probably won't be no more than 200KB minus the captured packets.

But like I stated, I could be way off. And I said eBay.

yes, you are WAY off. And I'm guessing they do not dump logs to a .txt file. And I am sure that nobody sits there and "reviews" them manually.

Amazon gets upwards of 200 Million visitors per month or something like that (more than double eBay). They do nothing like you do.

 

[John Connor]

They do nothing like you do.

Explain. :twisted:

 

[Hugh Jass]

Good lord...who gives a shit?

 

[John Connor]

Apparently you do?

 

[kt]

Are you sure? I could run a network sniffer and capture a season at Amazon and it probably won't be no more than 200KB minus the captured packets.

But like I stated, I could be way off. And I said eBay.

I highly doubt logs are stored on the individual servers. Logs are most likely captured to a centralized location where the data can be analyzed and monitored.

 

[John Connor]

That's what I was thinking.

 

[IronWing]

Each morning, Jeff Bezos prints out and reviews the previous day's server logs before the morning executive briefing. It allows him to ask pertinent questions at the morning executive briefing.

 

[John Connor]

Each morning, Jeff Bezos prints out and reviews the previous day's server logs before the morning executive briefing. It allows him to ask pertinent questions at the morning executive briefing.

Sounds prudent. I like this... who? Jeff Bezos?

:awe:

46 billion all started from his garage. If that's not the American dream I don't know what is.

 

[Crusty]

Are you sure? I could run a network sniffer and capture a season at Amazon and it probably won't be no more than 200KB minus the captured packets.

But like I stated, I could be way off. And I said eBay.

Logging does not mean they are storing the raw network packets in the log file. A typical log line will include all sorts of text data like logging level, timestamp, process ID and then the actual log statement as formatted by the application/server writing to the log file. Lot's of server applications will also write out instrumentation data to a log stream, stuff like how long it took querying the database, rendering the HTML page, or total time processing a request/response life cycle.

Log files are formatted in such a way that automated analysis of them can be done to provide an overview of what's going. Stuff like server response types(success, errors), user behaviors, server performance. I highly doubt that there is someone on the payroll who's job is to just read server logs looking for things.

 

[zinfamous]

Sounds prudent. I like this... who? Jeff Bezos?

:awe:

46 billion all started from his garage. If that's not the American dream I don't know what is.

Why, the Stucky's Pecan Log; that's the American dream.

 

[Deeko]

I used to work at Amazon. Their log files are not small. You get good at grep.

 

[Red Squirrel]

My guess is they have scripts that look for "triggers" ex, suspicious activity, then that gets flagged, then they get someone to review it only then.

 

[nakedfrog]

When we wanted to do in-depth log analysis, we imported our logs into a SQL database.

 

[KentState]

yes, you are WAY off. And I'm guessing they do not dump logs to a .txt file. And I am sure that nobody sits there and "reviews" them manually.

Amazon gets upwards of 200 Million visitors per month or something like that (more than double eBay). They do nothing like you do.

While I don't know the internal infrastructure of Amazon, I assume they do some type of HDFS store of logs and use their MR expertise to parse and analyze the data. I know someone on the inside that does web development so I may ask for some general idea, though I doubt he can share.

 

[Jaskalas]

Manually review logs?
Oh, hell no!

You have a program run statistical analysis on the log files. You review the stats.
If there's a specific matter to address, you run a search and pull out lines with key words in them.

 

[vi edit]

Anyone have freeware suggestions for text editors to review large log files? A 150 meg HL7 log file can take notepad like 10 minutes to churn and open. I tried Ultra edit for a while and it was decent then the trial expired. Anything free that doesn't suck or riddled with malware?

SMS Trace is awesome for reviewing small log files. But it can't do big stuff.

 

[NikolaeVarius]

Are you kidding me? At my company (Fairly well known internet site) we have servers that generate 15mbps of pure log data by themselves. I think EXTREMELY conservatively, we generate at least 10 TB of pure log files every single day, post compression.

A small part of my job is to make that data usable. A majority of the data is kept for about 2 years due to PCI compliance issues, majority of the rest of the data is kept for a year for long term archival. A small portion is then kept for 2 weeks at a time for short term analytics. Our 2 week archival cluster is at about 70TB of logs for those 14 days. uncompressed.

 

[lxskllr]

Never dealt with logs like that, but notepad++ always worked well for me.

 

[NikolaeVarius]

Anyone have freeware suggestions for text editors to review large log files? A 150 meg HL7 log file can take notepad like 10 minutes to churn and open. I tried Ultra edit for a while and it was decent then the trial expired. Anything free that doesn't suck or riddled with malware?

SMS Trace is awesome for reviewing small log files. But it can't do big stuff.

Unix + less or upload it to Splunk, or elasticsearch, or hdfs.

 

[brianmanahan]

Unix + less or upload it to Splunk, or elasticsearch, or hdfs.

yeah, i assume they'd use their own custom variant of a splunk-like system