How do large companies capture, store, and validate their websites?

[SarcasticDwarf]

This came up in a class this evening. How do large companies with highly dynamic websites capture, store, and validate their websites?

I ask because we were discussing how just the "normal" backup method does not necessarilyaccount for the minute to minute changes in a website.

 

[FoBoT]

dynamic websites are just databases, or many databases working together
databases by their nature are dynamic
not sure what you are asking

 

[SarcasticDwarf]

Originally posted by: FoBoT
dynamic websites are just databases, or many databases working together
databases by their nature are dynamic
not sure what you are asking

Let's take an example:

You have a website with millions of products held within a database. Now for whatever reason (lawsuit, federal investigation, whatever) you need to produce a RECORD from your website. That record is a page on your website.

How do you prove what example.html looked like at exactly 19:27:54?

 

[KingGheedora]

Data that changes minute to minute is usually in databases. Companies run those on redundant drive arrays, and also employ clustering and/or replication, where a separate server or servers mirror the data on the active ("live") server. These can act as warm spares, where in more sophisticated setups the warm spare can kick in almost immediately after the main server experiences failure.

Replication is also used to keep fairly up to date redundant copies on other database servers.

Nightly and weekly, and monthly backups are also kept.

 

[drinkmorejava]

Just to take at guess at one of the many ways to do. At 19:27:54, have the master DB server stop communicating with a slave DB server and backup the slave.

 

[FoBoT]

history

the tables in the database have to include history if the data is changing and needs to be kept over time

so what ever information needs to be retrieved is pulled from the database as it existed at 19:27:54 on whatever day is in question

 

[hellman69]

Most websites like that are dynamic, driven by a database. For example, take Ebay. You see millions of listings, but they only have a few actual web pages (files that need to be backed up) that display that information. They query the database and display all those millions of pages. So for backup, the actual web page is probably changed, at most, once a week. That's pretty easy to backup. As for the database server, which is important, here is the typical backup plan. Do a full backup once a day. During the day, store and backup the change log to it. The change log can be applied to the full backup to re-create the changes for the day.

Disclaimer: I program a site similar to Ebay. We have about 200 distinct web page files that we backup daily. We have over 30,000 listings that change by the hour. The result is that, as a user, you will see 30,000+ different web pages that are changing minute by minute, but it is really an illusion. This is how we backup our website. Worst case, we will lose the last 5 minutes of changes before a disaster.

Trevor

 

[KingGheedora]

Originally posted by: SarcasticDwarf

Originally posted by: FoBoT
dynamic websites are just databases, or many databases working together
databases by their nature are dynamic
not sure what you are asking

Let's take an example:

You have a website with millions of products held within a database. Now for whatever reason (lawsuit, federal investigation, whatever) you need to produce a RECORD from your website. That record is a page on your website.

How do you prove what example.html looked like at exactly 19:27:54?

Most companies cannot do this in as much detail as you describe. They'd need to track changes to all records (some companies might do this).

Most companies do keep historical versions of some of their data. The mechanism for doing this though is to just keep a version of each record with timestamp indicators of when the record became active and when it became invalid (changed).

This is one aspect of data warehousing. But data warehousing is not necessarily the only reason people would version their data.

 

[Safeway]

Originally posted by: FoBoT
history

the tables in the database have to include history if the data is changing and needs to be kept over time

so what ever information needs to be retrieved is pulled from the database as it existed at 19:27:54 on whatever day is in question

Yup, like Wikipedia. You can see exactly what it looked like at a given time on a given date.

 

[Safeway]

Originally posted by: KingGheedora

Originally posted by: SarcasticDwarf

Originally posted by: FoBoT
dynamic websites are just databases, or many databases working together
databases by their nature are dynamic
not sure what you are asking

Let's take an example:

You have a website with millions of products held within a database. Now for whatever reason (lawsuit, federal investigation, whatever) you need to produce a RECORD from your website. That record is a page on your website.

How do you prove what example.html looked like at exactly 19:27:54?

Most companies cannot do this in as much detail as you describe. They'd need to track changes to all records (some companies might do this).

Most companies do keep historical versions of some of their data. The mechanism for doing this though is to just keep a version of each record with timestamp indicators of when the record became active and when it became invalid (changed).

Tracking changes is easier than tracking multiple versions of the same content.

 

[KingGheedora]

Originally posted by: Safeway

Originally posted by: KingGheedora

Originally posted by: SarcasticDwarf

Originally posted by: FoBoT
dynamic websites are just databases, or many databases working together
databases by their nature are dynamic
not sure what you are asking

Let's take an example:

You have a website with millions of products held within a database. Now for whatever reason (lawsuit, federal investigation, whatever) you need to produce a RECORD from your website. That record is a page on your website.

How do you prove what example.html looked like at exactly 19:27:54?

Most companies cannot do this in as much detail as you describe. They'd need to track changes to all records (some companies might do this).

Most companies do keep historical versions of some of their data. The mechanism for doing this though is to just keep a version of each record with timestamp indicators of when the record became active and when it became invalid (changed).

Tracking changes is easier than tracking multiple versions of the same content.

Do you mean with text? I'm not sure it's that easy if you are tracking database records (product ID's, product prices, customer info, etc) rather than "content", like wikipedia pages. It's seems like it'd be easier to keep copies of each version of each record (and i know that people do this) than come up with some way of tracking the changes only, not sure how that would be done.