bit rot

[drag]

One thing I have always been wondering about since I've started downloading all my OS's over the internet is what is the rate of destroyed or corrupted bits in a 650 meg download? I know that tcp/ip protocal allows to error correction and it resends packets that do not meet the pariaty bits.. But that can't stop everything. How much does digital information degrade from being compiled on someone's computer copy/moved, tar'd, zip'd, uploaded, downloaded , compressed on a cd, tested, dd'd into a iso file, uploaded to ftp, downloaded thru a hundred or so assorted routers and nodes of different speeds and technology, copied to my thru my network into my computer onto my harddrive, then burned to a cd, handled, get pop spilled on it, cleaned, get drop kicked by the cat, and finally installed on a freshly formated harddrive? Why the hell does this even work at all?

And how well does a modern harddrive store information, at what rate does the bits degrade on the surface of the platters? How long can you hope to store information reliably on a harddrive, or tape medium, or a profesional cd/dvd, a burned one?

What I don't understand is when a program gets some of it's bits radomly shuffled, why does it still work ok?

What are the weaknesses of checksumming? If one bit goes from 0 to 1 and another goes from 0 to 1, don't these cancel each others out. So am I right in thinking that we just realy on the law of statistics to gaurd our information? (Which is why I suppose backups are a nessessity)

 

[AndyHui]

You should check out Cyclic Redundancy Checking. CRCs are actually very good.

 

[Shalmanese]

Hmm... CD's die all the time, its part of the cycle of life. Hard drives, you dont expect to die too often. And when they do, its usually pretty obvious as the entire HD becomes useless. Error checking sums are actually very good. Especially with dedicated data lines instead of dirty analog lines.

 

[dejitaru]

So how many bytes are corrupted? One in a million?

 

[CTho9305]

Originally posted by: dejitaru
So how many bytes are corrupted? One in a million?

Before or after correction? After correction, WAY fewer, or CD ISOs would hardly work at all.

edit: MD5 sums do much more than count 1s and 0s. In fact, it isn't really possible to come up with 2 different things that have the same MD5 sum in a given amount of time. If the MD5 of your file matches the MD5 of the original, you can be sure it is the right file, with no errors.

 

[dejitaru]

What is MD5? If MD5 is so great, why not just restore the entire file from the MD5?

 

[CTho9305]

Originally posted by: dejitaru
What is MD5? If MD5 is so great, why not just restore the entire file from the MD5?

You can't go backwards with MD5. It is a one-way hash... that is why it is often used for password systems. In a unix environment, your password isn't actually stored anywhere. What is stored is the MD5 of your password. When you try to log in, it hashes your what you typed, and compares it to the hash that is stored. If it matches, then you got the password right.

The idea is that you can't generate two files with the same MD5, but not that no such files exist. Obviously for files longer than the MD5 size, more than one will result in the same hash.

 

[Shalmanese]

The thing is that you need to have a REALLY corrupted file in order to get the same MD5 checksum. Were not talking about 1 or 2 bits so the actual chance of getting a corrupt file with the same MD5 checksum is incredibly remote. However, a MD5 checksum is only 128 bits so you cant re-extract the information out of it .

 

[glugglug]

TCP checksum is 16-bit, not 32 like you would have in most archives.

So the chance of a corrupt packet going unnoticed is 1 in 65536.

So if something was REALLY bad on your network and literally 50% of your packets were getting corrupted and needing a retransmit on average this would make it through the TCP layer every 100MB or so using normal 1.5K packets. Actually the odds are better than this because the IP layer has its own checksum that will catch some of the errors the TCP checksum doesn't, there is error detection built into every modem since 9600bps (and even a some 2400bps modems have v.42), and chances are your PPP connection (or PPTP between the cable/DSL modem & ISP for broadband) has a 32-bit checksum on each packet, which if your connection was really bad and getting 50% errors would let one error through every 6 terabytes or so.

 

[MrDudeMan]

Originally posted by: glugglug
TCP checksum is 16-bit, not 32 like you would have in most archives.

So the chance of a corrupt packet going unnoticed is 1 in 65536.

So if something was REALLY bad on your network and literally 50% of your packets were getting corrupted and needing a retransmit on average this would make it through the TCP layer every 100MB or so using normal 1.5K packets. Actually the odds are better than this because the IP layer has its own checksum that will catch some of the errors the TCP checksum doesn't, there is error detection built into every modem since 9600bps (and even a some 2400bps modems have v.42), and chances are your PPP connection (or PPTP between the cable/DSL modem & ISP for broadband) has a 32-bit checksum on each packet, which if your connection was really bad and getting 50% errors would let one error through every 6 terabytes or so.

those numbers sound a little bogus. 1 bit every 6TBs? if that is true...wow

 

[dzt]

Originally posted by: MrDudeMan

Originally posted by: glugglug
TCP checksum is 16-bit, not 32 like you would have in most archives.

So the chance of a corrupt packet going unnoticed is 1 in 65536.

So if something was REALLY bad on your network and literally 50% of your packets were getting corrupted and needing a retransmit on average this would make it through the TCP layer every 100MB or so using normal 1.5K packets. Actually the odds are better than this because the IP layer has its own checksum that will catch some of the errors the TCP checksum doesn't, there is error detection built into every modem since 9600bps (and even a some 2400bps modems have v.42), and chances are your PPP connection (or PPTP between the cable/DSL modem & ISP for broadband) has a 32-bit checksum on each packet, which if your connection was really bad and getting 50% errors would let one error through every 6 terabytes or so.

those numbers sound a little bogus. 1 bit every 6TBs? if that is true...wow

for one statistical freak every 6TB, I'll give you a bigg "WOW"



computer goes GIGO while people wanna LO

 

[0roo0roo]

use smart par? sweet parity and corrupt file recovery.

 

[drag]

ok, so the checksumming in tcp is pretty rocken, so as long as you check your downloads with the md5 (asuming you have a checksum on a secure server to check it too) you'll be pretty safe thru the internet...

But how about your HD in constant use? How much bit drift is going to happen in a binary file that is accessed 30 or 40 times a day for 3 years? I know the magnet head has very little effect on static content, but is that cumulative? Could this be the cause of having to reformat your drive periodicly, or is that just a build up of user errors and bugs in programs slowly corrupting themselves...

We all heard of the ignorant computer user in a office setting using the side of ther computer to hold up various pictures of there children with refrigurator magnets...

 

[CTho9305]

Maxtor claims 1 in 10^15 bits is misread from a hard drive (diamond max 9).

 

[dejitaru]

It isn't the HD, but your OS screws itself, damaging system files, corrupting data. That's why it's good to backup.

 

[MrDudeMan]

Originally posted by: dzt

Originally posted by: MrDudeMan

Originally posted by: glugglug
TCP checksum is 16-bit, not 32 like you would have in most archives.

So the chance of a corrupt packet going unnoticed is 1 in 65536.

So if something was REALLY bad on your network and literally 50% of your packets were getting corrupted and needing a retransmit on average this would make it through the TCP layer every 100MB or so using normal 1.5K packets. Actually the odds are better than this because the IP layer has its own checksum that will catch some of the errors the TCP checksum doesn't, there is error detection built into every modem since 9600bps (and even a some 2400bps modems have v.42), and chances are your PPP connection (or PPTP between the cable/DSL modem & ISP for broadband) has a 32-bit checksum on each packet, which if your connection was really bad and getting 50% errors would let one error through every 6 terabytes or so.

those numbers sound a little bogus. 1 bit every 6TBs? if that is true...wow

for one statistical freak every 6TB, I'll give you a bigg "WOW"



computer goes GIGO while people wanna LO

wtf did that mean? can someone translate? learn to speak english, and if you are going to speak slang, at least dont be a moron about it.