Why Is "Reliable" TCP So Unrealiable

[ViRGE]

Ok, so this one is short and sweet: TCP is a reliable protocol, so all the data transmitted with it should end up on the other end correctly, right? So why is it that we still need PAR(Parity) files and recovery records and checksums for downloaded files, especially large ones? Most people that have used Newsgroups for example have experienced a download where they needed a parity file to fix it, but if TCP is reliable, why is it broken in the first place?

 

[Matthias99]

Originally posted by: ViRGE
Ok, so this one is short and sweet: TCP is a reliable protocol, so all the data transmitted with it should end up on the other end correctly, right? So why is it that we still need PAR(Parity) files and recovery records and checksums for downloaded files, especially large ones? Most people that have used Newsgroups for example have experienced a download where they needed a parity file to fix it, but if TCP is reliable, why is it broken in the first place?

TCP is 'reliable', but only up to a point, and only if all the hardware involved is working properly. If the parity calculations at any point between you and the server are broken, and a transmission error occurs at that hop, you'll get a corrupted file. It's also possible (although very unlikely) that a packet could be corrupted in a way that the parity calculations built into TCP/IP cannot detect. Any error-detection mechanism that is transmitted along with the data cannot be 100% reliable, since you're reliant on the same hardware being used to transmit the data.

That said, I can't remember the last time I downloaded a file via TCP/IP and found it to be corrupted. Most often, the problem is on the receiving end (a system that is slightly unstable, or a bad NIC, that sort of thing).

 

[SmoiL]

In regards to newsgroups I generally find most of the incomplete file errors are due to the newsgroup provider dropping messages.

 

[VirtualLarry]

Originally posted by: SmoiL
In regards to newsgroups I generally find most of the incomplete file errors are due to the newsgroup provider dropping messages.

Ding! We have a winner. Also, I sometimes use RAR + PAR for archiving data on CD/DVD that I really care about. That way, if the disc degrades, and one of the file segments is unreadable, I can use the PAR to re-construct them.

But in general, when you have a data-transmission, protected by a checksum of N bits, then there is a 1/(2^N) chance that the contents could get corrupted en-route, and result in an identical checksum. Kind of like how a stopped clock is still right, twice a day. (Speaking of analog clocks with dials, here.)

 

[Aves]

Originally posted by: SmoiL
In regards to newsgroups I generally find most of the incomplete file errors are due to the newsgroup provider dropping messages.

Exactly. TCP isn't the problem here.

 

[uOpt]

In failed downloads the data as actually went through TCP is fine.

The most common problem is truncated files due to the server aborting or timing out a TCP connection.

The next class of errors are truncated and corrupted files, which are usually caused by the TCP client screwing up on a permaturly terminated connection and writing unitialized buffers, or writes a previous buffer again etc.

The rest of file broken on the server's harddrive already.

I don't think I have ever seen a corrupted, not truncated in transfer, file which wouldn't be the same corrupted file on retransmit.

 

[glugglug]

TCP checksum is only 16 bits. So theoretically, one out of every 65,536 errors will make it through. I believe the IP header has a checksum as well though, so the chance of the error getting through BOTH checksums is more slim.

As others have said though, most bad downloads are from incomplete transfers (the end of the file was never received due to connection loss), not from data errors getting through.