Verify Data Integrity automatically

[Solotak]

I need a simple system to detect file corruption and bit-rot of both original and backup copies. I'm looking for a windows-based solution so that it can warn me if file corruption is occurring. I'm aware people are fond of zfs and its data scrubbing/integrity abilities, however I'm working with windows 7 ntfs systems. I rather not work with anything that has to do with linux, command lines or anything complicated.

Requirements:

1. Verification must be minimum md5 or higher (sha1 / bit comparison)
   
2. Verify file integrity after transferring files using MD5 or higher.
   
   
3. Schedule various checksum profiles and monitor for failed checksums

a) Schedule verification of original files with its original checksum fortnightly
b) Schedule verification of backups with original checksum monthly
c) Warn me if file integrity has failed checksum of its respective file/directory.
d) Profiles for monitoring checksum. Something like Syncback's profile management but purely for scheduling verifications of different directories (profiles). Some folders may require more frequent verification against data corruption (personal photos) vs other media (music)

Eg. 2 profiles
(Profile 1) - Original Photos [A] directory creates checksum/(snapshot?). Verifies fortnightly for failed checksums.

(Profile 2) - Backup Photos directory is verified with Original Photos [A] checksum. Verifies monthly for failed checksum​

I'm not familiar with snapshots but would that be useful for what I'm after?

I'm not sure how to address modifications made to Original Files. This will send integrity mismatches of its associated [A] checksums. How would I elegantly address this issue?

What tool(s) do I require? What other options are there that are stress-free and easy to setup and maintain?

Tools I have tried:
Beyond Compare - So far the best tool for comparing files directories but not so much for validating original file integrity. It doesn't handle checksums. It uses md5 for comparisons of directories A-B but does not detect data corruption of the original directory [A-A]. That's where checksum comes handy and that's where this program is lacking. Otherwise an almost fit solution for the above requirements.

Teracopy - It's okay for simple file transfers but need md5 (+) verification

Syncback - Good program for making backup profiles BUT it only does (CRC) verification if backup is succesfully transferred. You can't use the program to detect data corruption of the original files or backups.

FileVerifier++ - Good checksum tool that will flag if a file has a modified checksum, but you have to manually browse to the directory each time you open program. No profiles or schedules can be made

Exactfile - Similar to above but not as easy to detect failed checksums

 

[Elixer]

You can use PAR2 to create recovery data (recovery sets) of the original file.
Then you can copy the .par2 and the original file over to where you want the backup.
Then, all you do is run a par2 verification scan on the backup, and it will report if it is corrupted. If it is, it can recover said data as long as you created enough recovery sets for the original data.

You can do a automated verification / recovery via a batch file, and a cron job (Scheduled Task)... while not perfect, and it does require some work on your part, it should get most of what you need done.

 

[razel]

Try WinMerge. Beyond Compare's open source competitor. It will generate md5 checksums and is capable of generating patch files to 'correct' differences.

Honestly though, unless you are looking to test longevity of media, this is all extremely over engineered. By the way, Hello! I've seen this identical post on hardforum.

 

[code65536]

Option 1: Unless you absolutely require NTFS, get Windows 8.1 and use the new ReFS. It does the same kind of automatic checksum-of-data and data integrity checks, except it's Microsoft, and it's for Windows. If you create a mirrored volume with 8.1's Storage Spaces, ReFS will also do automatic data repair.

And really, that should be the only option. Because manually doing this stuff is a PITA and I don't recommend it. But if really do you want to do it manually...

Option 2: PAR2, as already mentioned. Check integrity and can repair using Reed-Solomon redundancy if necessary.

Option 3: Use HashCheck. It can generate checksum manifest files of directory trees that are easy to verify. It's like using PAR2, except without the recovery. And for the price of losing the ability to recover, you avoid all the slowness of PAR2. That author also has a command-line utility called vcopy that can be used to copy directory trees while also generating a MD5 manifest during the copy that can be later verified using HashCheck (the checksums are to stdout, so you'll want to redirect it to a .md5 file if you want to save it). I personally use that a lot when backing up data.

 

[thecoolnessrune]

As others have stated, we have to make sure we're clear on these terms. NTFS is a file system, as is ZFS, and BTRFS. Now if you *must* use NTFS, you *must* use software tools. NTFS is not capable in any fashion of doing these current generation file-system tasks.

If you want simple, and you want Windows, then upgrade to Windows 8.1 (or Windows Server 2012), and use Storage Spaces using the ReFS file system. It does everything you need it to.

If you refuse to use an operating system supporting a modern File-System, then the only other way to do it is with Software programs that require more work, such as WinMerge, and PAR2.