Question Long term storage on HDD

[pcm81]

I have a raid 6 NAS and 3 16 TB drives in my computer. Let's call the drives Data1, Data1-BACK-UP and Data1-back-up. So basically 3 back-ups of Data1, counting the NAS.
Some files do not get accessed on these drives for years. Most of the time when i synch to the back-up i just use file name + date for matching and update. Once in a while i do full compare. The problem is: 99.9% of my file operations are read, not write so i am starting to worry about bit rot and bad sectors creeping into the files that do not get regularly updated (written to).

One option i am considering is to reformat my back-up drive and rewrite all data to it to essentially update its magnetic fields on all bits. Is there a better way to check/update file integrity on linux? Since i have 4 copies of data a program that checks all 4 and overwrites 1, completely updating all data, with data that 2 of remaining 3 agree on would be ideal. Is there something like that? The 3 drives in my machine are 2x ext4 and 1x fuse (not sure how i ended up with this fs) file systems the NAS is raid-6.

 

[Tech Junky]

Rsync can did that in a cron job.
Just setup a separate cron for each destination and configure the options as wanted. I have one set to backup my primary to a secondary every 6 hours with flags for diff only. I also use the exclude folders option for stuff that's not important for recovery. I have a second cron set to my storage drive as well.

 

[manly]

ZFS? The following article is already 10 years old, but still feels like a good read (AFAIK Btrfs is still not ready for production).

Bitrot and atomic COWs: Inside “next-gen” filesystems

We look at the amazing features in ZFS and btrfs—and why you need them.

arstechnica.com

 

[mikeymikec]

btrfs - a customer of mine who gave me their old NAS drives had them set up as btrfs and the guy isn't particularly techie, I assume they simply accepted the defaults that their NAS wanted to use... it seems unlikely that a commercial product would use a pre-release filesystem, no?

 

[manly]

Synology DSM offers btrfs, but they expressly avoid its "RAID" features.

What is the RAID implementation for Btrfs File System on Synology NAS? - Synology Knowledge Center

Synology Knowledge Center offers comprehensive support, providing answers to frequently asked questions, troubleshooting steps, software tutorials, and all the technical documentation you may need.

kb.synology.com

I like reading Jim Salter's *nix coverage; here's his 2021 update:

Examining btrfs, Linux’s perpetually half-finished filesystem

This btrfs filesystem overview highlights some longstanding shortcomings.

arstechnica.com

 

[pcm81]

I looked up Rsync, it looks like it is a back-up tool similar to FreeFileSync that i use now. Can it actually refresh source file which is different on content level from 2 other sources that are in agreement?

 

[Tech Junky]

Depends on the flags you use for the diff calculation. Also, if you string together commands to be run concurantly when triggered. It can look for the time as an option, size, etc.

After the first sync it handles updates fairly quickly. For the 3rd backup you could have a format trigger and then on the next line say 5m later have it trigger the full backup for the bit rot potential.

It's been awhile since I've looked at the options but, there might even be a metadata check flag you could use. It can be as simple or complicated as you desire.

 

[manly]

If you're concerned about bitrot, how would you know an original file didn't have a bit flipped unless you're running ZFS?

rsync is a fabulous tool, but it isn't for everybody. I don't use it, but I've read good things about restic as a powerful backup solution.

Beats reinventing the wheel/rolling your own solution.

 

[pcm81]

If you're concerned about bitrot, how would you know an original file didn't have a bit flipped unless you're running ZFS?

I have 4 copies of the data, the thought is if i have bitrot in the one being updated and even in one of the 3 copies the 2 other copies will correct the drive being updated.

 

[bigboxes]

I have a raid 6 NAS and 3 16 TB drives in my computer. Let's call the drives Data1, Data1-BACK-UP and Data1-back-up. So basically 3 back-ups of Data1, counting the NAS.
Some files do not get accessed on these drives for years. Most of the time when i synch to the back-up i just use file name + date for matching and update. Once in a while i do full compare. The problem is: 99.9% of my file operations are read, not write so i am starting to worry about bit rot and bad sectors creeping into the files that do not get regularly updated (written to).

One option i am considering is to reformat my back-up drive and rewrite all data to it to essentially update its magnetic fields on all bits. Is there a better way to check/update file integrity on linux? Since i have 4 copies of data a program that checks all 4 and overwrites 1, completely updating all data, with data that 2 of remaining 3 agree on would be ideal. Is there something like that? The 3 drives in my machine are 2x ext4 and 1x fuse (not sure how i ended up with this fs) file systems the NAS is raid-6.

My understanding is that data is fairly safe on HDDs for about 10 years before bit rot/corruption might become an issue. Your NAS is NOT backup. It's your data. I'm not sure how many backups you have. You want one set of your backup as cold storage (i.e. not powered on) and a third set of your data offsitte (to mitigate flooding/fire/theft/physical damage.