Copying massive amounts of data HD-->HD

[Muse]

I have a new NAS (Synology DS214play), my first, with two WD Red 3TB RAID1 configured HDs installed. Up until today, I've had my networked data on a 2TB USB HD hanging off one of my laptops, which is ethernet connected to my wireless N router. Bad environment, no redundancy, very iffy inconsistent connectivity, ergo the NAS.

On the 2TB USB HD there's around 380GB of data. So, last night I selected all the folders I want to copy to the NAS and in Windows (XP is on the "server" laptop), I did a Copy to the NAS. Of course, it took a long time, but after copying around 65GB of data there was an error, something like this:

File or folder Tri5 is corrupt and can't be copied.

I clicked OK, and that was it, the rest of the data wasn't copied over. I know, Copy is a poor command for this sort of thing. What should I do and how? Piecemeal will sort of get it, I guess, but there must be a better way. It sucks that Windows didn't tell me shit about the problem, where it was. I don't recognize the file or folder designation.

Thanks for suggestions, etc.

 

[Dahak]

could use something like teracopy or robocopy to do the copy which will allow you to continue if there is an error and then look at the log to see what folder/file the error is located at.

 

[Chipfiref]

Sounds like you understand the issues pretty well. Unfortunately you should probably go through and copy one folder at a time if you want to see what is going on. Could be a lot of things, but I would bet on file system corruption on the USB drive. Its hard for Windows to always be in control of a USB drive hanging off a laptop I agree Robocopy has decent logging. When you find the folder(s) where there are issues you can dig deeper.

A NAS on your LAN with a RAID array brings the level of complexity up a bit and you don't want to introduce corruption to your RAID on your NAS so you should do a file system check/repair on the disks you want to copy from, after you get everything of value off as best you can. schedule a boot-time repair in the checkdisk tool to get a good check. You could have file corruption, or even physical disk issues. USB drives are, as you have alluded to, subject to corruption issues. So you may need to run the vendor utility to check that drive.

 

[Muse]

In the course of doing more work in moving over the data today I stumbled across files whose names nearly or do coincide with the error message I saw. Those files are associated with a program I have a license for, so I emailed their support (High Criteria software, the program being Total Recorder SE). Here's the post I sent them:
- - - -
I have a problem in my data. I have a separate directory for temporary files named \MP3 Temp

I have deleted all but 7 of the files in the directory, but the rest cannot be deleted or moved. All of the remaining files are 0 kb. Three of them are .mp3, the other four are .tmp. They have file names such as ~TRr2.mp3 or ~TRr5.tmp.

When I attempt to delete any of the files I get a message such as this:

--------------------------------------------------------------------------------
Error Deleting File or Folder
--------------------------------------------------------------------------------

X ......... Cannot delete ~TRi5: The file or directory is corrupted and unreadable.

[OK]

--------------------------------------------------------------------------------

Do you know how I can deal with this? I'm unable to delete the temporary directory. Thank you for help.
- - - -

Actually, I think they may have no take on it and it really doesn't matter, I suppose. The files in question won't copy over. As you say, Chipfiref, I can move stuff over, and when I'm satisfied I have what I want I can run chkdsk/ r on the HD and presumably it will be fixed. I don't know why those files aren't deletable, there were several other similar files in the folder which I could delete.

 

[Charlie98]

Can you use something like Search Everything to locate the folder and delete it?

Every time I try to move TB's of data something always goes wrong...

 

[grimpr]

Try a good sync utility like GoodSync to move massive amounts, they always work better.

 

[imagoon]

robocopy c:\thesource d:\thedestination *.* /s /e /MIR /tee /log:c:\thelogfile.txt /r:1 /w:1 /copyall

is pretty much all you ever need. Make sure to run it as an admin command prompt. It will do the copy, retry a file once then spit it out to a log file that you can scan for the error in. No need for extra apps etc.

As for corrupted files there isn't much you really can do about that other than hope chkdsk repairs them.

 

[code65536]

The problem with robocopy--and why I never waste my time with it when I need to salvage data from a dying disk--is that its fault tolerance is at the file level, not at the sector level.

For example, say I have a large, 10GB rar file containing thousands of smaller files. Let's say this file has a single bad sector. Robocopy will fail to copy the entire file when the desired behavior is to copy the good parts of the file and just fill in that bad sector with zeros.

And if this rar file has some Reed-Solomon recovery (or if I added external recovery by way of some PAR2 files), I'd need to copy as much of the broken file as possible in order to work on it and perform the recovery (it's not a good idea to do it in-place on the bad drive). But with robocopy (and many other utilities), it's all-or-nothing at the file level.

 

[beginner99]

SyncToy. Free tool from MS. I use it to backup to my external hdd (4 TB). But it can be also used to copy over to the NAS and in the future to backup the nas.

 

[Captain_WD]

Hey there Muse,

What I would try is downloading and running WD Data Lifeguard Diagnostic on the portable drive to check if there are any issues. This shouldn't damage any files on the drive so you can do it before copying everything from it to the NAS device. I've put a link at the bottom.
If an error with a bad sector pops up or anything that is threatening your data, I would try a backup software to automatically copy everything you need from the portable hard drive to the NAS. WD SmartWare is a good program to use for this. I lets you choose which files and folders to backup so you can avoid anything that's presumably corrupted.
After you're done with this you can try running chkdsk/ r to temporary repair any bad sectors or any other damage.

WD Data Lifeguard Diagnostic: http://support.wdc.com/product/download.asp?groupid=810&lang=en
WD SmartWare: http://www.wdc.com/en/products/products.aspx?id=940

Hope this helps you. Please, keep me posted about how your problem is developing,

Captain_WD

 

[imagoon]

The problem with robocopy--and why I never waste my time with it when I need to salvage data from a dying disk--is that its fault tolerance is at the file level, not at the sector level.

For example, say I have a large, 10GB rar file containing thousands of smaller files. Let's say this file has a single bad sector. Robocopy will fail to copy the entire file when the desired behavior is to copy the good parts of the file and just fill in that bad sector with zeros.

And if this rar file has some Reed-Solomon recovery (or if I added external recovery by way of some PAR2 files), I'd need to copy as much of the broken file as possible in order to work on it and perform the recovery (it's not a good idea to do it in-place on the bad drive). But with robocopy (and many other utilities), it's all-or-nothing at the file level.

It does help to use the correct tool for the job. On the flip side you should generally not be using a sector recovery tool copy an HDD that is in good working order..

IE you seem to have discarded a tool because you were using it incorrectly, not because the tool is faulty.

If sectors are truly lost then I think robocopy won't help you. If the drive is intermittent, /r and /w will help a lot there.

 

[code65536]

It does help to use the correct tool for the job. On the flip side you should generally not be using a sector recovery tool copy an HDD that is in good working order..

IE you seem to have discarded a tool because you were using it incorrectly, not because the tool is faulty.

If sectors are truly lost then I think robocopy won't help you. If the drive is intermittent, /r and /w will help a lot there.

But what is robocopy good for? The only thing that drew me to trying it was its claims of being able to continue after an error, which I quickly found out was a pretty useless file-level feature. I have never encountered an error like that on a healthy disk. The only time I get copy errors is when the disk is unhealthy or if there has been an random error on a healthy disk (stray bit flip, head flux, etc.) that affects a file that I want recovered--both cases calls for a proper tool capable of sector salvage, not robocopy. So that feature of skipping errors at a file level seems to fit a very narrow niche that I don't yet see a use for.

As for its other features, I've never found them to be that useful, and prefer vcopy instead (which preserves creation/access timestamps, directory timestamps, compression status, and generates md5s... which is about all I ever need) because of the simpler (and thus quicker to type) syntax.

 

[imagoon]

But what is robocopy good for?

Well rather than sending you to a "lmgtfy.com" link...

Some small examples:

straight copy.
copy an entire directory tree
set copy retry and wait periods, handy if the files are open and in use
easy to read log of results if needed
copy all metadata like ACL create dates, modify dates
multithreaded copies
rsync like copies
one way mirroring

built in to vista+, available for xp and older

just to name about 5% of what it can do.

 

[Blue_Max]

Don't know anything about error correction, etc, but here's another nod to TeraCopy for being a fantastic utility. It copies my SID/MIDI/MOD music collection of 80,000 TINY files in far, far less time than Windows' own system would. Smarter system for when things run out of space, too.

 

[code65536]

Well rather than sending you to a "lmgtfy.com" link...

Some small examples:

straight copy.
copy an entire directory tree
set copy retry and wait periods, handy if the files are open and in use
easy to read log of results if needed
copy all metadata like ACL create dates, modify dates
multithreaded copies
rsync like copies
one way mirroring

built in to vista+, available for xp and older

just to name about 5% of what it can do.

Google? Meh, that's what robocopy /? is for.

What I'm saying is that robocopy's exclusive features are something that I rarely use, and more common features (like straight copy or tree copy) are better-implemented by many other utilities. Robocopy also has an unwieldy, long syntax. And robust, but royally-painful-to-use selection mechanisms.

And given the context of this thread, where there is corruption on the disk, I've found robocopy's handling of corruption to... be less than robust.

It's not that robocopy doesn't have its uses (I've found legitimate uses for it... like twice), but it's not the end-all-be-all savior of file copy.

 

[piasabird]

Might try defragging first.

 

[Muse]

Might try defragging first.

Yeah, I'm not sure, I'm going to weigh my options. I thought I was in the clear for the most part because I'd copied all the data that mattered over to the NAS. However, an issue has come up on the NAS and it lost ~1/2 the data when I did a cut/paste within it. Nutty, and I posted a thread at Synology Forums. However, those forums appear to be pretty weak in terms of garnering support. Well, I still have the data on the USB HD, so I can recopy it, this time to the preferred location. I can only keep my fingers crossed that such a thing doesn't happen again. Meantime, I'm leaving the data on the USB HD and am going to make a copy of the NAS data ASAP! I thought that Synology's DSM was a seasoned product, but now I have serious doubts.

 

[imagoon]

Google? Meh, that's what robocopy /? is for.

What I'm saying is that robocopy's exclusive features are something that I rarely use, and more common features (like straight copy or tree copy) are better-implemented by many other utilities. Robocopy also has an unwieldy, long syntax. And robust, but royally-painful-to-use selection mechanisms.

And given the context of this thread, where there is corruption on the disk, I've found robocopy's handling of corruption to... be less than robust.

It's not that robocopy doesn't have its uses (I've found legitimate uses for it... like twice), but it's not the end-all-be-all savior of file copy.

That's fine, however I have found the opposite. Once I took 5 minutes to learn the syntax, I've found legitimate uses for other applications... like twice.

Never thought I would hear a coder mentioned syntax either. Most coders love to live in the arcane.

 

[imagoon]

Yeah, I'm not sure, I'm going to weigh my options. I thought I was in the clear for the most part because I'd copied all the data that mattered over to the NAS. However, an issue has come up on the NAS and it lost ~1/2 the data when I did a cut/paste within it. Nutty, and I posted a thread at Synology Forums. However, those forums appear to be pretty weak in terms of garnering support. Well, I still have the data on the USB HD, so I can recopy it, this time to the preferred location. I can only keep my fingers crossed that such a thing doesn't happen again. Meantime, I'm leaving the data on the USB HD and am going to make a copy of the NAS data ASAP! I thought that Synology's DSM was a seasoned product, but now I have serious doubts.

We use some Synology units for backup targets. DSM 5+ is solid. We romp the crap out of them with large backups and then deduping those. They rarely see less than 50MB/s random IO from the deduplication for years. You may have a disk fault. Do you have scrubbing on? If not you may have never detected the fault prior.

 

[Muse]

We use some Synology units for backup targets. DSM 5+ is solid. We romp the crap out of them with large backups and then deduping those. They rarely see less than 50MB/s random IO from the deduplication for years. You may have a disk fault. Do you have scrubbing on? If not you may have never detected the fault prior.

I don't know, everything was default, so whatever the default is. DSM reports that everything is "Good" but who knows? I don't know how to troubleshoot it. AFAIK, the fact that there are WD drives in there is opaque, so I don't know if I can run WD Data Lifeguard services.

 

[imagoon]

Well "good" indicates that the drives are not giving you a SMART error nor has the system detected a read fault (which is odd to me unless it is purely a corrupted filesystem.)

How are you accessing these files? Via the CIFS connector?

Can you see the files in the file station?

 

[Muse]

Well "good" indicates that the drives are not giving you a SMART error nor has the system detected a read fault (which is odd to me unless it is purely a corrupted filesystem.)

How are you accessing these files? Via the CIFS connector?

Can you see the files in the file station?

Can't see the files in File Station or Windows Explorer, nor the subdirectories. Vanished! There's just the top level folder and the one file in it, which had the most recent creation/modified date, which was at 1:04AM this morning.

What's the CIFS conntector? Part of DSM?

I just talked to Synology Support, Leo. He sounded weary, I must say, burnt out, but he listened and responded. He said the stuff disappearing was weird, he'd never heard of that happening. He suggested using Windows Explorer to do things such as file transfers, not File Station, says it's definitely the way to do it on the diskstations.

 

[imagoon]

CIFS is the "windows shares" part. It is sounding more and more like the system did a fsck internally and was purging the corrupt stuff. I am not sure that is good though.

 

[Muse]

Hey there Muse,

What I would try is downloading and running WD Data Lifeguard Diagnostic on the portable drive to check if there are any issues. This shouldn't damage any files on the drive so you can do it before copying everything from it to the NAS device. I've put a link at the bottom.
If an error with a bad sector pops up or anything that is threatening your data, I would try a backup software to automatically copy everything you need from the portable hard drive to the NAS. WD SmartWare is a good program to use for this. I lets you choose which files and folders to backup so you can avoid anything that's presumably corrupted.
After you're done with this you can try running chkdsk/ r to temporary repair any bad sectors or any other damage.

WD Data Lifeguard Diagnostic: http://support.wdc.com/product/download.asp?groupid=810&lang=en
WD SmartWare: http://www.wdc.com/en/products/products.aspx?id=940

Hope this helps you. Please, keep me posted about how your problem is developing,

Captain_WD

I already have WDDLD installed on the laptop that the WD 2TB ELEMENTS HD is attached to. Passes Quick Test, I just started the extended test, figure that's what you meant. I hope you're right and that no data will be lost. I want to copy over the ~166GB, some 2000+ MP3 files that were lost this morning. I don't have a backup of those files, so fingers crossed here. As I stated in the previous post, a Synology support rep just told me that he'd never heard of this sort of thing happening. I wonder if one or both of the WD 3TB Red HDs in the NAS has problems... Is there a way I can run WDLGD on those?

 

[imagoon]

Muse, you would need to take the drives out of the NAS and run the WDLD tool on the disk via a PC.