Moving Large Amounts of Data Between Computers on Network

[UpstartXT]

Hey there,

Not sure if this is the right section for this but here it goes. I record myself playing computer games with a program called FRAPS. It basically creates a 4GB .avi file for every 30 seconds or so of footage. I routinely can get up to 600GB of footage on my gaming computer.

I then would like to transfer this to my workstation where I do editing on said files. The trouble is how to move large data like that quickly. I have tried making a public shared folder on my workstation, but whenever I copy and paste to that I get these weird errors about file transfer failed for seemingly no reason. I can't figure that out.

I have a Windows Home Server and that's currently how I am doing it: I transfer from the gaming computer to the WHS, which takes a long time, then transfer from the WHS to the workstation, which takes an extremely long time.

Is there a faster way to do this, a solution to my shared folder problem, or at least a way to batch compress these videos? I have tried using "Super" to batch compress but I can't get it to work, and most other programs like VDUB would take way too long to manually do it one at a time since you can't batch compress things.

 

[denis280]

Did you try winrar or winzip.

 

[UpstartXT]

Yeah but to zip up 600GB of data would take half a day

 

[mindless1]

I assume you are creating uncompressed avi video because the CPU overhead would be too great while gaming, which it would be in many cases.

4GB or 4096MB/30 seconds is 137MB/s, significantly too high for GbE networking. It may even be higher than the receiving disk subsystem on your networked workstation can handle. The solution is a newish HDD with a direct SATA2 or higher transfer speed. For convenience, put the HDD in an external enclosure so you can walk it from system to system for editing, OR wait until after the gaming is finished to manually transfer the data from one to the other over the network.

The other option if you have more than a couple CPU cores and they have sufficient idle time during gaming, is find some other video screen capture software that is capable of compressing it on the fly... it need not compress it all that much, a meager 3:1 compression ratio might allow realtime network transfers over GbE if both systems are otherwise up to the task, but note also that the gaming system may still suffer a performance hit in gaming while transferring at that rate, especially if using a budget grade card or typical motherboard integrated network adapter.

 

[wsaenotsock]

For convenience, put the HDD in an external enclosure so you can walk it from system to system for editing, OR wait until after the gaming is finished to manually transfer the data from one to the other over the network.

pretty much this.

And for transferring the data over the network, there are programs that do a littl e better job at this than Windows.

Try TeraCopy, it has a pause/resume/skip & error checking features and better network throughput than windows does.

 

[VirtualLarry]

Use an external eSATA 2TB HD.

 

[UpstartXT]

Wow thanks for all the replies guys. I think I am definitely going to give that tera copy program a try.

@mindless1 I already have a 2TB SATA 2 RAID 0 array that I record to inside the gaming computer. I was talking about how to copy it from that computer to my workstation computer.

 

[mindless1]

^ in that case, try Teracopy or google for other file copying programs.

However with the amount of data you are processing, then editing, what I propose you do is use a program like VirtualDub (which can do batches) to compress the source, uncompressed video down some with a lossless codec to retain max quality for editing purposes, writing the compressed VD output file directly to the WHS, or faster still directly to the workstation.

It would take some testing to determine if a low compression rate will help, or harm, total transfer time going from gaming box to WHS box, but certainly the transfer from WHS to workstation will be multiple times faster and the storage more manageable due to smaller sized files.

 

[UpstartXT]

There's no doubt doing something like h.264 compression to these videos would help tremendously, and while familiar with VDub, I'm not sure how to do batches, I thought you could only do one video at a time (which, sense the videos are broken into 30 second segments would take forever since I would have to sit there at the computer manually going through them)

 

[heymrdj]

I batch convert with SUPER when I want to really turn things out fast with great quality. It's awesome stuff.

http://www.erightsoft.com/SUPER.html

 

[mindless1]

No, I mean lossless compression. Yes there is a learning curve but since you don't yet have a working solution this is true with any software you pick.

The easiest way to get started doing batches in VDub is load "a" file, set the processing settings you want for it, then do FILE-> Save Processing Settings, to save that for future use.

Next, you just click save file (FILE-> Save As AVI) as you normally would, but at the bottom of the Save dialog box, checkmark the option "Don't run this job now; add to job control so I can run it in batch mode". Next drag another video and again "Save as AVI" with the box ticked. There may be an even more automated way, but considering how long it takes to transfer 600GB of footage, taking a few seconds to drag and <ALT-F><a> keystroke to save each is minimal time spent.

Yes it would take awhile, but if you tick the option to "autostart" in the VDub FILE-> Job Control window, your network transfers are running right after the first one and you'd be working faster than the transfers went.

... I'd still consider the other option, finding some screencap util that saves in a lossless compressed format in realtime. VDub will apparently do that too, but I don't know whether it or something else has the lowest learning curve but here is one guide for Vdub to do that:
http://www.genadmission.com/vdubguide.html

 

[UpstartXT]

I batch convert with SUPER when I want to really turn things out fast with great quality. It's awesome stuff.

http://www.erightsoft.com/SUPER.html

I can't figure out how the hell to get SUPER to work, it always gives me a compression failed mesage

 

[MarkLuvsCS]

Just a suggestion, have you checked out Msi afterburner for ingame recording. It transcodes the video in a high quality but reasonably sized format. The new afterburner can record audio now too. I never understood why fraps hasn't truly changed in the last 8 years

Sent from my MB860 using Tapatalk

*EDIT* just checked the video I recorded before and 3:20 video recorded took 5.4gb recording 1920x1080 @ 30 fps. I did have a problem with the video format of RTV1, but its default MJPG compression worked fine. virtual dub is unable to work with video format because of some error with not being in multiples of 16. Handbrake worked to re-encode into an m4v file. There was one older program that allowed you to use pretty much any format for transcoding like the same options you find under the Compression / codec tab in Virtual Dub. It was called Taski Video Capture. Unfortunately, the dev stopped working on it years ago so I don't think it can run in any DX10/DX11 games. It was a highly customizable software that sadly didn't get the respect it deserved. Although a years ago, real time transcoding while gaming was not typically viable for most systems.

 

[Atheus]

As a well known hacker once said - never underestimate the bandwidth of a truck full of DVDs...