Crashplan: Seed Drive Super Slow

[GWestphal]

I'm backing up 1TB to a seed drive for Crashplan and it's going realllly slow. Even taking into account USB 2 seed drive it seems slow. 20-30Mbps or ~2-3MBps. Do I need to change a setting to make it pick up the pace? Normally if I copy over USB2 I easily get 10x the speed.

 

[dave_the_nerd]

What are you copying?

3MBps wouldn't be unheard of if it was USB 2, there were other active devices on the bus, and you had umpteen brazzillion tiny text files. (Like source code a software developer might have.) But 10-20MBps would be more common for the mix of files typically on a system drive.

If it's the only device on the bus, and the same files copy to other drives fine (do you have any other externals?) then it's probably an issue with the drive. But I would want to try a different external HD to verify that it's not a problem with your system.

The Crashplan guys would come over and ship those seed kits out of the Fedex store where I used to work. They seemed like alright dudes; I wouldn't hesitate to drop customer service a line.

 

[thecoolnessrune]

How are you backing up? Seed creation can take a while depending on the amount of data. Keep in mind that Crashplan is encrypting your data (with up to a 448 bit hash key if chosen). Additionally, Crashplan performs full compression and/or data de-duplication. All of these operations (especially de-duplication) can really take a toll on transfer speeds and make the seed backup take a while. Setting Crashplan to allow the use of high amounts of CPU resources even while the system is active can speed this up. If you have de-dupe on then you'll get disk thrashing if you do not have at least 1GB of free memory reserved per 1TB of data you're backing up, otherwise your speeds will be rough.

If you want to see the "unburdened" transfer speed, make sure you've turned off Compression and De-Duplication on the backup sets. They'll always be encryption though.

 

[GWestphal]

I have 4 gb of memory allocated for it, maybe it's the compression and dedup, but I figured writing to the crashplan seed drive would be a bit faster, I'm not sure it will finish before the deadline they want it shipped back by!

 

[GWestphal]

I have 4 gb of memory allocated for it, maybe it's the compression and dedup, but I figured writing to the crashplan seed drive would be a bit faster, I'm not sure it will finish before the deadline they want it shipped back by!

Standard mix of data, docs, photos, music, videos.

 

[thecoolnessrune]

De-Duplication is always a drag unless you have big memory, and big processing power. Even in the Enterprise market, de-dupe stays off unless capacity needs far outstrip performance needs (like on large, redundant sets of archived snapshots).

I work with the Crashplan guys quite a bit on multiple OS's (Solaris, Linux, Windows, and OSX), and the behavior is the same. Because of the large memory databases, and the overhead of sending bits back and forth to be compared and tested, de-dupe is a dog. Myself and some of the Engineers at Crashplan have discussed this at length. The DDT table used by Crashplan isn't anymore special than the systems employed by ZFS, NetApp, EMC, etc. They're all very similar with each implementing their own special sauce to hopefully make De-Dupe more powerful, but not necessarily faster.

Turn off de-dupe, and you'll see your speeds skyrocket I promise. If you don't need the space, then there's no reason to leave it on. You can even transfer your data now, get it to Crashplan, and then once your seed is up, you can enable full De-Dupe on the Backup set later, and have it do so on the next maintenance cycle (with the penalty of an extremely long maintenance cycle). But at least your data will be up there and usable.

 

[jimhsu]

Set dedupe to minimal or off. I don't have the blog post handy, but default dedupe settings scale as a function of O(log(n)) [with CPU power (usually limited), not bandwidth (usually not limited, with modern cable connections)], which is terrible if your dataset is millions of files / TB in size.

Post is here: http://networkrockstar.ca/2013/09/speeding-up-crashplan-backups/

From what I understand, it's hard to "do" any better with dedupe. Essentially, you're performing a search of every single block that already exists with every new block coming into existence -- it's hard to optimize that.