Drive Pooling for Windows

[Carson Dyle]

What are some recommendations for good drive pooling software for Windows Server? I dont need RAID.

 

[Doomer]

Stablebit Drive Pool works for me. It has folder duplication if you need it.

 

[DaveJ]

Another vote for Stablebit Drivepool. I've had it running for 2 years on a WHS 2011 box and it's rock solid.

 

[Towermax]

Another vote for Stablebit Drivepool. I've had it running for 2 years on a WHS 2011 box and it's rock solid.

Same here--Stablebit has been trouble-free and easy to use.

 

[greenhawk]

Another for StableBit. Just installed it about 2 weeks ago and finding it much better than my previous option of NTFS Junctions.

There are a few other offerings around, but some are either nearly un heard of by the masses (few comments about them), they are not windows or they appear to have stagnated without updates.

Other option I was looking at was FlexRaid as they now have a drive pooling option ( I like there data protection system better than nearly all other options around currently), but currently, just after a stable pooling setup and the scandisk option from stablebit seemed to be worth trying via using windows one. My down side is that the disk scanning software does not work through my SAS controller / SAS expanders to read SMART information.

 

[Carson Dyle]

How is drive spin up and spin down handled with Stablebit's DrivePool? I expect that directories, possibly even files, are spanned across drives. Does this mean that every drive in a pool will be spun up when any folder in pool is accessed?

I'm not sure which would be worse - having all of the drives spun up any time I access anything, or having to wait at random times, because I'm trying to access something that happens to be on a different, powered down drive. Right now with JBOD at least I know in advance when I'll probably need to wait.

 

[Ig]

http://community.covecube.com/index.php?/topic/355-sleeping-drives-pool-access/?hl=+spin++down

• 
  For unduplicated files, StableBit DrivePool forwards all file I/O to the individual disk that the file is on.

• If the file exists on more than one disks (i.e. is duplicated), then:
  • All file modification requests (such as writing to the file, setting its attributes, etc...) go to all of the disks that the file is on.
  • Read requests will either go to the first disk that the file is on (when read striping is disabled), or to one of the disks that the file is on, as determined by the read striping algorithm.

• A directory listing operation works by querying all of the disks in parallel. This will force NTFS to read the MFT directory indexes on all of the disks where the directory being listed exists. These indexes can be cached and can theoretically be served entirely from RAM.

• Opening a file is similar to a directory listing. This will tell NTFS to query its directory indexes on all of the disks part of that pool in order to locate the disks that contain that file. This also is done in parallel and can be served by the system cache.

So in short, StableBit DrivePool may spin up disks in many circumstances, I haven't really done testing to see how often this is.

 

[Phaetos]

how is this better than just adding a new HD, and assigning a drive letter?

 

[Kneedragger]

Windows 8 has it baked in (Storage Spaces) and you also have Drivebender.

 

[Carson Dyle]

how is this better than just adding a new HD, and assigning a drive letter?

It gives you the convenience of seeing a large, contiguous storage space ("drive") that doesn't require you to actively manage the physical locations of files.

For example, I have a lot of downloaded TV shows - a little over 4 TB worth - located on two 3 TB drives. I could combine them in a drive pool, but right now I just put most on the first drive and the rest on the second drive. I can put series with names starting from T-Z on the second drive, or I can put series from the 1970s on that drive. But whatever I do, I have to both manage it myself and I have to remember where I put things.

I keep going back and forth about whether or not drive pooling would be desirable for me. One of the things that concerns me is the loss of a hard drive. I can easily replace any file I like, so I don't need RAID or even a backup of any of the content. But if I were using a drive pool and don't know which drives contain which files, if I lose a drive, it would be chaos trying to replace individual files scattered across drives. Right now, if I lose drive two and lose all of my 1970s shows, I know to download all of Barney Miller or the Rockford Files again.

The other thing I don't like is the idea of what it might be like having to wait for drives to spin up whithin the pool. Again, with files scattered across drives and not knowing where they're located, I can see how browsing might incur multiple waits for drives.

 

[greenhawk]

how is this better than just adding a new HD, and assigning a drive letter?

It is something people look at doing when they have 2 or more drives as it makes several drives look like one. It is done to make it easier to organize large file collections without having to separate different parts to different drives.

 

[greenhawk]

The other thing I don't like is the idea of what it might be like having to wait for drives to spin up whithin the pool. Again, with files scattered across drives and not knowing where they're located, I can see how browsing might incur multiple waits for drives.

Some drive pool setups give you control over where a file is stored so that file storage is based on how you want it. Some users might just use StableBit in its default mode, but some probably use the option of grouping files by some other method.

http://stablebit.com/DrivePool/Features
look under "Control File Placement and Organization"

edit: if you do have enough space, StableBit's drivepool does have a basic backup protection at the file/folder level by keeping a second copy of a file on another drive, but like raid 1, it needs twice the space.

 

[Doomer]

Drivebender sucks ass. I had a problem that their tech support couldn't solve. On top of that, every freaking time I uninstalled it and reinstalled as per their instructions, I had to email them to release the license so I could proceed. I finally gave up and said never again. Moved to Stablebit and never looked back.

 

[fleshconsumed]

I keep going back and forth about whether or not drive pooling would be desirable for me. One of the things that concerns me is the loss of a hard drive. I can easily replace any file I like, so I don't need RAID or even a backup of any of the content. But if I were using a drive pool and don't know which drives contain which files, if I lose a drive, it would be chaos trying to replace individual files scattered across drives. Right now, if I lose drive two and lose all of my 1970s shows, I know to download all of Barney Miller or the Rockford Files again.

I'm using combination of Stablebit Drivepool and open source Snapraid software. The Stablebit Drivepool is there for hard drive pooling, Snapraid is there for data integrity. You can read up on Snapraid, but what it does is it essentially emulates snapshot based RAID4 in software. Or in other words it's an executable that you can run and that will calculate parity information on your data at your request. If one of your drives dies or gets corrupted, all you need is to reconfigure the configuration file and tell executable to fix missing data. I've tested it by deleting a couple of files and telling it to rebuild, and it works.

There are downsides to it, such as 1) it does not offer real time protection, so you have to set up a nightly job to recalculate parity every night, 2) it requires willingness to RTFM, follow it, set up configuration file, and run command line program and 3) and you will need to remove hidden and system attributes from stablebit drivepool folders which is not hard to do, but once again, requires certain level of proficiency RTFM and running command line commands.

Yes, it does sound a bit complicated and convoluted, but unfortunately there is nothing else that I could find that does it better. On the bright side it's really not as bad as it sounds and once you set up the system it's leave and forget. I used to have hardware RAID5, but when it was time to upgrade I moved to just a bunch of hard drives pooled together with Stablebit Drivepool and protected by Snapraid. I just didn't want to deal with hardware raid costs and upgrade complexities. The drivepool + snapraid replaces it quite nicely and is a lot cheaper and less stressful to maintain.

 

[Carson Dyle]

Yeah, I've pretty much come to that conclusion ... that I wouldn't want to use drive pooling without using at least some type of snapshot RAID.

There was another company with commercial products that I can't remember now, both drive pooling and snapshot RAID. Reasonably priced, but I can't recall the name...

 

[Cerb]

FlexRAID?

 

[Carson Dyle]

FlexRAID?

That may be it, but I thought they sold the pooling software and RAID as two separate products. I don't see that on their web site, but they may have changed their model a little.

 

[smitbret]

That may be it, but I thought they sold the pooling software and RAID as two separate products. I don't see that on their web site, but they may have changed their model a little.

You can buy them separately, but I suspect that most get the whole package.

Personally, I chalk FlexRAID up as one of my top 5 tech purchases of the last 5 years

 

[Carson Dyle]

You can buy them separately, but I suspect that most get the whole package.

Personally, I chalk FlexRAID up as one of my top 5 tech purchases of the last 5 years

Ok, I see it now. They have two different types of RAID product now. I haven't quite grasped what their tRAID ("Transparent RAID") does, but it's a lot pricier. The other is RAID-F ("RAID over File System"). That's the one with two components that can be purchased separately or as a package.

 

[Carson Dyle]

With FlexRAID's snapshot RAID (using RAID-F), how do you determine how many drives are needed for the parity data? I'd be looking at putting 11 drives with 23 TB into the drive pool, or possibly creating two pools, with 4 drives and 7 drives, respectively.

Are there any hard rules for the size of parity drive(s), such as they must be at least as large as the largest data drive?

 

[smitbret]

With FlexRAID's snapshot RAID (using RAID-F), how do you determine how many drives are needed for the parity data? I'd be looking at putting 11 drives with 23 TB into the drive pool, or possibly creating two pools, with 4 drives and 7 drives, respectively.

Are there any hard rules for the size of parity drive(s), such as they must be at least as large as the largest data drive?

You can decide how many parity drives you want to use. 1 parity drive covers the loss of one drive in the array, regardless of how many disks. You can run a 2 parity drive system if you want to cover the loss of 2 drives, etc. FlexRAID will let you use as many parity drives as you'd like. I imagine most users just use 1 which is similar to RAID 5 but without the striping, although I'm sure there Are plenty of 2 parity drive systems as well.

The data drives can only be as large as the smallest parity drive. If you have a 2TB parity drive and a 3TB data drive, you'll only be able to use 2TB on the data drive.

The parity drive is just a simple checksum of bits across the data drives. For instance in a 3 data drive setup, FlexRAID would look at the first bit on each drive and add up the total for that hit location from each of the drives. So, if the first bit on each drive is set to 1 on a 3 data drive setup, FlexRAID records a 3 for the first bit. If the bit 2 looks like this 1 0 1, then it records 2 for the 2nd bit. Basically, the parity drive is just a map of the entire array. When a drive drops out, FlexRAID compares the existing total and compares it to the parity information and just fills in with a 1 or 0 as needed.

I hope that makes sense.

 

[Carson Dyle]

Not much of that explanation sounds plausible to me, but maybe I really don't understand.

If it's to be able to reconstruct data in the event of a drive failure (not just detect an error) then it needs to store much more than just checksums.

I would think there has to be a limit to how much data a drive with X bytes of storage on it can protect. I can't see how a single 3TB drive can store enough information to protect 20 x 3TB data drives, for instance.

 

[smitbret]

Not much of that explanation sounds plausible to me, but maybe I really don't understand.

If it's to be able to reconstruct data in the event of a drive failure (not just detect an error) then it needs to store much more than just checksums.

I would think there has to be a limit to how much data a drive with X bytes of storage on it can protect. I can't see how a single 3TB drive can store enough information to protect 20 x 3TB data drives, for instance.

It works because it doesn't store any of the actual data on the parity drive. It works for the same reason that the parity drive has to be as big or bigger than any of the data drives. Let's really simplify this.

Let's say that you have 3 HDDs in a FlexRAID. They are all the same size, but are very small, only able to store 1 bit of information. So, let's say that 1 bit on drive 1 was a 1 and the bit on drive 2 was a 0. The parity drive says "OK, for the entire array, the sum of the data at that bit location is 1" and stores that information.

1 + 0 = 1

If for some reason, drive 1 drops out, when the parity check occurs, FlexRAID notices that the sum isn't the same as what it was before:

/ + 0 does not equal 1

FlexRAID then fills in the missing data so that everything equals 1 again:

X + 0 = 1 so FlexRAID calculates that x=1

It works for large numbers of HDDS because it is still only storing one number for that bit location on the entire array. If you have 8 data drives and 1 parity drive and they can all hold three bits of info then it looks like this:

1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 = 5 for bit location #1 across the whole array
0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 = 4 for bit location #2 across the whole array
0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 = 2 for bit location #3 across the whole array

The parity drive stores the sum only. Add a 9th data drive that is all 1s and the parity drive will now store 6,5,3, respectively. It doesn't store any more info, just different info. If 1 of the drives drops out, then FlexRAID can fill in the missing data so that it's totals match up again.

That's also why if you use 1 parity drive, you can only fully recover from 1 HDD failure. If you lose 2, FlexRAID has no way of knowing which drive had held what. Adding parity drives increases the number of drive failures. I don't know exactly how, but I assume it's some type of data sums + parity calc = x type of formula.

It's also why the data drives can't be bigger than the parity, cause the drive can't map to the sectors it doesn't have.

 

[smitbret]

Not much of that explanation sounds plausible to me, but maybe I really don't understand.

If it's to be able to reconstruct data in the event of a drive failure (not just detect an error) then it needs to store much more than just checksums.

I would think there has to be a limit to how much data a drive with X bytes of storage on it can protect. I can't see how a single 3TB drive can store enough information to protect 20 x 3TB data drives, for instance.

It works because it doesn't store any of the actual data on the parity drive. It works for the same reason that the parity drive has to be as big or bigger than any of the data drives. Let's really simplify this.

Let's say that you have 3 HDDs in a FlexRAID. They are all the same size, but are very small, only able to store 1 bit of information. So, let's say that 1 bit on drive 1 was a 1 and the bit on drive 2 was a 0. The parity drive says "OK, for the entire array, the sum of the data at that bit location is 1" and stores that information.

1 + 0 = 1

If for some reason, drive 1 drops out, when the parity check occurs, FlexRAID notices that the sum isn't the same as what it was before:

/ + 0 does not equal 1

FlexRAID then fills in the missing data so that everything equals 1 again:

X + 0 = 1 so FlexRAID calculates that x=1

It works for large numbers of HDDS because it is still only storing one number for that bit location on the entire array. If you have 8 data drives and 1 parity drive and they can all hold three bits of info then it looks like this:

1 + 1 + 0 + 1 + 0 + 1 + 1 + 0 = 5 for bit location #1 across the whole array
0 + 0 + 1 + 0 + 1 + 0 + 1 + 1 = 4 for bit location #2 across the whole array
0 + 0 + 0 + 0 + 1 + 0 + 1 + 0 = 2 for bit location #3 across the whole array

The parity drive stores the sum only. Add a 9th data drive that is all 1s and the parity drive will now store 6,5,3, respectively. It doesn't store any more info, just different info. If 1 of the drives drops out, then FlexRAID can fill in the missing data so that it's totals match up again.

That's also why if you use 1 parity drive, you can only fully recover from 1 HDD failure. If you lose 2, FlexRAID has no way of knowing which drive had held what. Adding parity drives increases the number of drive failures. I don't know exactly how, but I assume it's some type of data sums + parity calc = x type of formula.

 

[fleshconsumed]

@Carson Dyle that's how parity works. You do not need to do 1-1 duplication to be fully backed up. In theory yes, a single parity drive (PPU aka Parity Protection Unit) can protect against a single drive (DRU aka Data Risk Unit) failure in a 20 drive array. However, in general it is not a smart idea to have just one PPU drive for 20 DRU. In order to rebuild failed DRU ALL remaining 19 drives plus the PPU will have to be read from start to finish, which naturally increases risk of another drive failure. Plus once you get into that much data you may run into unrecoverable reading error issue. You can read about it over here http://forums.storagereview.com/index.php/topic/34094-is-raid-56-dead-due-to-large-drive-capacities/ . Basically, hard drives are specs say that every once in a while a hard drive will throw a random error after reading so much data, generally it's 1 error for every 10^14 reads due to some random act of universe. If that error happens during rebuild, your rebuild fails. You may try again hoping you won't get unrecoverable error again, but as the amount of data you have increases, so are the chances of unrecoverable read error.

So in general the more DRU you have, the more PPUs are recommended. My recommendation would be to have one parity drive for every 6 data drives. So if you have 6 or less data drives, have one parity drive. If you have 6-12 data drives, have two parity drives, and so on. Currently I have 10 drives in my file server, 8 of them are data drives, and 2 are parity drives.

On a sidenote. I was considering FlexRAID (as opposed to StableBit DrivePool + SnapRAID), however after thorough research I decided against it for multiple reasons. The developer sounds like a douche, see here http://lime-technology.com/forum/index.php?PHPSESSID=54ea1e2961fcad1c3ef757f690159302&topic=25927.0 Yes, he can and should promote your product, but if he can't respect his competitors, how can you be sure he can respect the clients? Plus it's just one guy running the entire show right now (a while back he was talking about hiring contractors to do the GUI while he concentrates on the core engine development, but I have no idea if that ever happened), so his resources are stretched thin. Right now he has two main projects Raid-F and tRaid. Raid-F is his original work that is essentially drive pooling + snapshot parity protection + use at your own risk real time parity protection system. tRAID is a new development that is fundamentally different from Raid-F. He has sunk most of his time into tRAID, so RAID-F is currently being ignored and the bug fixes are extremely slow. And the guy had history of slow bug fixes even before he started tRAID, so it's how he works. So my general impression is that the guy has too much on his plate so the quality of the product has definitely suffered, he lacks time to do the bug fixes, and in general he is overly defensive/combative/asshole. Personally I've made a decision not to give him my money. But do your own research.