Consumer SSDs (M4) for RAID-5 server use?

[DukeN]

Thinking of putting eight of these into an older server for an IO intensive splunk server.

Thoughts on anyone who's used SSDs in RAID-5?

Thanks

 

[Emulex]

raid-5 probably will amplify writes, why not just do jbod or spanning?

One write could cause all drives to write a new sector.

Jbod or file based raid -not so much.

Raid-1/10 - possible that less write amplification could occur.

 

[ohforfs]

My worry would be the wear rate due to rebuilds. I have run fakeraid (intel chipset) RAID 5 and mdadm software Raid 5 on Linux and the thing that I notice is the heavy disk access. To me it always seems excessive and I worry for my HDD's (which are consumer ones). Rebuilds and verifications are a lot more common than you might expect.

You pays your money, you takes your choice, I suppose. If it is mission critical then I wouldn't do it. If you can afford to go back and change the drives if they fail, then I think that would be an interesting experiment that we would *all* like to be updated on

 

[Cerb]

raid-5 probably will amplify writes, why not just do jbod or spanning?

One write could cause all drives to write a new sector.

One write will have to write to all drives (edit for posterity: not necessarily, it could be only 2 for a small in-place edit). A very small 4K-multiple 4K-aligned stripe size should be mandatory. Other than, it aught to be OK.

For writes themselves, if you want them delayed and to bunch up more, consider setting laptop mode on, mounting with noatime, nodiratime, or relatime, and increasing the writeback time. All but the mount options can be changed dynamically, through /proc/sys/vm, and permanently set via /etc/sysctl.conf.

Rebuilds and verifications are a lot more common than you might expect.

Have you tracked down the causes? I can't say that I can vouch for that, even with home servers not protected by so much as a UPS. You might want to see if the drives are in a more extreme power-saving mode, causing false dropouts.

 

[DukeN]

Looks like I'll be finishing this build shortly.

Anyone want any type of specific benches run?

 

[Emulex]

Yeah run 32 queue depth write overnight and tell us how much life it used up?

Also if you change 1 block only, does it cause writes to all drives? The drive life indicator would tell you so after a few days of continuos writes.

Good luck, I'll buy you a beer if you can get this to last a year with reasonable server like write conditions.

last means, no rebuilds, no lost data no dropped drives, no consistency checks. No drives stuck in read only mode

 

[vss1980]

Thinking about this logically, the wear rate will be impacted due to a single block update causing writes to at least 2 drives, but I think the problem is being exaggerated.

As per the diagram/demo from here: http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_5

A standard simple RAID 0 stripped set of 4 drives say would split a 100MB down in to 25MB being stored across the 4 drives, thereby lowering wear (to a degree) on those drives. RAID 5 would use up 33.33 MB across all 4 drives - not as good as RAID 0 (but hey you've got the tolerance) but still less wear per drive compared to RAID 1 or no RAID at all.

Of course, as I said at the beginning, a 1 bit change in a file will spawn writes to at least 2 drives. But I'd have thought this configuration would fare better than say 1 larger SSD drive would....
But, on the other hand, if you had a complete file re-write, again the wear would be divided among the amount of drives in the set, so compared to RAID 1 or a single drive, you would get less wear...

<...Dons flame proof coat and awaits shooting down...>

I will agree, RAID 5 will not be 'good' for the drives and usage types could make it worse still, but practically speaking HDDs suffered through use also, so RAID 5 was just as bad for them. But, the advantage is its stripped performance with the added bonus that you could take an arrow to an SSD drive but still carry on your adventure....

 

[hhhd1]

You need to overprovision, atleast 7%.

 

[Emulex]

remember ALL of the drives are going to fail at the same write age, so you can take an arrow to all drives at once.

hhhd1, how much does intel 710 overprovision for mlc?

 

[MarkLuvsCS]

remember ALL of the drives are going to fail at the same write age, so you can take an arrow to all drives at once.

hhhd1, how much does intel 710 overprovision for mlc?

~40% overprovision in the intel 710 drives.

 

[Zxian]

With any decent hardware controller, a write on a RAID5 array will only write to two of the drives in the array. If writes were performed on all drives for a single write operation, there's no way you'd see a performance increase for both reads and writes in RAID5.

How much writing are you looking at doing? I honestly wouldn't worry about write endurance unless you're looking at several hundred gigabytes per day. StorageReviewdid a test on a Patriot Wildfire comparing the performance between a fresh drive and one that they had written 270TB of data to. There's little to no performance degredation with 20% life left.

 

[Ferzerp]

Rebuilds and verifications are a lot more common than you might expect.

No, no they are not. This only happens commonly when using drives with consumer functions in RAID sets. This is using the wrong product. It is not inherent to raid arrays. When you're using the proper equipment, it only happens if a drive is failing/fails.

 

[Jeff7181]

With any decent hardware controller, a write on a RAID5 array will only write to two of the drives in the array. If writes were performed on all drives for a single write operation, there's no way you'd see a performance increase for both reads and writes in RAID5.

How much writing are you looking at doing? I honestly wouldn't worry about write endurance unless you're looking at several hundred gigabytes per day. StorageReviewdid a test on a Patriot Wildfire comparing the performance between a fresh drive and one that they had written 270TB of data to. There's little to no performance degredation with 20% life left.

How do you figure? Two data disks and one parity disk = three writes.

 

[Cerb]

How do you figure? Two data disks and one parity disk = three writes.

The total data set is {A,B,P}, where A is drive n%3's portion, B is drive (n+1)%3's portion, and P is drive (n+2)%3's portion, the parity.

Modifiy A -> {A,B,P} must be read, and P must be recalculated, but only {A,P} must be written. B never changed, so A xor B = P is still true with the old value of B.

Modify B -> {A,B,P} must be read, and P must be recalculated, but only {B,P} must be written. A never changed, so A xor B = P is still true with the old value of A.

Modify A and B -> {A,B,P} must be read, P must be recalculated, and all of {A,B,P} must be written.

 

[Jeff7181]

The total data set is {A,B,P}, where A is drive n%3's portion, B is drive (n+1)%3's portion, and P is drive (n+2)%3's portion, the parity.

Modifiy A -> {A,B,P} must be read, and P must be recalculated, but only {A,P} must be written. B never changed, so A xor B = P is still true with the old value of B.

Modify B -> {A,B,P} must be read, and P must be recalculated, but only {B,P} must be written. A never changed, so A xor B = P is still true with the old value of A.

Modify A and B -> {A,B,P} must be read, P must be recalculated, and all of {A,B,P} must be written.

So you're making the assumption that a write is a modify, not new data...

 

[Cerb]

So you're making the assumption that a write is a modify, not new data...

Any write is a modify, to any fully initialized array. That it has been initialized is the only assumption I made, and I'd say it's a pretty good one to assume. You may also notice that I short-circuited my brain in my first post, then edited after Zxian's posting, having reasoned it out, which is also why I took the time to explicate on it.

If the array is initialized, there is no longer any such thing as new data. All writes are replacing existing data.

If mdadm keeps bitmaps of used stripes, then there may be new writes as the FS gets used, and so it would be possible to need to write new data to the array. However, that would be a special optimization (easy to extend for TRIM support, if that is ever accepted), I don't know if it is actually done, and it could only occur once per stripe during the array's life time.

 

[Jeff7181]

Any write is a modify, to any fully initialized array. That it has been initialized is the only assumption I made, and I'd say it's a pretty good one to assume. You may also notice that I short-circuited my brain in my first post, then edited after Zxian's posting, having reasoned it out, which is also why I took the time to explicate on it.

If the array is initialized, there is no longer any such thing as new data. All writes are replacing existing data.

If mdadm keeps bitmaps of used stripes, then there may be new writes as the FS gets used, and so it would be possible to need to write new data to the array. However, that would be a special optimization (easy to extend for TRIM support, if that is ever accepted), I don't know if it is actually done, and it could only occur once per stripe during the array's life time.

Hmmm. I guess I've never gained that deep of an understanding of how RAID operates. So if I understand correctly, because there's existing 1's and 0's on the disk, parity has already been calculated for those even if they're "empty" space as far as the file system is concerned? That would make sense.

Is it really that often that A or B is written but not both? If the changes you make can be made by modifying A and not B, maybe your stripe size is too large?

 

[Cerb]

Hmmm. I guess I've never gained that deep of an understanding of how RAID operates. So if I understand correctly, because there's existing 1's and 0's on the disk, parity has already been calculated for those even if they're "empty" space as far as the file system is concerned? That would make sense.

Pretty much. The traditional method is to assume unused stripes are zeroes (though, any values would work, as long as the parity matches), and make them so if they aren't (0 xor 0 = 0).

Part of the reason to write it all out (and read it back again) is also to do a RAID-level check of bad blocks, as it would be kind of silly for a data integrity technique to assume that all the blocks are good. I'm sure most controllers also write out metadata across the drive, in the process.

Is it really that often that A or B is written but not both? If the changes you make can be made by modifying A and not B, maybe your stripe size is too large?

Yes. Hard drives seek slowly. Traditionally, stripe sizes have been very large, relative to data, because RAID 5 must read all stripes to be able to verify the data, and to change it. 64KB, 128KB, and 256KB were/are very common, to reduce drive thrashing with random access, as a larger stripe size forces more nearby data into RAM. Most random access tends to occur nearby, in real workloads on real files [that aren't swap files]. With SSDs, small sizes make a lot more sense.

 

[Zxian]

Cerb got it right. The purpose of initializing any RAID array is to ensure the integrity of the stored data according to the mirroring/parity scheme in place.

The file system also gets very little information about what is actually going on at the hardware level. When I make a write to my RAID5 storage array, Windows requests that a write be made to the hardware device. The device will then either a) queue up the write, tell windows it's been done, and then perform the write when possible or b) write the data and then tell windows that the write has been completed (these depend on your write caching settings).

One big misnomer to get out of your head when talking about RAID5 is the concept of a parity drive. You have a multiple data blocks and a parity block. All of the disks share these roles depending on which particular block you're looking at.