Which ZFS layout would you use for the following, and why?

[orian]

Hardware
24 drive bay case
4TB drives, all same make/model/etc.

Possible ZFS configurations, arranged by number of disks
20 disks
raidz2 = 1 zpool, 2 vdevs of 10 disks, 2 can fail per vdev, 4 can fail in total
32TB per vdev, 64TB total usable, 20% of total drives can fail

22 disks
raidz3 = 1 zpool, 2 vdevs of 11 disks, 3 can fail per vdev, 6 can fail in total
32TB per vdev, 64TB total usable, 27.27% of the drives can fail

24 disks
raidz2 = 1 zpool, 3 vdevs of 8 disks, 2 can fail per vdev, 6 can fail in total
24TB per vdev, 72TB total usable, 25% of total drives can fail

 

[Ayah]

a) do you need it all to be in one zpool?
b) what kind of drives? (mostly, do they have vibe sensors?)
c) how critical is data availability
d) what throughput do you need?

1) all same make and model, if from same batch can be an extreme risk, though this is mitigated a bit in "enterprise" disks.

 

[mfenn]

Unless you have on-site cold spares and have easy physical access to the box, I highly recommend keeping keeping a couple of spare drives in the zpool.

So for 24 drives I would do the following:

zpool1
raid z2 11 drives
raid z2 11 drives
spares 2 drives

If you are going to be splitting the data into fairly small filesets, you could also do two zpools to get more flexibility for upgrades and protection against catastrophic ZFS bugs, but that of course leads to a partitioning problem once the array gets close to full.

zpool1
raidz2 11 drives
spares 1 drive
zpool2
raidz2 11 drives
spares 1 drive

As more of a general piece of advice, don't let your zpools go about ~92% utilization. ZFS performance gets absolutely horrible when close to full, leading to excessively large application response times.

 

[orian]

a) do you need it all to be in one zpool?
b) what kind of drives? (mostly, do they have vibe sensors?)
c) how critical is data availability
d) what throughput do you need?

1) all same make and model, if from same batch can be an extreme risk, though this is mitigated a bit in "enterprise" disks.

a = I don't need it to be all in one zpool, and the more I thought about it the more it sounds like a bad idea as two large vdevs will be unwieldy

b = Seagate Desktop HDD.15 ST4000DM000 4TB 64MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive Bare Drive. I have been collecting them for the last year or so and they're not all from the same place.

c = I chose ZFS because I don't want to deal with issues like bitrot and losing files so I'd say data availability is important, but not mission critical at all.

d = Throughput is not super important as it will be only accessed by two or three machines at once at a max.

Some people may think these drives are not that great, but someone pointed me to an article on BackBlaze's blog, and it seems they're their preferred drives at the moment:

http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/

Unless you have on-site cold spares and have easy physical access to the box, I highly recommend keeping keeping a couple of spare drives in the zpool.

So for 24 drives I would do the following:

zpool1
raid z2 11 drives
raid z2 11 drives
spares 2 drives

If you are going to be splitting the data into fairly small filesets, you could also do two zpools to get more flexibility for upgrades and protection against catastrophic ZFS bugs, but that of course leads to a partitioning problem once the array gets close to full.

zpool1
raidz2 11 drives
spares 1 drive
zpool2
raidz2 11 drives
spares 1 drive

As more of a general piece of advice, don't let your zpools go about ~92% utilization. ZFS performance gets absolutely horrible when close to full, leading to excessively large application response times.

I was under the impression that you could not do hot spares with ZFS and if doing an 11 drive Z2 pool would result in ZFS complaining that it's not optimal. Does that even matter anymore?

I think I'll settle on the following:

2x 10 disk z2 zpools
1x 4 disk zpool mirror dedicated for ESXi datastore

Thanks for the help everyone!

 

[Ayah]

optimal means best performance for their dynamic stripe sizes, performance during scrubs will probably be very meh.
and yes, there are hot spares in zfs.

everyone looking to backblaze's data as gospel for the home environment is a moronic idea. If you look at their operating temperatures, you might realize why that is so. (<= 38C always, most far lower). for typical home use, you're probably going to be 35-45C and if you're not rackmounted + bolted to extremely solid base, you've added vibrations into the equation.

 

[dave_the_nerd]

I was under the impression that you could not do hot spares with ZFS and if doing an 11 drive Z2 pool would result in ZFS complaining that it's not optimal. Does that even matter anymore?

You can use an odd number of disks, but there's a performance penalty.

You can do hot-spares in FreeNAS. (Add a disk to a zpool as a spare, set auto-replace to true for the zpool).

I think I'll settle on the following:

2x 10 disk z2 zpools
1x 4 disk zpool mirror dedicated for ESXi datastore

Thanks for the help everyone!

If you're going to be actively using ESX (and not just messing around with it for a certification exam) you may find that dedicating four HDDs to your ESX hosts creates something of a bottleneck.

You can alleviate the bottleneck by feeding ZFS (FreeNAS?) RAM (for a write cache) and using an L2ARC (for a read cache). But I would still probably want to put my 24 disks in 2x 12-disk RAIDZ2 pools and just quota off a dataset to be my ESX datastore. There's a speed penalty for writes (which is obviated by ZFS ram caching) but the worst case read performance (most I/O is reads, not writes, so it's usually more important) is better, theoretically.

IMO. Your way would work too.

 

[orian]

Thanks again for the tips! I appreciate all the comments.

 

[mfenn]

In general, putting things into one big zpool is the best bet for aggregate/overall performance because you don't partition your available throughput/IOPS into separate domains. If you start making a bunch of small zpools, then you're going to be leaving a lot of overall performance on the table when some zpools are seeing heavy utilization and others are not. That's why I recommend to generally group zpools by disk type and that's it.

In particular, for demanding workloads like ESXi, you'll be better off with one big zpool like dave mentioned.