"Green" HDDs in FreeNAS RAID-5 Array

[Yellowbeard]

OK, I do not study HDDs at all so please overlook any ignorant questions.

I am seeing some pretty good deals on some of these "green" type HDDs. My minimum capacity is going to be 4 x 1TB drives in RAID-5, so 3 x 1.5TB or 2TB or more is good.

Example: http://www.newegg.com/Product/Produc..._-22136490-L0B

I don't mind going green if the performance and reliability are still good. However, decently rated 7200rpm drives are only about $5 more right now.

Examples: http://www.newegg.com/Product/Produc..._-22136490-L0B

Typical heavy usage for me is something like watching high def video on my HTPC while also having a download coming into the server.

In general:

Are there any specific drives or HD series that I should absolutely avoid for use with FreeNAS in software RAID-5?

I have seen it posted to avoid larger than 1TB drives for reliability issues. Is this factual or just a few people with issues?

Is there a "most reliable" and/or best "bang per buck" HD suitable for this? The bigger the better of course.

Thanks in advance. If this is already answered somewhere, just point me in the right direction. I searched generically but since I am asking about specific drives I thought a new thread was best.

 

[Yellowbeard]

OK....more reading already.......seems green drives are no good for RAID. Agreed?

Been finding a little bit of info saying to stay away from the WD drives also.

The more I read, the less I know.

 

[Blain]

Link up the WD raid drive info you've found.

 

[taltamir]

the difference is really only firmware and compatibility...

When an error is found on a desktop edition hard drive, the drive will enter into a deep recovery cycle to attempt to repair the error, recover the data from the problematic area, and then reallocate a dedicated area to replace the problematic area. This process can take up to 2 minutes depending on the severity of the issue. Most RAID controllers allow a very short amount of time for a hard drive to recover from an error. If a hard drive takes too long to complete this process, the drive will be dropped from the RAID array. Most RAID controllers allow from 7 to 15 seconds for error recovery before dropping a hard drive from an array. Western Digital does not recommend installing desktop edition hard drives in an enterprise environment (on a RAID controller).

Western Digital RAID edition hard drives have a feature called TLER (Time Limited Error Recovery) which stops the hard drive from entering into a deep recovery cycle. The hard drive will only spend 7 seconds to attempt to recover. This means that the hard drive will not be dropped from a RAID array. While TLER is designed for RAID environments, a drive with TLER enabled will work with no performance decrease when used in non-RAID environments.

note that if you buy a raid edition WD drive and call their support and ask for it, they will email you a tool to disable TLER (aka, enable deep recovery state). They will NOT however, let a non RE drive enable TLER, its weirdly named since TLER is just a fancy marketing name for "we disabled deep recovery because dumbass raid controllers can't handle it"...
the key being dumbass raid controllers, I have never had a Green Edition drive drop from my array on my ZFS fileserver (running open solaris, for about 2 years now) I had 3 data corruption events that were fixed thanks to ZFS' built in checksumming and error correction algorithms (something no other FS can do). Altohugh, it is likely that those 3 errors were due to bits flipped by cosmic rays rather then an actual case of unreadable sectors.

Anyways, just use RAID1, that way even if the array breaks you lose no data. And can just recreate by copying the data from drive A to drive B. and the dead sectors can be an early warning sign that a drive is about to fail.

EDIT: Looking at the title you are making a freenas RAID5 array...
1. Don't, RAID5 is horrible POS, never use it.
2. FreeNAS is easier than FreeBSD or solaris (illumos or open solaris), but you don't get the awesome ZFS. However, if you don't have the time to learn it, then go right ahead and use freenas, just use multiple RAID1 arrays instead of a RAID5 array.

EDIT2: FreeNAS actually supports ZFS my mistake. So use it to make a RAIDZ array, it is a lot like raid5, only with all the failures of raid5 fixed. And it works fine with green drives since ZFS is smart and doesn't get stymied by TLER.

 

[sub.mesa]

You only need TLER on Windows-based software RAID or all hardware RAID arrays. You do not want TLER to be enabled if you don't really need it, since enabled TLER comes at a tradeoff: less recovery time means less protection against BER or Bit-Error-Rate; roughly the rate at which you develop bad (unreadable) sectors, or rather where the number of error bits exceeds the error correction capabilities for that sector.

Aside from TLER, the newer Green drives use 4K sectors, which can affect performance and improves resilience against BER. I produced a benchmark chart with initial testing results, which looks like this:

For more information about these tests, i would suggest looking through the 4K testing thread on HardOCP forums, thread URL:
http://hardforum.com/showthread.php?t=1546137

I agree with taltamir that you may want to avoid RAID5. The complexity that it requires isn't worth it since you don't have a checksumming filesystem to protect against corruption. You really want to use ZFS, especially if you want RAID5-like protection using RAID-Z.

Also, please allow your NAS/server the opportunity to do a 24-hour memory test using MemTest86+ before you commit real data to this. RAM errors can be very bad; though even ZFS tries to fix those in the case when i had bad RAM.

 

[RebateMonger]

Also, please allow your NAS/server the opportunity to do a 24-hour memory test using MemTest86+ before you commit real data to this. RAM errors can be very bad; though even ZFS tries to fix those in the case when i had bad RAM.

It appears that both you and taltamir had experiences with memory errors that caused storage errors that ZFS tried to correct.

Can either of you provide more details on what you encountered?

I recall that Intel used to say that 1 GB of memory would likely have a radiation-caused bit flip maybe once every few months. Or something like that. I imagine that ECC memory would help here, but nobody seems to be pushing ECC for non-server use.

Thanks.

 

[taltamir]

I run a scrub operation regularly, scrub tests all currently written data for checksum consistency, if any data fails, it will be repaired if possible (aka, if you have redundancy)
on 3 separate occasions I had scrub report that it found and corrected 1 error.

 

[sub.mesa]

The fact that RAM is not ECC yet may be related due to Intel's strategic decision to keep ECC exclusive to its Xeon family. AMD doesn't do this, but has much lower marketshare. So there would be alot of demand for non-ECC memory. IMO it makes sense to have all the RAM use ECC even for consumer systems, but given product availability and your wallet you have to make choices.

My experience with ZFS with faulty RAM on my test server when doing its first rsync (remote file synchronization) run. It struck me as odd that the script was interrupted with strange errors now and then, but it left it running over the night. Upon the morning, without checking whether it was finished, i initiated a scrub on the ZFS pool; which checks all files and parity to conform to known checksums. Only later i noticed lots of errors in the zpool status output; the scrub found many errors and 'corrected' them. I suspect, however, that most of these were NOT faulty on-disk data but rather the data corrupted in RAM when read from the disk to RAM.

The number of errors exceeded 10.000 (10k); after which i ran through the memtest procedure and replaced the bad RAM module, after which i let it run for another 48 hours, since i didn't need to rush setting this up. After i booted the system again with healthy RAM, i initiated another scrub which found more errors (real on-disk errors) and corrected all of them. I used RAID-Z so only 1 disk of redundancy, though some datasets had copies=2 which is great in this case since it adds another copy along with redundancy; so now you have at least 2*2=4 sources of the same data. The more sources, the better, as ZFS tries all known sources for the known good checksum and thus distinguishes between corrupt and non-corrupt data; something traditional RAID can't do.

The ZFS folk were very keen on ECC memory though, even saying this is a necessity for any stable server. I think that's somewhat misplaced; for a company server you wouldn't want to make sacrifices. But for a home NAS with ZFS i think you can do without ECC memory. Heck; all those people are using NTFS without ANY protection against corruption right now! So going ZFS is a giant leap forward in their data security, i would argue.

My experience does indicate ZFS has some resilience against RAM errors; but i wouldn't leave a system running with bad RAM for long as sooner or later your luck will run out. But aside from bad RAM that continuously corrupts data, an occasional bit-flip from non-ECC memory over many months shouldn't be something to worry about too much. In fact, you would want this memory corruption to happen inside ZFS since it can fix it, rather than inside an application causing it to write corrupt data in the first place or all kind of other unforeseen circumstances that arise from memory corruption.

 

[Emulex]

ecc starts in the cpu, follows through the ram, and definitely covers the bus. which is why server boards and periphs cost more.

you want ECC protection for the entire trip or you might as well not have it.

Anyone that's spent enough time ftp'ing a ton of files (legal of course) has run into a random bit-flip along the way.

 

[taltamir]

you want ECC protection for the entire trip or you might as well not have it.

eh, thats like saying you might as well not wear a helmet because motorcycles don't have seatbelts. Its better to have some ECC than no ECC.
but I agree that proper ECC going all the way through is best. its also expensive.

 

[Blain]

ECC must be supported by all the pertinent components, I think is what Emulex is saying.
You can't have ECC if the CPU doesn't support it.
You can't have ECC if the MB doesn't support it.
And of course you can't have ECC if the memory isn't ECC.

 

[Yellowbeard]

Link up the WD raid drive info you've found.

It was in chat from a tech rep at a retailer I'd rather not name. He generally knows his stuff but we did not get into written documentation. He based that on internal feedback thru their support and also customer and forum feedback.

As far as RAID-5 issues with FreeNAS, I have been running it with either 5 or 6 500GB HDDs for almost 4 years now without 1 hiccup. I don't want to ignore good advice but my individual experience is not bad. I did tear the original RAID-5 (5 x 500GB) array down and run RAID.,....I honestly can't remember if I did 0+1 or 1+0. But, that was with a total of 4 500GB drives and I only did it for a few weeks. I have since rebuilt the RAID-5 array with 6 x 500GB drives.

All in all, the RAID-5 was easier to set up but I guess the main advantage was simplicity. However, RAID 10 (aka the 1+0 I had) may be the best option for performance and fault tolerance.

So, I guess we're back to what disks to use, or not to use. Let's assume 4 or 6 drives in RAID 10. Thanks again for any info.

 

[Yellowbeard]

Hmmm.....ZFS via FreeNAS.....sigh.....more reading to do :-/

http://sourceforge.net/apps/phpbb/freenas/viewforum.php?f=97

 

[Yellowbeard]

Hmmmm.....4 of these in a FreeNAS RAID 1+0 setup seems like a good idea assuming the 5900RPMs won't prove to be any sort of bottleneck. I'd like to get away from the RAID-5 as I have a single failed drive right now (1 of 6) and I'm nervous.

http://www.newegg.com/Product/Produc...82E16822148413

Thoughts?

Specs:

Brand Seagate
Series Barracuda LP
Model ST32000542AS
Packaging Bare Drive Performance
Interface SATA 3.0Gb/s
Capacity 2TB
RPM 5900 RPM
Cache 32MB
Average Latency 5.1ms

 

[Chaoticlusts]

one thing to take into account with these reply's on freenas with ZFS, there's just been a *big* update so for a little bit now it's inherently supported 4k file systems (which it didn't before hense the performance issues listed they should be more or less resolved now) and also with freenas 8 it's moved onto a newer version of openbsd so it supports a far more modern version of ZFS (the older one was getting very very dated) so you can benefit from the ease of freenas with a lot less downsides now (though no matter what you'll always have the latest running opensolaris or straight freebsd)

I'm in the process of backing up my NAS which uses all green drives to rebuild it and i'm sticking with ZFS on freenas...I've really had very little issues with it since I got it running aside from some speed issues (due to the 4k thing that I wasn't aware of until recently)

basically i'd say go for green drives for a NAS...they're cheaper to buy and more importantly cheaper to run (power bills) which will really add up if your NAS runs 24/7..the few downsides there are just can't match those upsides...and yeah as stated above ZFS over raid5...it's simply better

 

[sxr7171]

the difference is really only firmware and compatibility...



note that if you buy a raid edition WD drive and call their support and ask for it, they will email you a tool to disable TLER (aka, enable deep recovery state). They will NOT however, let a non RE drive enable TLER, its weirdly named since TLER is just a fancy marketing name for "we disabled deep recovery because dumbass raid controllers can't handle it"...
the key being dumbass raid controllers, I have never had a Green Edition drive drop from my array on my ZFS fileserver (running open solaris, for about 2 years now) I had 3 data corruption events that were fixed thanks to ZFS' built in checksumming and error correction algorithms (something no other FS can do). Altohugh, it is likely that those 3 errors were due to bits flipped by cosmic rays rather then an actual case of unreadable sectors.

Anyways, just use RAID1, that way even if the array breaks you lose no data. And can just recreate by copying the data from drive A to drive B. and the dead sectors can be an early warning sign that a drive is about to fail.

EDIT: Looking at the title you are making a freenas RAID5 array...
1. Don't, RAID5 is horrible POS, never use it.
2. FreeNAS is easier than FreeBSD or solaris (illumos or open solaris), but you don't get the awesome ZFS. However, if you don't have the time to learn it, then go right ahead and use freenas, just use multiple RAID1 arrays instead of a RAID5 array.

I thought setting up my Nexentastor appliance was one of the easiest things I've done. IMHO, overall it is easier than even WHS, because I never have to troubleshoot stupid random WHS errors/issues. I set it up and it just works. Simply amazing. Set up itself was as easy as booting from a CD allowing it to install itself, setting up the DHCP, setting up a password. The rest is all web interface. Extremely easy.


EDIT: Community Edition now supports 18TB! http://www.nexentastor.org/boards/1/topics/1396

 

[sxr7171]

one thing to take into account with these reply's on freenas with ZFS, there's just been a *big* update so for a little bit now it's inherently supported 4k file systems (which it didn't before hense the performance issues listed they should be more or less resolved now) and also with freenas 8 it's moved onto a newer version of openbsd so it supports a far more modern version of ZFS (the older one was getting very very dated) so you can benefit from the ease of freenas with a lot less downsides now (though no matter what you'll always have the latest running opensolaris or straight freebsd)

I'm in the process of backing up my NAS which uses all green drives to rebuild it and i'm sticking with ZFS on freenas...I've really had very little issues with it since I got it running aside from some speed issues (due to the 4k thing that I wasn't aware of until recently)

basically i'd say go for green drives for a NAS...they're cheaper to buy and more importantly cheaper to run (power bills) which will really add up if your NAS runs 24/7..the few downsides there are just can't match those upsides...and yeah as stated above ZFS over raid5...it's simply better

WDC Green Drives all the way. I can't hear them, and I know they are using less power. Transfer speeds saturate Gig-E anyway. There's no benefit to using faster drives at all.

Good info on FreeNAS I need to do some research.

 

[taltamir]

I thought setting up my Nexentastor appliance was one of the easiest things I've done. IMHO, overall it is easier than even WHS, because I never have to troubleshoot stupid random WHS errors/issues. I set it up and it just works. Simply amazing. Set up itself was as easy as booting from a CD allowing it to install itself, setting up the DHCP, setting up a password. The rest is all web interface. Extremely easy.

Note that I never mentioned nexenta stor or how hard or easy it is... Nexenta stor is an opensolaris (illumos) derivative meant to make it easy to set up a NAS, just like freeNAS is a FreeBSD derivative meant to make it easy to set up a NAS. I didn't mean anything by not mentioning it specifically.

Now that it has been mentioned though, I would point out that nexenta stor costs money, while all those other options are free and work just as well.

 

[Yellowbeard]

OK, so the drive decision is out of the way. Can any of you guys point towards a good NOOBS guide to ZFS? I don't want to play 20 questions here if there is already a good resource online that is well written and easily understood by someone who knows virtually nothing about any OS other than Windows.

If all else is equal, I have had very good luck with FreeNAS, I can afford it, and I'm somewhat familiar with it. So, I'd prefer to stick with FreeNAS and either RAID 1+0 or ZFS.

Thanks again for all the input. It is much appreciated and makes coming to this forum even more enjoyable for me.

 

[taltamir]

If you are comfortable with FreeNAS then just continue using FreeNAS... it has a nice GUI and supports ZFS out of the box. just make sure you choose ZFS as your file system.
Any question you have has probably already been answered here: http://freenas.org/documentation:setup_and_user_guide
and here: http://sourceforge.net/apps/phpbb/freenas/viewforum.php?f=97

What do you mean by RAID1+0 OR ZFS?
ZFS is a file system, you can use ZFS without any raid, with raid1, raid0, raid5 (equivalent called raidz) raid6 (equivalent called raidz2) and raid 1+0.

If you want RAID 1+0 then use ZFS RAID1+0 rather then NTFS RAID1+0 or ext4 RAID1+0.

 

[VirtualLarry]

IMO it makes sense to have all the RAM use ECC even for consumer systems

Amen! What Intel has done to the computing industry as a whole, by relegating ECC support to only their "server" systems, is a travesty. (Tragedy?)

It's ridiculous to think that ANY amount of memory corrupt is "acceptable" in a computer system. ECC is just good, basic, computer science - which seems to have taken a back seat to marketing these days.

 

[taltamir]

Amen! What Intel has done to the computing industry as a whole, by relegating ECC support to only their "server" systems, is a travesty. (Tragedy?)

It's ridiculous to think that ANY amount of memory corrupt is "acceptable" in a computer system. ECC is just good, basic, computer science - which seems to have taken a back seat to marketing these days.

do you mean to tell me AMD supports ECC ram with their non server chips?
Also, what about mobo makers and ram makers? don't they ALSO want a slice of the business dollars pie?

All of them know that companies can afford to pay more, and that they absolutely need ECC... ECC is a way for every one of the sellers to force business to pay more.

 

[Yellowbeard]

What do you mean by RAID1+0 OR ZFS?
ZFS is a file system, you can use ZFS without any raid, with raid1, raid0, raid5 (equivalent called raidz) raid6 (equivalent called raidz2) and raid 1+0.

LOL....see what I mean about NOOB? Once I leave my area of familiarity I know NOTHING about what you guys are talking about.

 

[taltamir]

Alright, I will endeavor to explain.
ZFS stands for Zettabyte file system, it is a system for storing data to the drive. ZFS is actually a set of combined system, that include file system, partitioning, raid tools, and various other tools all designed to work together as a singular system.
There are other file systems such as NTFS (NT File System, the file system microsoft uses for windowsNT and above) and FAT/FAT16/FAT32 (file allocation table, older file systems from microsoft) and exFAT (a new derivative of FAT from MS designed specifically for USB thumb drives, since NTFS is bad for them and FAT32 is ancient).

File systems reside on partitions, which separate a drive into multiple chunks (which show up as separate drives in windows) or, rarely, combine multiple drives together into one partition (ZFS). Common partitions are MBR (master boot record, old and outdated) and GPT (general partition table, modern, lacks support) and the zfs file system (zfs's method can actually combine drives together at the FS/partition level unlike MBR and GPT).
So you can have a 2TB drive running MBR split into two 1TB partitions... one running NTFS and windows, and another running ext4 and linux.

RAID is a technique to create a "virtual drive" from multiple drives for various purposes (which are then partitioned and then used by the file system).
RAID1 is just storing an identical copy on all drives (write speed is the same as single drive, read speed is number of drives times speed of a single drive). All drives but one can fail and your data remains safe. Space is the size of only one drive (so 2x2TB drives give you 2TB of space only)

RAID0 is splitting the data between all drives (write and read speeds are the speed of a single drive times the number of drives.) if ANY drive fails ALL data is lost. Highly unsafe.
Space is just the sum of the drives (2x2TB gives 4TB of space)

RAID5 is using like raid 0, except 1 drive is reserved to contain the mathematical results of performing a calculation on the data on the other drives, such that you could recover the data if any one drive fails. write and read speeds can in theory go as high as the speed of a single drive times the number of drives minus 1... except in reality the complexity of the process means that only the fastest controllers (300+$ ones) can do that, most perform much slower than a single drive.
Space is the size of all drives but one... so 5x2TB drives will give (5-1)*2TB = 8TB space.
There are some problems with the implementation that can cause dataloss in a variety of conditions (write hole, reconstruction fails due to corrupt sectors, and more), and motherboard implementations are all terrible (slow AND unreliable).

RAID6 is raid5 only with 2 drives each containing a different mathematical result, allowing 2 drives to fail without data loss, and losing the space / speed of 2 drives.

RAID1+0 or 0+1 or 10 or 01 is when you make a RAID1 of a RAID0 array, or vice versa. reliability and speed is a mix of raid0 and raid1. pretty nice but requires lots of drives.

RAIDZ is a special implementation of RAID5 that fixes all the problems that can cause dataloss in RAID5 (except, of course, drive failure... I mean it fixes the write hole, the reconstruction errors, etc). However in order to do so it has to limit you to ZFS file system.
RAIDZ2 is the same thing for RAID6.

ZFS has several huge advantages over older file systems, foremost that it has unmatched data security, it is the only filesystem that prevents data corruption by saving checksums for all your data. As long as you have a redundant storage system (raid1, raidz, raidz2, raid10, or have set it to keep multiple copies of a file via the copies=N command) then it will repair data corruption on the fly.
NTFS and ext4 don't have that...
ZFS is the only file system that does that that is complete and commercially available...
Google has a custom file system (that actually sits on top of ext2, although they have begun work migrating it to ext4) that is their greatest trade secret and not available for sale. It also has robust data safeties.

The company that designed ZFS (Sun, which was purchased by oracle) released it as open source under the CDDL open source license. GPL forbids the use of software of certain license types, and as a result linux cannot implement ZFS as is, only via a userspace module (FUSE), which is how linux implemented NTFS, but this is a slow and long process.
Apple begun implementing ZFS but it seems to not go anywhere, and FreeBSD has ported ZFS (the BSD open source license called BSL permits the use of CDDL type software).
CDDL, GPL, and BSL all are legal licenses that allow the use of open source software under various conditions.

Anyways, due to its incompatibility with license of ZFS source code, and fear of nefarious intents from SUN, linux community began development of their own GPL alternative to ZFS called BTRFS (better file system), which is currently not ready for use. They could have just written ZFS compatible code from scratch instead, but they didn't trust SUN (or whomever buys them) to not later sue them for patent issues.
This turned out to be wise as SUN was purchased by oracle, who immediately discontinued opensolaris (a fork called illumos is now being maintained by companies that came to depend on it) and has sued google for using java (which is open source SUN product, and which SUN gave many guarantees that you couldn't be sued for using even if they wanted to) in their android OS.
So oracle is actively hostile to open source and is being predatory... not a good sign.

despite its uncertain future, ZFS will not magically disappear in a puff of smoke just because companies change hands and policies, you can still download and use, for free, existing implementation and software. and it is still the ONLY resistant to data corruption file system in the history of humanity available to everyone (the first is the google exclusive google FS). So enjoy it until BTRFS is ready.

 

[Yellowbeard]

RAIDZ is a special implementation of RAID5 that fixes all the problems that can cause dataloss in RAID5 (except, of course, drive failure... I mean it fixes the write hole, the reconstruction errors, etc). However in order to do so it has to limit you to ZFS file system.
RAIDZ2 is the same thing for RAID6.

ZFS has several huge advantages over older file systems, foremost that it has unmatched data security, it is the only filesystem that prevents data corruption by saving checksums for all your data. As long as you have a redundant storage system (raid1, raidz, raidz2, raid10, or have set it to keep multiple copies of a file via the copies=N command) then it will repair data corruption on the fly.
NTFS and ext4 don't have that...
ZFS is the only file system that does that that is complete and commercially available...
Google has a custom file system (that actually sits on top of ext2, although they have begun work migrating it to ext4) that is their greatest trade secret and not available for sale. It also has robust data safeties.

The company that designed ZFS (Sun, which was purchased by oracle) released it as open source under the CDDL open source license. GPL forbids the use of software of certain license types, and as a result linux cannot implement ZFS as is, only via a userspace module (FUSE), which is how linux implemented NTFS, but this is a slow and long process.
Apple begun implementing ZFS but it seems to not go anywhere, and FreeBSD has ported ZFS (the BSD open source license called BSL permits the use of CDDL type software).
CDDL, GPL, and BSL all are legal licenses that allow the use of open source software under various conditions.

Anyways, due to its incompatibility with ZFS source code, and fear of nefarious intents from SUN, linux community began development of their own GPL alternative to ZFS called BTRFS, which is currently not ready for use. They could have just written ZFS compatible code from scratch instead, but they didn't trust SUN (or whomever buys them) to not later sue them for patent issues.
This turned out to be wise as SUN was purchased by oracle, who immediately discontinued opensolaris (a fork called illumos is now being maintained by companies that came to depend on it) and has sued google for using java (which is open source SUN product, and which SUN gave many guarantees that you couldn't be sued for using even if they wanted to) in their android OS.
So oracle is actively hostile to open source and is being predatory... not a good sign.

despite its uncertain future, ZFS will not magically disappear in a puff of smoke just because companies change hands and policies, you can still download and use, for free, existing implementation and software. and it is still the ONLY resistant to data corruption file system in the history of humanity available to everyone (the first is the google exclusive google FS). So enjoy it until BTRFS (better File System) is available.

Thanks a lot for this info. I have been doing some ZFS Googling today while I watched my Falcons go to 8-2. I especially like the part in bold.