"Green" HDDs in FreeNAS RAID-5 Array

[Emulex]

most modern filesystems require ecc/checksums so they can dedupe - so it only makes sense. Sucks that ZFS has hit a brick wall with sun. Hopefully BTRFS won't take as long as ZFS has taken to get into mainstream supported o/s.

 

[sub.mesa]

do you mean to tell me AMD supports ECC ram with their non server chips?
Also, what about mobo makers and ram makers? don't they ALSO want a slice of the business dollars pie?

I'm not sure if all AMD motherboards support it, but regular Micro-ATX $50 AMD-boards from Asus just support ECC.

With modern chips the memory controller is integrated, so you only need CPU+BIOS support to activate ECC, as far as i know.

All my AMD systems support ECC; shame i didn't buy ECC for all of them. I will only buy ECC from now on, and it's good to see memory prices are falling rapidly; 175 euro for 12GiB ECC DDR3 SDRAM; great deal IMO!

All of them know that companies can afford to pay more, and that they absolutely need ECC... ECC is a way for every one of the sellers to force business to pay more.

Yes, but this is a technology that could be employed for normal consumers as well; if Intel integrated ECC in consumer chips, non-ECC memory might even disappear and it wouldn't really be more expensive; just one more chip out of 16; boo-hoo!

So if you want cheap ECC support, AMD platform is your only choice. AMD has some nice chips for a low-power NAS though; the 45W quadcore or 25W dualcore looked nice and cheap, and is excellent for this purpose.

 

[taltamir]

Yes, but this is a technology that could be employed for normal consumers as well; if Intel integrated ECC in consumer chips, non-ECC memory might even disappear and it wouldn't really be more expensive; just one more chip out of 16; boo-hoo!

Something is worth how much people are willing to pay for it, not how much it costs to manufacture. I KNOW it is cheap as hell to make. My point is that they do it on purpose to gouge businesses.

So if you want cheap ECC support, AMD platform is your only choice. AMD has some nice chips for a low-power NAS though; the 45W quadcore or 25W dualcore looked nice and cheap, and is excellent for this purpose.

And yet, RAM makers are still going to gouge me... since ECC is prices for the enterprise.

 

[sub.mesa]

Well i think right now, the biggest problem is that Intels core ix chips do not support ECC. The ECC memory itself is fairly cheap now; only 15 euro more expensive than non-ECC at 12GiB; though these are bottom prices for Kingston valueRAM. DRAMeXchange suggests prices for DRAM would be falling 25% in Q4, i've seen some of those price drops in my area already. Perhaps january would be the best time to buy, the oversupply and weak demand in January might drop prices to their lowest point yet. Just speculation, though.

If you want to use ECC and you haven't yet bought any equipment, then you have all the opportunity to get ECC by careful planning. But if you already have a DDR2 system or a DDR3 system but Intel non-Xeon cpu, then this is alot harder and it may indeed not be worth it.

I think anyone should get ECC if you can, without too much added costs. For any system, really. Sure on some consumer systems a crash, or if you are unlucky a corrupt filesystem, isn't the end of the world. But isn't it time computers start WORKING always, no more bluescreen or crashing apps or whatever? ECC is a logical step to stable computer systems, and it's already employed in numerous other technologies in every computer.

But as usually is the case, marketing and product strategy is winning from technical arguments. I can think of no other reason why not to use ECC on all RAM and memory controllers.

 

[SViscusi]

Just a note on advanced format drives like the new WD Green's and RAID. While in some configurations they have trouble with becoming unaligned, if you keep the data drive array to a divisible of 4096 eg. 2 data 1 parity, 4 data 2 parity, it works fine.

 

[sub.mesa]

You can use the green drives in 'odd' combinations as well, like 4-disk RAID5 (3 data drives). This 'limitation' would only apply to RAID-Z on ZFS since it uses a variable stripesize.

Any stripesize is a multiple of sectorsize; so the stripe is always aligned. You just need to use aligned partitions and you can use the Green 5400rpm advanced format disks in any RAID config.

Also never use the jumper on these disks; it makes aligned partitions misaligned again.

 

[SViscusi]

You can use the green drives in 'odd' combinations as well, like 4-disk RAID5 (3 data drives). This 'limitation' would only apply to RAID-Z on ZFS since it uses a variable stripesize.

Any stripesize is a multiple of sectorsize; so the stripe is always aligned. You just need to use aligned partitions and you can use the Green 5400rpm advanced format disks in any RAID config.

Also never use the jumper on these disks; it makes aligned partitions misaligned again.

Good to know. Zfs has been my only experience with RAIDing.

 

[Chaoticlusts]

I maybe misinformed but I don't believe ZFS is dead because of the oracle buyout anymore..I know it was for a period but due to various reasons people managed to save it and open source development on ZFS will continue..I maybe wrong on this but yeah..as someone mentioned above opensolaris is continuing essentially under a new name (the open source movement will always push forward ) that's not to say the other filesystem listed won't come out and supercede ZFS by simply being better (I know nothing about it) but as far as I know there will be continued development of the ZFS filesystem

 

[taltamir]

I maybe misinformed but I don't believe ZFS is dead because of the oracle buyout anymore..I know it was for a period but due to various reasons people managed to save it and open source development on ZFS will continue..I maybe wrong on this but yeah..as someone mentioned above opensolaris is continuing essentially under a new name (the open source movement will always push forward ) that's not to say the other filesystem listed won't come out and supercede ZFS by simply being better (I know nothing about it) but as far as I know there will be continued development of the ZFS filesystem

Wise conclusions that happen to be right on the money.

FreeBSD and several others have committed to continuing ZFS development.

Oracle killed the OpenSolaris "brand", but as you said, its open source. Development continues (with several corporate sponsors but under community control) under a fork named illumos that contains all the source code from opensolaris.

 

[sub.mesa]

2011 will be a good year for ZFS. And second ZFS v28 patchset is coming soon for FreeBSD, finalizing v28 boot support. In 2011, ZFS will be more available and sexier than ever.

Debian/kFreeBSD now also supports ZFS, meaning you can use an Ubuntu OS running FreeBSD kernel with ZFS. I've not tried this myself yet, but it should only be matter of time until this gets more popular.

OpenSolaris is dead though; but other projects continue to proper that are derivatives of OpenSolaris. So Oracle can't stop this, that also is the power of open source; the code is stronger than the companies wanting to control them. Oracle is an evil company i wouldn't trust anything to them, but they can't really influence how the CDDL-released code is used throughout other open source projects.

I'm not sure how things go after v28, but if we get v28 then we got a lot of cool features already. Even if ZFS never grew beyond this version, it would make one hell of a filesystem/RAID-engine-hybrid.

 

[Emulex]

Is ZFS patent-free? is it free from anyone suing them and putting the kibosh on the filesystem?

I'm serious.

And ECC - you do realize there are two ECC's. RAMBUS style and Unbuffered.

RDRAM is like rambus - serial ecc protected ADDRESS - how do you think they get 9 DIMM's stable 1000% on each socket - not using unbuffered address path that is noisy, and unprotected.

Unbuffered dimms (and unbuffered ecc) is like PATA for the address stream .Lots of wires - lots of energy load - lots of room for issues to come up - even unbuffered ECC has no protection on the ADDRESS path - this could be bad. it is bad. It only protects the DATA PATH. You think to tell the ram what you want to (GET,PUT) and then the DATA STREAM. Sure it's not likely to have errors in the address lines but it happens. When does it happen? When you start loading up with a crapton of dimm's. Same reason we ditched SCSI parallel and PATA - Point to point connections make more sense; less wires. easier to stabilize. Only a cost of some latency (the whole wide highway-slow versus narrow highway-super-bullet-train-fast).

Not having ecc from the cpu through bus to controller (raid/etc) to the actual drive path (sata,sas,pci) to the circuits on the drive itself - the whole shebang will seem dumb - until you shove 100 drives into a box running ZFS being pounded at 24x7x365 for years on end.

Everything works fine and dandy at burst load but push your load level up 100 (unix) and only the best will survive for years on end - been there done that. It's damn near impossible using commodity parts.

What i want to know is:
how the EFFFFFFFFFFF does qnap and drobo handle drives with all types of error correction (IE ignore all smart etc); how do we get our hands on that? ZFS is useless if i have a stack of AV drives with TLER=0s and the drives drop out every time an error occurs - someone tell me that answer - what o/s will handle its own ERROR CORRECTION (not detection). Think of SSD - it handles bad sectors - it uses reserve - which o/s with ZFS is going to give me that - let me reserve 10% of my ZFS raid and let the o/s handle all errors -ignore smart -ignore the drive -if the drive times out when i (the raid maker) decides to drop the drive otherwise keep on rocking.

Tell me that my friend - ZFS is useless without a subsystem that can use common drives like (drobo/qnap) - 1 AV 1TB, 1 TLER RE4; 1 BLACK; 1 samsung consumer; 1 HP Enterprise; 1 no name generic taken out of a external - make them all work together with the differences.

help me out bro's

 

[taltamir]

Is ZFS patent-free? is it free from anyone suing them and putting the kibosh on the filesystem?

you can sue someone even if you don't have the legal standing to win... that is, I could, if I had the money to hire the lawyers, sue someone for a patent I don't own. I would lose, eventually, but it is a tool often used by rich companies to shut down individuals or smaller companies.
For example, to shut down lik sang sony sued them in 5 countries at once. lik sang could not afford the lawyers and just went under. Sony neither won nor lost the lawsuits, and we will never know if they would have won or lost.

Oracle is suing google for using java (another SUN technology) in their android OS. Google is counter suing claiming that the patents are invalid, that they aren't infringing even if the patents are valid because it was released under a FOSS license, and so on and so on.

There is a very good chance of google winning, there is also a good chance of Oracle suing people for ZFS use since they are patent trolls. But the best they would achieve is the halting of development, in which case you could continue to use existing releases (since they don't magically disappear and it is NOT illegal for you to continue using) until BTRFS comes along. If they don't sue then development can continue.

 

[gonenutsbrb]

Alright, I will endeavor to explain.
ZFS stands for Zettabyte file system, it is a system for storing data to the drive. ZFS is actually a set of combined system, that include file system, partitioning, raid tools, and various other tools all designed to work together as a singular system.
There are other file systems such as NTFS (NT File System, the file system microsoft uses for windowsNT and above) and FAT/FAT16/FAT32 (file allocation table, older file systems from microsoft) and exFAT (a new derivative of FAT from MS designed specifically for USB thumb drives, since NTFS is bad for them and FAT32 is ancient).

File systems reside on partitions, which separate a drive into multiple chunks (which show up as separate drives in windows) or, rarely, combine multiple drives together into one partition (ZFS). Common partitions are MBR (master boot record, old and outdated) and GPT (general partition table, modern, lacks support) and the zfs file system (zfs's method can actually combine drives together at the FS/partition level unlike MBR and GPT).
So you can have a 2TB drive running MBR split into two 1TB partitions... one running NTFS and windows, and another running ext4 and linux.

RAID is a technique to create a "virtual drive" from multiple drives for various purposes (which are then partitioned and then used by the file system).
RAID1 is just storing an identical copy on all drives (write speed is the same as single drive, read speed is number of drives times speed of a single drive). All drives but one can fail and your data remains safe. Space is the size of only one drive (so 2x2TB drives give you 2TB of space only)

RAID0 is splitting the data between all drives (write and read speeds are the speed of a single drive times the number of drives.) if ANY drive fails ALL data is lost. Highly unsafe.
Space is just the sum of the drives (2x2TB gives 4TB of space)

RAID5 is using like raid 0, except 1 drive is reserved to contain the mathematical results of performing a calculation on the data on the other drives, such that you could recover the data if any one drive fails. write and read speeds can in theory go as high as the speed of a single drive times the number of drives minus 1... except in reality the complexity of the process means that only the fastest controllers (300+$ ones) can do that, most perform much slower than a single drive.
Space is the size of all drives but one... so 5x2TB drives will give (5-1)*2TB = 8TB space.
There are some problems with the implementation that can cause dataloss in a variety of conditions (write hole, reconstruction fails due to corrupt sectors, and more), and motherboard implementations are all terrible (slow AND unreliable).

RAID6 is raid5 only with 2 drives each containing a different mathematical result, allowing 2 drives to fail without data loss, and losing the space / speed of 2 drives.

RAID1+0 or 0+1 or 10 or 01 is when you make a RAID1 of a RAID0 array, or vice versa. reliability and speed is a mix of raid0 and raid1. pretty nice but requires lots of drives.

RAIDZ is a special implementation of RAID5 that fixes all the problems that can cause dataloss in RAID5 (except, of course, drive failure... I mean it fixes the write hole, the reconstruction errors, etc). However in order to do so it has to limit you to ZFS file system.
RAIDZ2 is the same thing for RAID6.

ZFS has several huge advantages over older file systems, foremost that it has unmatched data security, it is the only filesystem that prevents data corruption by saving checksums for all your data. As long as you have a redundant storage system (raid1, raidz, raidz2, raid10, or have set it to keep multiple copies of a file via the copies=N command) then it will repair data corruption on the fly.
NTFS and ext4 don't have that...
ZFS is the only file system that does that that is complete and commercially available...
Google has a custom file system (that actually sits on top of ext2, although they have begun work migrating it to ext4) that is their greatest trade secret and not available for sale. It also has robust data safeties.

The company that designed ZFS (Sun, which was purchased by oracle) released it as open source under the CDDL open source license. GPL forbids the use of software of certain license types, and as a result linux cannot implement ZFS as is, only via a userspace module (FUSE), which is how linux implemented NTFS, but this is a slow and long process.
Apple begun implementing ZFS but it seems to not go anywhere, and FreeBSD has ported ZFS (the BSD open source license called BSL permits the use of CDDL type software).
CDDL, GPL, and BSL all are legal licenses that allow the use of open source software under various conditions.

Anyways, due to its incompatibility with license of ZFS source code, and fear of nefarious intents from SUN, linux community began development of their own GPL alternative to ZFS called BTRFS (better file system), which is currently not ready for use. They could have just written ZFS compatible code from scratch instead, but they didn't trust SUN (or whomever buys them) to not later sue them for patent issues.
This turned out to be wise as SUN was purchased by oracle, who immediately discontinued opensolaris (a fork called illumos is now being maintained by companies that came to depend on it) and has sued google for using java (which is open source SUN product, and which SUN gave many guarantees that you couldn't be sued for using even if they wanted to) in their android OS.
So oracle is actively hostile to open source and is being predatory... not a good sign.

despite its uncertain future, ZFS will not magically disappear in a puff of smoke just because companies change hands and policies, you can still download and use, for free, existing implementation and software. and it is still the ONLY resistant to data corruption file system in the history of humanity available to everyone (the first is the google exclusive google FS). So enjoy it until BTRFS is ready.

Sorry for the bump, just reading through some old posts and want to make a few technical corrections and recommendations. RAID 5 does not use dedicated parity drives, neither does RAID 6. RAID 4 was the last level that does so. RAID 5 and above uses a distributed parity system. [See here for reference.]

Also, modern processors and motherboards can actually handle smaller RAID 5 systems pretty well. Granted, if you're designing and enterprise level server and have enterprise funds then a couple hundred dollar RAID card isn't going to be an issue, but for home users or small servers, onboard RAID (with a decent mobo) shouldn't see a massive performance hit.

Finally, if you do decide to use consumer "green" drives, make sure you update them to latest firmware to resolve any timing issues. Enterprise (aka RAID Edition, designated by "RE") drives tend to be the preferred drives that are recommended by IT professionals and data recovery experts for any server builds or mission critical systems. For the kicker, WD makes RE drives that are now "green" as well.

 

[taltamir]

Sorry for the bump, just reading through some old posts and want to make a few technical corrections and recommendations. RAID 5 does not use dedicated parity drives, neither does RAID 6. RAID 4 was the last level that does so. RAID 5 and above uses a distributed parity system. [See here for reference.]

I am aware, I was trying to keep simple and perhaps chose non ideal phrasing.

Also, modern processors and motherboards can actually handle smaller RAID 5 systems pretty well. Granted, if you're designing and enterprise level server and have enterprise funds then a couple hundred dollar RAID card isn't going to be an issue, but for home users or small servers, onboard RAID (with a decent mobo) shouldn't see a massive performance hit.

This is false, mobo raid5 is still utter crap. Not only in terms of speed but in terms of reliability.

Finally, if you do decide to use consumer "green" drives, make sure you update them to latest firmware to resolve any timing issues. Enterprise (aka RAID Edition, designated by "RE") drives tend to be the preferred drives that are recommended by IT professionals and data recovery experts for any server builds or mission critical systems. For the kicker, WD makes RE drives that are now "green" as well.

Timing issues are a huge problem with traditional raid controllers, but totally irrelevant in ZFS. ZFS doesn't care if you have TLER or Deep Error recovery, its not gonna lose its data and will work smoothly and perfectly with either option.

But your advice is sound if someone plans to go with a traditional RAID5 controller instead of using ZFS.