Build or buy a NAS?

[Cerb]

I could go lower through a box store... but the price I get on the i3 would get me Celeron-ish ranged CPUs at a box store.

I'm not above going ECC memory. The reason I looked at server motherboards was feature sets such as dual+ ethernet ports, more SATA ports, SAS ports, etc.

Iff you want to go that route, look at Lenovo TS140 and TS440 systems on Amazon, from Computer Upgrade King, and add your own drives and RAM (or use the 4GB typically included). The TS140 lacks much internal expansion for drives. You can get caddies for the TS440 for around $15-20 per, if you look around Amazon and eBay (that's the main negative of such servers from big OEMs). You can add DAS if you want to go bigger later.

If you're not going to be using SSDs, just make sure you have enough PCIe bandwidth for the drives, when it comes to adding controllers. Used LSI cards that can be flashed to HBA, or otherwise configured as HBAs, or that are only HBAs, are good bets, since they tend to be x8, which offers tons of bandwidth for spinners, especially when limited by 1Gbps for file data. With a little researching, you should be able to find 8-port cards (may need break-out cables) for under $100 any day, and under $50 if you spend more time and effort.

For a home NAS, 4GB should work, and 8GB will be bordering on overkill (4GB might even be overkill, actually).

As already said, dual NICs are going to be difficult to make use of, and IMO won't be worth it. To make good use of it, you'll need to either have multiple VLANs, or multiple computers able to team them, with switches to match.

Here is an example. $300, with more CPU than you need, but just needs drives and caddies for up to 4, with room for 4 more (needs Lenovo 03X3703 and 03X5999). Not super duper cheap, but quite good compared to other ways of building with hot swap, and cheaper than any other big OEMs.

Another way would be to just get a Fractal Design Node case, and fill it up with non-hot-swap drives, with a Pentium or i3. If you want ECC, the motherboard is what will get you, there, cost-wise.

 

[Lil'John]

The Lenovo ThinkServer is kind of tempting.

How critical is ECC memory for FreeNas, Nas4Free, etc?

I may have a "retiring" computer that I might take advantage of:
CPU : i5-4570S (65W)
MB : ASRock B85M-ITX

It is my current "media" machine that I watch movies from direct. The case is the right size for my network cabinet. The two downsized are the 4 sata ports and max of 16GB ram(currently at 8)

I thought I had a taker on it when I upgrade to a Broadwell i3 NUC

 

[Cerb]

No more or less critical than with any other software for the job. You just don't get the full protection within the system that ZFS may offer without ECC. OTOH, your memory may well never have an error, throughout its entire life. The thing about ECC is that without it you won't know if an error ever occurred or not.

For business, it's a must have, being dirt cheap in the scheme of things. For a media box, it's not an easy judgment call.

 

[Lil'John]

Cerb, thank you for the explanation of the ECC memory. The quick scans I've done on the NAS "OS's" made it feel like if you didn't have ECC memory, the software wouldn't work.

From a "cheap" CPU standpoint, I don't think any of the ones I have access to support ECC: i3-4370, i5-4570T, i5-4690s, and i5-4590. And the two that would support ECC are a bit overkill for a NAS: E5-2630v3 and E5-2695v3.

For the content that will be on the NAS(CD and DVDs that I own), it isn't a loss if the machine dies except for a bit of time

 

[Emulex]

Celeron, pentium, and some I3's support ECC memory with the appropriate server/workstation class motherboard! So yeah cheap cpu's do support ECC, cheap motherboards/ram not so much!

ECC errors do register with the OS on server grade motherboards, i've had servers register thousands of ECC corrections per minute before ESXi disabled the use of said memory due to a bent pin on a motherboard (DL380 Gen8)! So you might not realize it but once corrected (or uncorrectable) they do have ability to log /notify you!

 

[CiPHER]

ECC is a hot-debated issue for ZFS, especially on the FreeNAS forums very spartan opinion exists and if you do not have or use ECC, the people there will look down on you.

In reality though, you do not really need ECC for a home NAS. It kills the End-to-End data security that FreeNAS is hot about, but ZFS can correct at least some of the corruption cause by RAM bitflips, whereas other filesystems can't.

Better yet, you can even detect memory corruption without ECC memory. Sure, you won't know the exact number of bitflips as would ECC counters tell you. But you can see what checksum errors occurred on the ZFS pool. A pool with checksum errors randomly spread over the member disks is a sure sign for a faulty RAM module. This happens so often and never have i experienced a broken pool due to RAM corruption from any of the people that i help.

I have also done tests myself with a faulty RAM module which yielded tens of thousands of errors in MemTest86+. I let ZFS scrub the pool with the faulty memory. Then after i scrubbed again with good memory, and checked the checksums of some files manually. The result was that while ZFS is scrubbing the pool, it thinks many times there is corruption on one of the disks - while in fact there isn't. This causes ZFS to correct the damage and you can see checksum errors in the 'zpool status' output. But ZFS is actually corrupting the disk data, and not correcting corruption.

However, due to the RAID-Z redundancy, all damage was just as easily corrected after scrubbing with good RAM again. And none of the files were permanently damaged. If this happens with tens of thousands of RAM bitflips, then surely all the panic about one or two bitflips per year isn't worth it. The worst that can practically happen is that a file fails because its checksum was created while being affected by RAM corruption/bitflips. In this case you have a dead file. But you know which file. No corruption will ever be undetected for long with ZFS.

 

[Lil'John]

Some follow-up questions with regards to ZFS and NTFS with regards to transferring content and adding drives. I apologize if the following is covered in one of the free NAS documents... I will be spending some time this weekend scouring those

The current system has three 1TB drives with NTFS and movie content on all. About 600GB on each drive.

Can any of the free NAS's read NTFS file system? I believe the term is mounting the drive.

Does ZFS require any special processes for adding more drives?

My thought is to run a free NAS on a flash drive, add a fourth 1TB drive using ZFS, and mount the three NTFS drives. I would then transfer the content from one of the NTFS drives onto the ZFS drive. I would then "format" the NTFS drive and add it to the ZFS system. Then repeat.

I'd rather do the above method than install all the drives onto another system and do large network transfers

 

[CiPHER]

You can mount them read-only. But the best would be to transfer via the network so that a Windows computer reads from the NTFS; because things like journaling is skipped.

Adding more drives to ZFS is different than other storage platforms. For instance, you cannot add a single drive to a 3-disk RAID-Z pool and get a 4-disk RAID-Z pool. But you can add one more disks to a ZFS pool under different circumstances. Simply put, you add vdevs or 'arrays' to the pool. Each time a single disk, a mirror of at least 2 disks, or a raid-z of at least 2 disks.

 

[Cerb]

From a "cheap" CPU standpoint, I don't think any of the ones I have access to support ECC: i3-4370, i5-4570T, i5-4690s, and i5-4590. And the two that would support ECC are a bit overkill for a NAS: E5-2630v3 and E5-2695v3.

Quite a few Pentiums and i3s support ECC. But, they need to be paired with a motherboard that also supports ECC (because ECC support is a way for Intel to segment the market).

 

[Lil'John]

ZFSguru is the least mature of all projects and has the least number of features. But it might be the most user friendly and 'easiest' ZFS distribution suitable for novice users, and has very few restrictions. For example, FreeNAS doesn't allow you to use the system disk for anything else. So using an SSD to install FreeNAS to is kind of a waste.

Do you recommend running the NAS OS on a USB thumb drive? If so, do you have a recommended brand?

From the reading I've done, it appears anything over 2GB starts having issues with booting.

 

[hasu]

It will only protect against memory corruption, which is rather rare. Without ECC, ZFS can detect all and can correct at least some of the corruption caused by RAM bitflips. If it couldn't fix it, at least you know which files are affected. Also note that - even with ECC - you are still not protected against 3 bitflips; only against 1 (correction) and 2 (detection).

Ok.. I will take your word for it, even though I am not convinced yet . I am planning to build a Raid-6 (6x3TB) NAS for an effective storage of 12TB to consolidate all my data including that from old drives. I already have plenty of DDR3 in stock so using 12GB for NAS is not a big deal. I was reading more about the need for ECC, and was looking for used ECC to keep costs within limits and then again I will need a new motherboard as well. Budget is already way stretched and is really going out of hand.

So if I just go with ZFS and RaidZ2, I should be able to safeguard against data corruption caused by bad memory, correct? That sort of extra safety is only required for about 3TB of data (family photos, videos and documents). Remaining data is mainly enternainment videos, movies and music, which I don't care if I loose (I still keep most of the DVDs and CDs, and you can always buy if you ever really miss any of that).

I have one question though, how will I know if a bad memory has corrupted one of my files while saving a document? The workstation sends the data and NAS receives it intact, but during the process it gets corrupted with in the RAM, so how will ZFS know that the data that is going to be written to the storage is already modified after receiving from the client?

 

[CiPHER]

So if I just go with ZFS and RaidZ2, I should be able to safeguard against data corruption caused by bad memory, correct?

That means you have some ability to correct damage (corruption) caused by RAM bitflips. But running without ECC means your applications could get faulty data and on-disk data is (temporarily) corrupted. The latter is not that bad; the RAID-Z2 protection means two blocks may be corrupted if the other ones are good. You would need RAM bitflips at the same time at very specific portions of your RAM to break the double parity protection.

When creating files or modifying existing files however, the checksum will be re-created and updated. If a RAM bitflip were to corrupt the result of this calculation, then even if all the on-disk data is good and not corrupted, ZFS would reject all those blocks. This is because each block would not conform to the checksum, which was corrupt to begin with.

This all is nasty for companies. But for home users, not so much. Short term you might have corruption, but next-time that data is touched, ZFS would detect it and correct it when possible.

And while some corruption would not be instantly detected, it will be detected anytime later. Meaning, you will not have silent corruption on the long-term, only the short-term because applications may be affected.

So what this means is that one day you might see zpool status -v output say: data errors: 1 followed by the filename in question. Byebye file! That is pretty-much a worst case scenario for non-ECC RAM. The better case is that you will have no corrupt files but only see some checksum errors on the disk.

non-ECC RAM doesn't produce bit errors all the time. My tests were done with really bad modules producing tons of errors in MemTest86+. So the risk of one or two bitflips each year is probably very much acceptable to home users.

And remember: just ECC on the ZFS server is not enough! You need it on your clients too. All good that your ZFS box will never corrupt, but if your workstations deliver corrupt data to begin with, all the protections that ZFS employs are futile. I actually went as far saying that those considering ECC for their ZFS server, should consider applying it on their workstations first. They would have zero protection to detect/correct, while the ZFS server would have 100% detection chance and a decent chance to correct the damage. Not all will agree, i'm sure. But you get the point: with just ECC for the server you aren't done yet.

 

[hasu]

And remember: just ECC on the ZFS server is not enough! You need it on your clients too. All good that your ZFS box will never corrupt, but if your workstations deliver corrupt data to begin with, all the protections that ZFS employs are futile. I actually went as far saying that those considering ECC for their ZFS server, should consider applying it on their workstations first. They would have zero protection to detect/correct, while the ZFS server would have 100% detection chance and a decent chance to correct the damage. Not all will agree, i'm sure. But you get the point: with just ECC for the server you aren't done yet.

You are right! I thought about that too (ecc on client). Having ecc on every client is not easily achievable (think tablets) and may not be done even in business environments. So I am going to go ahead with my existing desktop motherboard with the best RAM modules that I can find (in my possession) to build the NAS. May be later someday I can upgrade the system with server grade motherboard and ECC ram.

A couple more questions:

1. If I build a new NAS can I just import the existing ZFS array (6 drives) as it is? (RaidZ vol s exlusively for data).

2. Can I install Ubuntu as the base OS and create a VM for NAS (FreeNAS, Solaris or somthing else)? I never installed ZFS before, so a lot to read.

3. Any comments on ZFS on Linux?

4. Other than FreeNAS what other OS is best for ZFS?

Thanks for all your comments in these forums. They are very informative, an immensely helpful!

 

[CiPHER]

Having ecc on every client is not easily achievable (think tablets) and may not be done even in business environments

One good thing is that ZFS allows easy use of snapshots. So you snapshot your data once a week or even once a day/hour/second. And you can go back in history any time you want. So if your tablet corrupted your data somehow, you could revert that. Assuming those were static files.

1. If I build a new NAS can I just import the existing ZFS array (6 drives) as it is? (RaidZ vol s exlusively for data).

Yes, you can 'import' a pool that the system does not know about. This is a protection to prevent two ZFS instances from accessing the same disk, which may happen in some exotic configurations.

But you cannot import a ZFS pool on a system that doesn't support the ZFS features that the pool was created with. So an older version of the same product may not work, and not all ZFS platforms are optimally compatible with each other. I strongly recommend you create your pool with all ZFS features disabled and only manually enable the features you like, such as LZ4 compression (*) which is awesome. By leaving the more new/recent features disabled, you make sure your pool will be cross-compatible between virtually all major ZFS platforms. As all of them support ZFS v5000 by now.

(*) LZ4-compression is awesome because it is essentially 'free'. It is ultra-fast with multiple GB/s and it has a special feature that will stop compressing when an incompressible file/data block has been found, and just stores it without compression instead. This makes LZ4 so cool that you can enable it pool-wide and have fair good compression at extremely low overhead in terms of CPU/latency. LZ4 was one of the first features to make it into the v5000 OpenZFS.

Can I install Ubuntu as the base OS and create a VM for NAS (FreeNAS, Solaris or somthing else)? I never installed ZFS before, so a lot to read.

Ubuntu itself can also do ZFS, but you would be without a web-interface. Without anything to guide you. Using a ZFS platform inside a virtualised environment is not recommended. Some use ESXi which is a decent hypervisor, but the trick is always to give ZFS direct access to the disks. For that you need CPU VT-d support which only the high end consumer CPUs get, and some high end 'K' versions even go without.

You can test any of the ZFS solutions inside a VM easily. Just try Virtualbox and hop in a .iso and one of more virtual disks and give it a try!

The ZFS platforms generally are much easier to install than base operating systems like FreeBSD/Solaris/SmartOS or even Linux. ZFSguru, for instance, provides a web-interface ready after booting the livecd. So all you need to do is use your web-browser to select the installation options. This is a good start for a beginner. I do not advise a solution like FreeBSD/Linux/Solaris to start with. Only experienced users should want that. If you want to learn the operating system, you can do that with the ZFS platforms as well. There are also many small things and tweaks that can be better or even important, which a ZFS distribution will take care of.

By the way, i enjoy helping people with their ZFS builds. So feel free to private message me should you have small questions during your build. Or even a live IRC-chat is possible. Let me know, and good luck!

 

[hasu]

You can test any of the ZFS solutions inside a VM easily. Just try Virtualbox and hop in a .iso and one of more virtual disks and give it a try!

The ZFS platforms generally are much easier to install than base operating systems like FreeBSD/Solaris/SmartOS or even Linux. ZFSguru, for instance, provides a web-interface ready after booting the livecd. So all you need to do is use your web-browser to select the installation options. This is a good start for a beginner. I do not advise a solution like FreeBSD/Linux/Solaris to start with. Only experienced users should want that. If you want to learn the operating system, you can do that with the ZFS platforms as well. There are also many small things and tweaks that can be better or even important, which a ZFS distribution will take care of.

SmartOS sounds very attractive but installation failed on a KVM. I will try that on a real hardware soon.
With SmartOS, I can create KVM machines if needed.

 

[Cerb]

I have one question though, how will I know if a bad memory has corrupted one of my files while saving a document?

You won't, without ECC, but the chances are small anyway. Hence ECC remaining a hot topic indefinitely.

 

[Cerb]

3. Any comments on ZFS on Linux?

It's slow, compared to FreeBSD, whether you use RAID or not, and most efforts are going into ironing out BTRFS, instead. Whether that means it will make ZFS a bottleneck, instead of GbE, is an open question. However, you should be fine with HDDs in a mirroring RAID, and doubly so if can spare an SSD for read caching, if it's a bit sluggish. Performance is probably fine all-around for a home NAS, though.

 

[darkfalz]

I've been looking at the Thecus 7 drive model, and 7 6 TB drives in RAID 5. My concerns are if the unit dies, needing to get another of the same model. My other option is making a FreeNAS or even Windows based NAS but finding a uATX board with more than 6 SATA ports seems a near impossible feat.

 

[hasu]

I've been looking at the Thecus 7 drive model, and 7 6 TB drives in RAID 5. My concerns are if the unit dies, needing to get another of the same model. My other option is making a FreeNAS or even Windows based NAS but finding a uATX board with more than 6 SATA ports seems a near impossible feat.

For that many drives, raid-6 or raid-Z2 might be a safer choice with its double parity (or two redundant drives). If a drive fails you can replace it with a drive of the original capacity or larger. mdadm or ZFS are software raid implementations independent on the hardware you use. Since the raid array can be imported to another machine, a motherboard failure need not be a concern. For additional SATA you can get 4 port PCI-E card. That being said, some other catastrophic accident (say phyiscally destroyin the setup, fire and water damage etc), can result in total data loss. So you will have to consider an appropriate backup strategy for that much amount of data.

 

[Hulk]

What I'm suggesting isn't a NAS but it might fit your needs.

I've had a 3TB Seagate external drive connected to my Netgear R7000 router for about 8 months now and I have to say for serving media files to other computers and Smart TV's it's been flawless. It only draws about 7 watts. I have all of the files backed up on my main computer and HTPC on a weekly basis using FreeFileSync so no backup worries.

Hardly any setup, low power usage, and easy to replace.

Just a thought.

 

[Ranulf]

If you want hot swap drive bays, this case is pretty good in my experience:

http://www.amazon.com/Silverstone-Tek-Mini-ITX-Computer-DS380B/dp/B00IAELTAI

I have an i3 4330 with an asrock c226 board and 16gb ecc ram running in it. SFX psu required though.

I started out with a node 304 which is nice but I decided I wanted hot swap and theoption for more than 6 drives. That said I think at this point I wish I had gone with a 4 bay Synology or Qnap device and an Unraid system in an atx or matx case. Along with that I'd have a small itx system (like the node 304 case) as a 2nd or 3rd backup with a linux OS so I could just plug it in wherever and use it as a normal computer easily.

Also, the Freenas forum folks do themselves no favors with their ecc zealotry. Mostly because I'd read posters there saying things like "it doesn't cost that much more! How much do you value your data!" to home users building these systems from spare parts. ECC ram costs $100-200 more on average when you factor in mobo/chip requirements.

 

[hasu]

It's slow, compared to FreeBSD, whether you use RAID or not, and most efforts are going into ironing out BTRFS, instead. Whether that means it will make ZFS a bottleneck, instead of GbE, is an open question. However, you should be fine with HDDs in a mirroring RAID, and doubly so if can spare an SSD for read caching, if it's a bit sluggish. Performance is probably fine all-around for a home NAS, though.

How about kFreeBSD from debian?

I tried to install SmartOS and it did not work for me. I am trying to find a matured OS that can host ZFS and KVM.

Edit: I could not succesfully install any "native ZFS" OS. FreeNAS won't work for me because I want to run VMs on the same machine. Finally what worked for me was Linux ZFS on Ubuntu as shown here http://www.howtogeek.com/175159/an-introduction-to-the-z-file-system-zfs-for-linux/. With 4 hard drives in Raid-Z2, I get 201MB/s on a brand new volume. This figure may have some influence of caching. I have 4GB ram on the machine. But when I give oflag=direct dd fails to open a file on zfs volume

Update: I built a test setup with 6x3TB/7200RPM consumer grade drives (Toshiba and Seagate) in Raid-Z2 configuration under Ubuntu 14.10. Based on an old AMD CPU and 4GB-DDR2. With an empty pool I got 260 MB/s with no compression, and that speed remained more or less steady even when the pool is 96% filled (400GB remaining out of 11TB), I am still getting 247 MB/s. Bandwidth measured using dd command creating zero filled files each of size 100GB.

 

[WaitingForNehalem]

My QNAP TS-451 failed after a couple of months of usage. It was working fine then all of the sudden at 5AM one morning I was woken up to a failed NAS making loud beeps. It turns out the motherboard went bad. I had to RMA it and pay $20 for shipping. It took about a week to get it back and everything is working again but my confidence in the NAS's robustness is much lower than before. In fact, I purchased the NAS because my external Seagate drive died and took all of the data with it. Now I see why storing important data on the cloud is so important. I was planning to backup my important software projects locally to my NAS but now I will just host them on GitHub.

Honestly, the ease of setup on these consumer NASs is really nice. Things that would take tons of configuration on Linux just plain work. However, the hardware is very cheaply made and QNAP's QTOS is riddled with typos and shoddy interfaces. If this fails again I will roll my own with FreeNAS or a Linux distro.

 

[dave_the_nerd]

ECC ram costs $100-200 more on average when you factor in mobo/chip requirements.

That's about right - the motherboards are typically around $100 more, and the ram itself is just about double the cost.

(Not trying to take issue with your post or nothing, just explaining the pricing slightly differently for the benefit of those reading.)

 

[hasu]

That's about right - the motherboards are typically around $100 more, and the ram itself is just about double the cost.

(Not trying to take issue with your post or nothing, just explaining the pricing slightly differently for the benefit of those reading.)

Newgg price for LGA1366 server motherboard and 12GB ECC UDIMM is about $360. Used CPU from eBay for about $60 may be OK. I never used any server CPUs after LGA1366 Xeons so I didn't check out other options. I myself would prefer a low power Motherboard/CPU. As it is now my experiemetal setup draws about 100-120W from wall.