Build or buy a NAS?

[hasu]

You won't, without ECC, but the chances are small anyway. Hence ECC remaining a hot topic indefinitely.

Very nice explanation here..
https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/
(very informative series)

 

[exar333]

My QNAP TS-451 failed after a couple of months of usage. It was working fine then all of the sudden at 5AM one morning I was woken up to a failed NAS making loud beeps. It turns out the motherboard went bad. I had to RMA it and pay $20 for shipping. It took about a week to get it back and everything is working again but my confidence in the NAS's robustness is much lower than before. In fact, I purchased the NAS because my external Seagate drive died and took all of the data with it. Now I see why storing important data on the cloud is so important. I was planning to backup my important software projects locally to my NAS but now I will just host them on GitHub.

Honestly, the ease of setup on these consumer NASs is really nice. Things that would take tons of configuration on Linux just plain work. However, the hardware is very cheaply made and QNAP's QTOS is riddled with typos and shoddy interfaces. If this fails again I will roll my own with FreeNAS or a Linux distro.

Your example is actually a great reason to buy a consumer NAS. The beauty of a NAS is that, except for all your drives failing simultaneously (very unlikely), completely recoverable.

Don't put mission-critical items solely on a NAS without an accessible backup and you are good. Generally MBs dying is pretty rare due to the rather low system loads and temps for a NAS, plus they are engineered for 24/7 use with little/no maintenance.

I love to build computers, but it is hard to bead a purpose-built NAS consumer SKU. They are really feature-packed, affordable and low-maintenance.

 

[doubledeluxe]

Maybe at 1 or 2 drives but they get rather expensive for the larger capacities.

 

[Ranulf]

Maybe at 1 or 2 drives but they get rather expensive for the larger capacities.

I used to think that but after building my own, not so much. I'd say the break point is 4-5 drive bays.

That's about right - the motherboards are typically around $100 more, and the ram itself is just about double the cost.

(Not trying to take issue with your post or nothing, just explaining the pricing slightly differently for the benefit of those reading.)

Agreed Dave. I found over the last 6 months that at each ram capacity the price difference increased. At 4gb ecc ram is usually within $5-10 of non-ecc. At 8gb sizes, it jumps to 10-$20. At 16gb these days its $50-80.

 

[hasu]

Amazon price:

8 bay Synology $1000 to $1399
5 bay Synology $815
4 bay Synology $583

 

[dave_the_nerd]

Amazon price:

8 bay Synology $1000 to $1399
5 bay Synology $815
4 bay Synology $583

Those are prices without disks, aren't they?

My server cost about $550 plus the cost of disks, and it'll beat the crap out of one of those Synology units all day.

I understand the appeal of the appliances - really, truly I do. But the cost/benefit is SOOOOOO in favor of BYO if you have the technical know-how to do so... yikes. It's just painful to watch.

 

[hasu]

Those are prices without disks, aren't they?

My server cost about $550 plus the cost of disks, and it'll beat the crap out of one of those Synology units all day.

I understand the appeal of the appliances - really, truly I do. But the cost/benefit is SOOOOOO in favor of BYO if you have the technical know-how to do so... yikes. It's just painful to watch.

I completely agree with you. It is certainly cost effective to build your own NAS. More importantly the flexibility you get on your own device is unmatched. I have my own php and shell scripts that I use to identify duplicate files, compare folders, sort photos etc. I cannot use any of those in any ready-made NAS machines (speaking from experience with QNAP).

I think it is cheaper to buy if you only want a two drive NAS though, that goes for around $100.

 

[exar333]

I completely agree with you. It is certainly cost effective to build your own NAS. More importantly the flexibility you get on your own device is unmatched. I have my own php and shell scripts that I use to identify duplicate files, compare folders, sort photos etc. I cannot use any of those in any ready-made NAS machines (speaking from experience with QNAP).

I think it is cheaper to buy if you only want a two drive NAS though, that goes for around $100.

Agree on most of your points.

I still like consumer NAS units if you want basic NAS functionality with 'ready to work' right out of the box. Pop the drives in and use the apps/built-in functions therein. Its really easy and fast. You do lose a lot of flexibility that more power-users enjoy, but if you just want a media center or local drive, its plenty adequate for that.

 

[nicolachel]

Thought about the same problem for a while and decided to build my own due to 1. the steep price of entry for the 4+ bay ones and 2. knowing something about computer related stuff (I guess most ppl on this forum would know some).

What I got as a basis is a lenovo TS140 (i3) and it only cost about the same as a 2 bay Synology one would have ($199 + tax in some states such as CA).

It hosts 2 3.5 drives very comfortably (tool less main bays with forced air cooling from the front) and has potential for 3 more (with no forced cooling) space/connector wise.

Has full sized DDR3 slots, full height PCIe and PCI slots, of course this makes it a bigger box but the flexibility is simply unrivaled by commodity NAS you can get at any reasonable price (full sized RAID controller? No problem, 4X8GB ECC memory? No problem, 'powerful' and 'cheap' CPU upgrading options down the road? No problem), even if you get a multi-bay rack mounted SAN storage you'll have a very hard time beating its PCIe expansion capability and general flexibility.

The box idles at ~22 Watts (with the ODD but no HDD, 1 DIMM) under either Win 8.1(with its SSD) or Solaris 11.2(On a live USB stick) and is pretty quiet (you would have a hard time hearing it from 10 ft away, but it is no silent as purposefully built PCs like my main rig).

The only downside is that I'm kind of thinking about moving away from my aging main rig (i7 875K) and use it as a combined storage/daily use machine now

 

[nicolachel]

Also some thoughts on going beyond 5 drives (a pretty hard limit for TS140 unless you're into DAS which kind of works for you if you think you'll one day need to grab it and run but might not help with reliability in all other cases)

1. Take a look at TS440 which has 8x hot plug bays and hot spare PSU capability and starts at $299 now. But you need to at least think about the caddy + backplane + HBA trio as the $299 ones floating around for sure would not have those ready for 8 drives, and that means another $299 if you are very lucky; to use the hot spare PSU capability is probably another $200 (this is a ballpark number, I think you can get decent (such as 820W quiet) super micro stick PSUs for under $200 but I'm not sure about lenovo's pricing on those).

2. Another way to think about this is to get 2x TS140, you will then have 'hot spare' for literally everything at probably a very similar total power rating and get a quieter experience for cheaper than a 8 drive ready hot spare PSU TS440. But you need to more consciously manage the mirroring (write your own script to compare and sync every x hours or actually setup hot spare by using some server OS? the point here is that if you do not have 100% confidence in this mirroring link you will be forced to have redundancy locally in both boxes which quickly drops your disk space efficiency down to 25% or maybe a little better (33%) if you trust HW RAID5/ can bear soft parity's performance)

3. Get a big box (such as the 12x 3.5 drive ready PC-A75X) and put the guts of TS140 and a HBA in there, the power connector does seem a little bit weird on the MOBO but it can probably somehow be made to work.

I think you are looking at $2000+ or so for a performant commodity NAS if you are thinking about 8/10/12 bay which pretty much means the methods above pays for your 8*4TB drives.

 

[hasu]

I assembled a test setup with 6x3TB/7200RPM consumer grade drives (Toshiba and Seagate) in Raid-Z2 under Ubuntu 14.10. The setup is based on an old AMD CPU and 4GB-DDR2. With an empty pool I got 260 MB/s with no compression, and that speed remained more or less steady even when the pool is 96% filled (400GB remaining out of 11TB), I am still getting 247 MB/s. Bandwidth measured using dd command creating zero filled files each of size 100GB.

CPU may be the bottleneck. Load average around 7 on the two core cpu.

 

[dave_the_nerd]

I assembled a test setup with 6x3TB/7200RPM consumer grade drives (Toshiba and Seagate) in Raid-Z2 under Ubuntu 14.10. The setup is based on an old AMD CPU and 4GB-DDR2. With an empty pool I got 260 MB/s with no compression, and that speed remained more or less steady even when the pool is 96% filled (400GB remaining out of 11TB), I am still getting 247 MB/s. Bandwidth measured using dd command creating zero filled files each of size 100GB.

CPU may be the bottleneck. Load average around 7 on the two core cpu.

Probably this. Although that's still faster than dual-1GbE and means ZoL is probably "fast enough" even if it's not perfect.

The other potential concern with mixed drives in a RAID is that if one is a slower model it may hold the rest back. (Not sure how well ZFS accommodates this, or if it tries at all.)

 

[hasu]

Probably this. Although that's still faster than dual-1GbE and means ZoL is probably "fast enough" even if it's not perfect.

The other potential concern with mixed drives in a RAID is that if one is a slower model it may hold the rest back. (Not sure how well ZFS accommodates this, or if it tries at all.)

I have imported that pool to another machine running L5640 (on ASUS Rampage Gene III) and the speed went upto 370MB/s. Power consumption increased to 130-140W compared to around 100W on the old AMD.

 

[VirtualLarry]

What do people think about the HP mini/micro-server boxes? Those were pretty popular in the WHS days. Seeing as how WHS is discontinued (sadly), have those fallen out of fashion too?

I have a relative that was just telling me about their BlackArmor 4-bay NAS, and how it was running really slowly. I searched, and there were quite a few Amazon reviews mentioning that, as well as some threads discussing it. It looks like it's just generically under-powered for the job, or at least for a gigabit LAN.

So I was thinking of helping them out, with a gigabit NAS / HP micro-server that would actually perform well. What's the skinny on something small, that would have good performance? Or should I look into building an ITX server rig?

Edit: Their usage is backups, over the LAN and over wireless. (I advised my relative to get some AC1200 USB wifi adaptors like the ones I picked up. They don't do well for streaming music, but for file-transfers and web browsing, they are excellent.)

Edit: Maybe something like this?
http://www.newegg.com/Product/Product.aspx?Item=9SIA24G2178447

Edit: This one is cheaper:
http://www.newegg.com/Product/Product.aspx?Item=N82E16859108028

 

[CiPHER]

440 dollars for a Celeron processor and 2GiB of memory? Doesn't look like such a good deal to me honestly.

I prefer to get proper hardware for my money. And today it is really easy to build with few components. For example, i'm a huge fan of these kind of boards:

Own power supply (no ATX needed)
Passive cooling
Quadcore SoC processor with low 10W TDP
Up to 16GiB of UDIMM memory
PCI-express expansion slot
Onboard chipset SATA

All excellent items for a low-power 24/7 Home NAS. Because you avoid an ATX power supply, you will have no moving components (no fans) for the base system and cut idle power consumption in half. The Q1900TM pictured above does about 12 watts idle at the wall socket. That is still more than my own builds (8,4 watts idle for Core i5 3570 ivy bridge) but those involve undervolting which the Q1900TM doesn't support.

Similar boards:

ASrock Q1900TM (up to 2 disks)

ASrock Q1900DC-ITX (up to 4 disks)

SuperMicro X10SBA (up to 6 disks)

ASrock C2550D4I (up to 12 disks, overpriced)

ASRock E3C224D4I-14S (up to 14 disks)

Asus P9A-I/C2550/SAS/4L (up to 18 disks)

Your money invested in 'real hardware' will provide much better quality stuff you get per dollar spent. Plus it's fun too, and easy to get excited once you get into the cool stuff.

 

[hasu]

ASrock C2550D4I (up to 12 disks, overpriced)

Your money invested in 'real hardware' will provide much better quality stuff you get per dollar spent. Plus it's fun too, and easy to get excited once you get into the cool stuff.

Wow! This does look pretty neat! I didn't even know what to look for I was looking at Super micro ATX/LGA1366 just because I have a couple of L5640 CPU lying around. But these look pretty close in price to just that motherboard. Do you think the CPU is sufficient to max out the throughput? My configuration is going to be (Ubuntu Server 14.04 LTS):

6x3TB 7200 RPM
+ 2 SSD cache (most probably)
+ 1 SSD boot

In addition to hosting ZFS and file server, this machine may have to run one or two KVM guests also.

Hope this will bring down idle power consumption as well. ASUS Rampage Gene III + L5640 idles around 120W with 7 hard drives spun down.

 

[CiPHER]

The CPU on that board is Avoton (business-class Atom) and is either a quadcore or octocore (8-core). The 2550 is quadcore, the 2750 is octocore. But you pay a lot for those extra cores...

Generally, ZFS is not very CPU-intensieve, but it is very RAM-intensive. Bottlenecked RAM will show as CPU usage instead. But generally the CPU doesn't have that much to do, unless you enable features like deduplication/compression/encryption. RAM quantity will determine how much performance potential you can squeeze out of your disks. ZFS basically scales with RAM quantity.

You should know that the chipset only provides 6 SATA ports (4x SATA/300 and 2x SATA/600) while the other 6 ports are provided by two Marvell controllers. Those do not provide full bandwidth, but are still good AHCI controllers if properly supported. ESXi might have problems, for example. But generally BSD and Linux should work great with it.

The power consumption will be a tad higher; since the board you quoted has IPMI feature meaning you have management over power/console. It can be a nice feature, but the downside is that the BMC-chip that makes this possible, uses as much power as a whole computer system (8W). So the power consumption of such a board is doubled to 16W. If you use an ATX-power supply this may grow well into the 25-30W range.

ZFS-on-Linux is not nearly as good as on BSD operating system however, so your performance will be lower, less features and you may encounter some stability issues. But it is manageable if you really like Linux.

 

[hasu]

The CPU on that board is Avoton (business-class Atom) and is either a quadcore or octocore (8-core). The 2550 is quadcore, the 2750 is octocore. But you pay a lot for those extra cores...

Generally, ZFS is not very CPU-intensieve, but it is very RAM-intensive. Bottlenecked RAM will show as CPU usage instead. But generally the CPU doesn't have that much to do, unless you enable features like deduplication/compression/encryption. RAM quantity will determine how much performance potential you can squeeze out of your disks. ZFS basically scales with RAM quantity.

You should know that the chipset only provides 6 SATA ports (4x SATA/300 and 2x SATA/600) while the other 6 ports are provided by two Marvell controllers. Those do not provide full bandwidth, but are still good AHCI controllers if properly supported. ESXi might have problems, for example. But generally BSD and Linux should work great with it.

The power consumption will be a tad higher; since the board you quoted has IPMI feature meaning you have management over power/console. It can be a nice feature, but the downside is that the BMC-chip that makes this possible, uses as much power as a whole computer system (8W). So the power consumption of such a board is doubled to 16W. If you use an ATX-power supply this may grow well into the 25-30W range.

ZFS-on-Linux is not nearly as good as on BSD operating system however, so your performance will be lower, less features and you may encounter some stability issues. But it is manageable if you really like Linux.

Even with 25-30W the whole system should consume much less than my Gene III. I am getting good performance on on Ubuntu/ZFS with L5640 about 360MB/s.

Will this motherboard have enough juice to run two KVM guests? I was hoping that I can run some sort of Media Streaming Server on a VM.

 

[XMan]

Interesting info in here, I've been considering a QNAP TS-451 but I have a Core 2 system on a microATX board just laying around. Doesn't have a ton of SATA ports, though. Are there any reasonably priced PCiE SATA controllers that are known to work well with ZFS?

 

[Emulex]

Just get an LSI IR raid controller used on ebay. $50 8 ports of SAS/SATA compatible with most ZFS setups!

 

[CiPHER]

I recommend IBM M1015, which also has the LSI2008 SAS chip, because this controller can be flashed to IT-firmware. This means it will function in HBA mode instead of RAID mode. HBA is the term for a regular 'controller' (host adapter) without any RAID functionality. The controller will work in RAID mode as well, but has limitations and issues the non-RAID IT-firmware does not.

For example, the LSI controllers yield I/O errors on all disks if just one disk has a bad sector. This was fixed only days ago in FreeBSD 11-CURRENT branch. Not sure about Linux.

A card with LSI chip will also consume a fair amount of power (6 - 8 watts) when doing nothing, whereas AHCI controllers usually are 0,3 - 0,6 watts. The Intel controller will use zero watts with DIPM enabled, which is just awesome.

As Emulex said: these LSI cards can often be picked up for cheap, because they are sold by people who buy a ready-made server and do not need the controller and sell them off for cheap. That is why particularly the IBM M1015 is popular, i think, because there was quite some units sold on ebay. Very popular card for ZFS, but not my personal favourite. I prefer AHCI controllers instead. Those often have less bandwidth per port, though.

 

[dave_the_nerd]

Interesting info in here, I've been considering a QNAP TS-451 but I have a Core 2 system on a microATX board just laying around. Doesn't have a ton of SATA ports, though. Are there any reasonably priced PCiE SATA controllers that are known to work well with ZFS?

ZFS isn't the key to that equation, OS and supported drivers are.

Some of my hard drives are running off of a SI-PEX40064 (Syba) with a Marvell 88SE9215 chipset and it's been fine. I think I paid around $30 for it, it has four SATA ports. It is supported by FreeNAS / FreeBSD as well as Linux.

It's not fancy or high-performing, but it's fast enough. (The HDDs are generally going to be the bottleneck.)

The more expensive RAID cards will get you better overall performance (more PCI-E channels) and more ports, but their advanced features (RAID) are wasted, since you're using ZFS for that.

If you're in the "cobble it together from as many used parts as possible" club, then a $30 dummy SATA controller with no RAID functions is adequate. (Recycling an mATX C2D machine makes it sounds like that's where you're at.)

 

[XMan]

I recommend IBM M1015, which also has the LSI2008 SAS chip, because this controller can be flashed to IT-firmware. This means it will function in HBA mode instead of RAID mode. HBA is the term for a regular 'controller' (host adapter) without any RAID functionality. The controller will work in RAID mode as well, but has limitations and issues the non-RAID IT-firmware does not.

For example, the LSI controllers yield I/O errors on all disks if just one disk has a bad sector. This was fixed only days ago in FreeBSD 11-CURRENT branch. Not sure about Linux.

A card with LSI chip will also consume a fair amount of power (6 - 8 watts) when doing nothing, whereas AHCI controllers usually are 0,3 - 0,6 watts. The Intel controller will use zero watts with DIPM enabled, which is just awesome.

As Emulex said: these LSI cards can often be picked up for cheap, because they are sold by people who buy a ready-made server and do not need the controller and sell them off for cheap. That is why particularly the IBM M1015 is popular, i think, because there was quite some units sold on ebay. Very popular card for ZFS, but not my personal favourite. I prefer AHCI controllers instead. Those often have less bandwidth per port, though.

My purpose would just be general archival and streaming of HD MKVs or Blu-Ray ISOs, so not huge bandwidth, around ~30MBps max generally. What AHCI controller do you recommend?

 

[hasu]

ZFS-on-Linux is not nearly as good as on BSD operating system however, so your performance will be lower, less features and you may encounter some stability issues. But it is manageable if you really like Linux.

On Ubuntu, I am getting 360-380MB/s (1GB to 50GB files). Three simulataneous sessions yielded about 130 MB/s each. Do you think the performance can be better than that on FreeBSD? Also, is there a way to enable encryption on Ubuntu/ZFS?

Edit: I tried FreeNAS, and I am getting 2GB/s write speed (created a 50GB file).
52 428 800 000 bytes
2 067 700 748 bytes/s

Edit2:When the compression was turned off I got the following results in line with that of Linux
104 857 600 000 bytes transferred in 288.559959 secs (363 382 364 bytes/sec)

 

[XMan]

So after doing some research, it seems like the spare motherboard I have doesn't support ECC RAM, and a max of 4GB at that.

Maybe QNAP it is . . .