Good RAID 5 Solution

[Weatherton]

I am planning to upgrade my computer, mainly so that I can get solid storage solution.

I currently own 8 500GB hard drives and want to use those in my machine in some form of RAID, probably RAID 5 with 4 drives a piece. I then want to use two single 1.5TB hard drives external to the system for incremental backups.

What is a solid, inexpensive hardware RAID solution for RAID 5? Do I dare consider using on-board Intel ICH10R?

 

[tcsenter]

Hardware RAID 5 with support for eight drives starts at $300 and goes-up quickly from there.

How's software RAID sounding now?

 

[Weatherton]

Could you describe some software RAID solutions? Are you talking something that is built into the mobo? I was considering the Asus P5Q-E:

http://www.asus.com/products.a...l1=3&l2=11&l3=709&l4=0

http://www.newegg.com/Product/...ntel-_-ASUS-_-13131296

I was thinking about it some more and this is what I would like to have:

Boot: RAID 1; 2x250GB
Storage1: RAID 5; 4x500GB
Storage2: RAID 5; 3x500GB
Capture: 1x500GB

External backups:
1x300GB for Boot (already own)
1x1.5TB for Storage1
1x1TB for Storage 2

Given these hard drives, what would be my best solution?

 

[VirtualLarry]

I own several PNY S-Cure (search for that on NewEgg) RAID cards. They use the Netcell Revolution chipset, have 5 ports (SATA1 speed), hardware RAID chipset. Driverless, the card appears as an IDE drive controller to the OS, so it uses standard IDE drivers. The RAID GUI app only runs on XP (used for configuring the array).

I haven't taken them out of the box yet, so I can't give you any word on performance or compatibility. They are cheap (<$40) though. I'm going to be using them in one of my next builds, my storage server is going to have three 5-drive RAID3 arrays. (The Revolution does "RAID3", and requires either three drives, or five drives, not four.) Performance will obviously be limited to PCI bus speeds, but for me that's enough.

You could use two cards, and have either two 3-drive arrays, or one 3-drive and one 5-drive array.

Or, you could buy this for $250. 8xSATA2 ports, hardware RAID, requires PCI-E x4 slot.

 

[XR250rdr]

I'm running a Dell PERC 5/i (OEM LSI 8408e) in my file server and it works quite well. I have a 4x500GB RAID5 array running on it and my OS drive is running off the ICH9R.

 

[tcsenter]

Originally posted by: VirtualLarry
Or, you could buy this for $250. 8xSATA2 ports, hardware RAID, requires PCI-E x4 slot.

The High-Point is hybrid RAID (a.k.a. intelligent software RAID), not true hardware. It has a fairly modest XOR offload engine (HPT601) but no IOP, thus making it both OS and host CPU dependent. Its better than software RAID, to be sure, but for the purists out there, its not quite accurate to say its hardware RAID in the same category as the High-Point RR3xxx series with dedicated IOP.

The chip hiding under the heatsink is just an Intel PCI-X to PCI-E bridge chip.

 

[Weatherton]

If possible, I would like some suggestions for hardware RAID, just so I can get a good comparison. I would want this card to run the following two configurations at the same time:

Storage1: RAID 5; 4x500GB
Storage2: RAID 5; 3x500GB

P.S. A lot of these hardware RAID solutions mention battery-backup. What is the importance of that? What if I want to unplug my computer and put it in storage or something?

 

[RebateMonger]

Originally posted by: Weatherton
P.S. A lot of these hardware RAID solutions mention battery-backup. What is the importance of that? What if I want to unplug my computer and put it in storage or something?

The battery backup supplies the write cache on the controller card. So if the power goes down on the server, whatever was to be written is saved until the battery runs out. This is important when using a write-back cache, where the RAID card tells the system that a write has been completed BEFORE it's actually been completed.

IBM discussion of write-cache mode for logical drives.

 

[mooseracing]

Originally posted by: XR250rdr
I'm running a Dell PERC 5/i (OEM LSI 8408e) in my file server and it works quite well. I have a 4x500GB RAID5 array running on it and my OS drive is running off the ICH9R.

Best and cheapest way to go, you can get these cards for $100-200 on ebay or cheaper. Run at the performance of a $300 or more card.

If you need more cache step up to a Perc6 as it allows for 512mb cache.

 

[VirtualLarry]

Originally posted by: tcsenter

Originally posted by: VirtualLarry
Or, you could buy this for $250. 8xSATA2 ports, hardware RAID, requires PCI-E x4 slot.

The High-Point is hybrid RAID (a.k.a. intelligent software RAID), not true hardware. It has a fairly modest XOR offload engine (HPT601) but no IOP, thus making it both OS and host CPU dependent. Its better than software RAID, to be sure, but for the purists out there, its not quite accurate to say its hardware RAID in the same category as the High-Point RR3xxx series with dedicated IOP.

The chip hiding under the heatsink is just an Intel PCI-X to PCI-E bridge chip.

Thanks for the correction. I didn't know such an expensive card was still essentially software raid.

 

[QuixoticOne]

You complain about RAID-5 complexity and cost, and yet you will surely get either or both of those things (according to some perceptions of what is complex and what is costly) if you're looking for relatively high end multi-port hardware based RAID controller cards and integrating them into a non-dedicated computer system.

The most inexpensive way (in some analyses) to proceed is to get enough drive controller ports *somehow* to interface to all of your drives independently in JBOD mode. For instance here is a very basic 4 port SATA card for $36 shipped:
http://www.newegg.com/Product/...16816132013&Tpk=RC-209

Then you have the option to use its RAID-0 or RAID-1 or whatever capabilities assisted by an OS based software layer driver since it is not a completely autonomous / intelligent / high end RAID card, but it is 1/4 or less the cost of one.

If you do not like its capabilities for mirroring in pairs or whatever then use it in JBOD (non-RAID) mode and just have it control the discs independently. They you must use your OS to generate a software based RAID implementation that is suitable for your needs. Typically your criteria for reliability, stability, security, administration ease / flexibility, et. al. would dictate your choice of OS and typically that would result in you not sharing the function of the file server as a general use workstation, and typically you would not choose to run Microsoft Windows Home edition software.

If you want the controller to "only" support mirroring as an assurance of "impossibility" of configuring it incorrectly, I can sympathize, but I think it is unrealistic to expect solutions to this that are "impossible" to fail or be administered incorrectly. With anything except dedicated purpose built mirroring drives / storage SYSTEMS (not cards) you'll always have the opportunity for some configuration setting or poor sysadmin circumstance to interfere with proper and reliable functioning.

Perhaps a mirrored storage SYSTEM might offer more simple (lower complexity) management as well as higher reliability against sysadmin errors :
http://www.newegg.com/Product/...x?Item=N82E16822136282
http://www.newegg.com/Product/...x?Item=N82E16822136281

 

[Weatherton]

QuixoticOne, I appreciate the excellent post.

I think if I explain my reasoning for RAID, that may help everyone here help me to arrive at a good solution.

1. I have a lot of data I want to protect, I do a lot of video editing and the files are large and precious
2. I had a close call where I had thought I'd lost three hard drives of data, I had Windows suddenly decide that my main drive (not raided) and my archive drive (2 in RAID 1). I was able to get my data back using a $50 scanning utility. I'm still not sure what happened here, but it has scarred/scared me.
3. I have had several hard drive failures in the past two years. These failures have resulted in strange system errors and hangs, blue screens and general instability.

In light of these three things, I have 3 goals:
1. Develop a large, safe data storage strategy.
2. Avoid system instability due to storage.
3. Never store important data in a single location.

My performance needs are:
1. Storage space first, performance second.
2. Significant processing needs for rendering video. I don't want my storage solutions to be weighing heavily on my rendering capabilities.

My current plan is this:
1. Use RAID to avoid system instability by allowing faulty hard drives to be replaced before crashing my OS
(***Is this correct? Will RAID with parity/mirroring prevent hard drive failures from affecting my OS? What will occur as a hard drive dies, how would I be notified and would the OS have any problems?)
2. Use external storage that is not always attached to mirror all drives.

My actual planned configuration is currently this:

Option 1: RAID 1

Computer eSATA Enclosure(s) - mirror System disks w/ Acronis TrueImage 11
2x250GB RAID 1 for Boot 300GB drive (store one fresh install image and one current)
2x1.5TB RAID 1 for data 1.5TB drive
2x500GB RAID 1 for capture 500GB drive
2x500GB RAID 1 for data 500GB drive

Option 2: RAID 1 and 5

Computer eSATA Enclosure(s) - mirror System disks w/ Acronis TrueImage 11
2x250GB RAID 1 for Boot 300GB drive (store one fresh install image and one current)
4x500GB RAID 5 for data 1.5TB drive
3x500GB RAID 5 for data 1TB drive


One reason I began considering the RAID 1 only option is that I realized that:
1. Gigabytes are relatively cheap (compared to RAID hardware)
2. I am more interested and space, reduced complexity and dependability than I am performance.

 

[taltamir]

forget about the crap cards, forget about the 300+$ cards... Use a fileserver for storage. For your OS, use RAID1.

I recommend you make a ZFS fileserver:
http://opensolaris.org/os/community/zfs/

There is a full explanation why it is superior at the site.
Just grab the latest build of opensolaris from http://www.genunix.org/
And follow this guide for setting it up to work with windows file sharing:
http://www.genunix.org/wiki/in...e_Solaris_CIFS_Service

There is not even a need to have raid1 for the OS on the fileserver. Since ZFS is so awesome, I tested it with multiple ZFS capable OSes, you just do a clean install, and the go to terminal and type:
# zpool import -f tank
-f is to force it to import the "filesystem in use" (well it was in use, by the previous operating system)
tank = the name of your array.

If you want something even simpler to implement, there is freenas:
http://www.freenas.org/


PS. You should only use RAID5 (raidz) arrays of 3-5 drives, and raid6 (raidz2) for 5-9 drives. Using more drives is considered too great a risk (risk of cascading failure, the act of reconstructing an array after a drive failure puts the other drives at tremendous risk of failing as well)

PS2. I love my ZFS file server... I have been running it for 3 months now. And just a few days ago a scrub (file verification) found a single read error and fixed the data. ZFS protects you against bitrot, and against cosmic rays flipping individual bits (most likely what happened to that one bit in my last scrub):
http://en.wikipedia.org/wiki/Bitrot

Actually ZFS protects against any data errors, since the data is all fully checksumed, if there is an error it is detected, and fixed from the rest of the array.

 

[Weatherton]

Taltamir,

Just to be sure I understand, are you suggesting that my solution should be:

Desktop:
250GB Boot drive (2 drives in RAID 1 using RAID built into my mobo)

NAS (connected via Gigabit Ethernet): Software RAID
4x500GB RAID 5 for data 1.5TB drive
3x500GB RAID 5 for data 1TB drive

My plan here would be to upgrade my current machine and use the remaining parts to build the NAS.

I like this option but I don't know about performance over the network and possible networking issues. I also am not very familiar with Linux. Please clarify this solution.

Also, I'm still very curious about one of the questions I posed in my previous post:
Will RAID with parity/mirroring prevent hard drive failures from affecting my OS? What will occur as a hard drive dies, how would I be notified and would the OS have any problems? This is a question also for the NAS, would I be notified of a hard drive failure? How would I know which one had failed?

 

[Keitero]

ebay is a wonderful place to look for high end cards on the cheap. I got my second Adaptec 31205 for 1/3 of the cost of a new retail one. As for setting up for the best backup solution, there isn't really one. It's a mix of DVDs/BR discs (if you can shell out the cash) and a nice RAID setup with a backup(s) drive(s). I have my current setup with 8x 750GBs in RAID5EE (RAID5 w/ Hot Swap drive). In addition to that, I back up my very important stuff on a 1TB eSATA HD once a week and put it in a safe as well as make copies of any work I need on DVDs. A bit overkill for most users, but at least I have two failsafes.

 

[taltamir]

yes, that is pretty much what i am suggesting.

if the drive in a redundant raid array fails (raid1, 5, or 6), the array will continue working as if nothing changed. The only noticeable difference to you is that it will be slower.
Depending on your drivers, controllers, and setup, it would notify you in one of many different ways (aka, it all depends on what setup you use). Performance will be low, but it will work.
If you use a NAS and don't want to manually check it via a monitor, then you can set it up to notify you via email.

GigaBIT ethernet allows a max speed of 125 megaBYTES per second. (bit is 1/8th of a byte). So it should be plenty fast, your computer's speed is more likely to be a limit in the NAS.
The 300+$ controllers have the speed advantage, but not the safety, or reliability, or flexibility (aka, it is a single point of failure, where you need to get the same model and use their own method to reaquire the array if it fails, if the nas fails, you repair it and the data remains)

Do you already own drives? because you, seem to be mixing different sizes there (1.5TB and 1TB). which is a bit odd to do. why not one raid6 array of 7 drives?

My NAS is an open solaris installed on an old 160GB IDE drive with a 3450 (20$) video card in an old nfroce 5 mobo with a new X2 BE (best efficiency) 2300 CPU and 4GB of ram.
with 5 SATA 750GB drives in RAIDZ2 (raid6) which gives me the space of 3 drives, and the ability of two of them to fail without loss.


I would like to point out that backups should be targetted at CAUSES of data loss:
1. Fire - offsite backup
2. Flood - offsite backup
3. FBI (a competitor places an annonymous calls the police saying you traffic guns or CP, they will confiscate your computers forever if you are charged, if you are NOT charged they will give them back after a mere 27 year holding period.) - offsite backups
4. bit rot - use a checksuming filesystem (only ZFS at the moment)
5. Cosmic rays bit flipping - use a checksumming filesystem (only zfs at the moment)
6. Drive failure - use redundancy
7. Controller failure - use a scheme that allows you to replace the controller (aka, NOT a motherboard... except for RAID1, raid1 is fully portable even on mobo controllers)
8. A controller failure with a controller no longer available on the market - use a system that allows you to switch to other controllers (aka, OS based software raid like in linux, solaris, windows server, or open nas).
9. Theft - offsite backup OR worthless media backups (DVD-R)
10. Virus - NOD32 antivirus , but if not, have an external backup that you unplug except for when you manually connect it / turn it on to perform a backup. (not that i have seen a virus that just deletes data)

 

[Weatherton]

Originally posted by: taltamirGigaBIT ethernet allows a max speed of 125 megaBYTES per second. (bit is 1/8th of a byte). So it should be plenty fast...

Well, 1Gbps is the theoretical speed, I know the actual is significantly less. Especially given that the bandwidth would have to be shared if others were accessing it. Of course, my current mobo (the one that I could make the NAS with) has 2 Gigabit Ethernet ports... that would be nice to use in parallel.

Originally posted by: taltamirDo you already own drives? because you, seem to be mixing different sizes there (1.5TB and 1TB). which is a bit odd to do. why not one raid6 array of 7 drives?

I currently own:
(all SATA)
2x250GB
7x500GB
2x400GB

I am specifically interested in the new 1.5TB drives, because they are only $180, which is a great price per GB. I got my 500GB's for $55 just recently.

 

[taltamir]

I see. well, you could always sell your current drives on ebay and move to all newer hardware. That is what I do.

Anyways, in regards to difficulty... freenas as is simple as it gets. But it doesn't do ZFS.
Opensolaris does ZFS... but as for the difficulty... you know how linux is really hard for a window user? solaris is the same level of difficulty for a linux user.
But it is getting better quickly, massive strides were made since the first version was released in may (4 months ago).

Why do I bother with it then? because as you notice, zfs is the only thing that can handle certain failures (and others i did not mention). And is very very powerful. oh, i forgot a few reasons...

 

[Weatherton]

I think FreeNAS would be more my speed. I see that it supports RAID 1, 0 and 5. Does it allow you to mirror drive images to external devices?

 

[taltamir]

what do you mean mirror an image to an external device?
RAID 1 is a mirror.
Image is when you make an exact single copy.

And it doesn't care where you place a device, it cares what sort of connection you use. So when you say external do you mean a USB connection? eSATA? a regular SATA but in a professional enclosure with a SATA splitter?

 

[Weatherton]

I mean an external enclosure using eSATA. I intended to say image, I want to be able to make periodic images.

 

[taltamir]

I don't know, sorry.

 

[QuixoticOne]

FYI in addition to FreeNAS here's an additional free storage appliance distribution I've recently heard about:
http://www.openfiler.com/products/feature-summary
http://www.linux.com/feature/146861

Also FYI, I personally agree with taltamir's commendation of a dedicated NAS file server PC running OpenSolaris + ZFS filesystem for extremely trustworthy storage of your data. As far as I know there is no better software OS and Filesystem technology that is available for end user purposes whether free or as a paid OS/storage product. You'd have to be getting into some pretty industrial / high end / commercial storage system hardware and software (think many thousands of dollars) to find good competitors. If I was going to build and administer a PC for a NAS file server, this is what I'd use, and this is indeed what I have chosen.

If you want less complex responsibility for sysadmin, however, but still want a NAS file server implemented in a fullly dedicated PC, then FreeNAS, OpenFiler, et. al. are simple solutions that may be attractive.

If you want a total product solution support as a turn key product then you'd be looking at dedicated purpose built NAS boxes or similar.

If you buy a nice $150-$500 RAID card(s) I'm sure it CAN generally do mirroring wonderfully satisfactorily, just as it could probably do a very satisfactory RAID5 or RAID10 or whatever. The complexity of configuring the product into your OS and administering the storage subsystem will be approximately the same whether you use such cards for RAID0 RAID1 or RAID5 though.

The downside of OpenSolaris is that the hardware compatibility for various ethernet, disc controller, wireless networking, printer, GPU, scanner, printer, et. al. hardware is not anywhere nearly as good as most LINUX / BSD operating systems right now. It is probably SUFFICIENTLY compatible that one can easily select very suitable hardware for a server machine to use this OS from the known / documented compatible hardware. Another downside is that you have to acquire at least a few hundred simple sysadmin skills about how to install, boot, configure, maintain, check, diagnose it to be successful in its use. It is actually much simpler than many other alternatives -- I think it is often simpler than Windows Server, Windows Vista Business, FreeBSD, OpenBSD, et. al. But still it is not something that is intended to be "plug it in and forget about it".

Actually no storage system is "plug it in and forget about it".. you will always need to occasionally diagnose / verify the health of the system, the capacity of the drives, the correctness of the configuration, clean the dust, identify and replace failed/failing drives, test the UPS functions, know how to upgrade / replace drives, et. al.

 

[QuixoticOne]

There is another option which, in theory, seems appealing to me although I cannot personally vouch for it since it is neither a free product nor one which I have direct experience with.

NexentaStor is a software based storage solution platform product that is in fact based around ZFS + Solaris. Given that is seems to have a solid core OS / Filesystem, one might expect that it would be a stable and powerful storage subsystem. You would basically build a server PC and then install this dedicated storage controller OS onto it to provide your NAS storage solution. It seems that for around $400 or so for the ANNUAL license (which presumably includes version upgrades) you could buy their "Basic-4" product which is configured to allow 4TB of storage IIRC and be in the ballpark of your needs. I ASSUME that MAY refer to usable storage, so if you built that 4TB useful storage out of an array of 8TB of mirrored discs it would be treated as a 4TB capacity product. On the other hand maybe they're going by raw disc capacity; I don't know, you'd have to ask them.

http://www.nexenta.com/corp/in...k=view&id=52&Itemid=67
http://www.nexenta.com/corp/in...=93&vmcchk=1&Itemid=93

I see they have some kind of wizard and management tools and so on to ease the installation and use of their product. I don't know how good those are, but maybe they are sufficiently valuable over the free option of just running raw OpenSolaris yourself that it could be worth it to you (or not).

Re: ZFS vs FreeNAS et. al. actually since you will (IIRC) have an EXTERNAL drive backup of the crucial data that is not part of the RAID / NAS, I suppose it matters much less what the reliability of the NAS is. Even if FreeNAS did crash and lose data you could easily recover it from the backup. I guess I'd worry more about silent corruption (which ZFS is excellent at preventing) than outright FS corruption to the point where you'd need to recreate the array from a backup provided that you HAVE such backups.

 

[QuixoticOne]

These folks seem to have turnkey (maybe BYOD) hardware storage array solutions based on NexentaStor's management software; the prices are certainly considerable, but perhaps not particularly high in comparison to various other actually successful / commonplace commercial NAS/SAN products.

http://www.pogolinux.com/products/storage_director