If you wanted to build a SAN, how would you do it?

[Red Squirrel]

I don't have the money but this is something I've been thinking that would be cool to do, build a low cost, high capacity San that is redundant. So at least raid 5 volumes, and maybe even more then one machine so if one goes down you're still in the game. That part may be harder though. It would be built with consumer parts, so maybe a core2quad box or something, and a really good case so you can have hot swap bays.

The idea behind using consumer parts is not only for cost, but for compatibility. If you buy a premade san, 3 years later it's considered end of line, good luck finding a replacement part without being anally rapped, if something dies.

An idea I have is a high end custom built PC, with a motherboard that has lot of sata slots. This can be hard to find (most I've seen is 8 slots 6 is usually more usual), but I recently heard of "port duplicators" so maybe that could help?). The OS would be a light weight Linux distro and be loaded off a flash card or other non sata device. The volumes would be MD raid volumes. So at minimum, 8 hard drives, so maybe 2 raid 5's, or two raid 10's. Now say you build 3 identical machines, I wonder if you could do raid 5's that span the drives from all 3 machines? Now that would be cool. Is this even doable? I'm thinking using NFS maybe...

The machines would then be plugged into a switch, and the servers that need to access the SAN would be plugged into this switch as well, and also plugged into the actual network, this way the SAN would not be part of the network.

Just curious if anyone has ever tried something like this, and what the performance was like. I would not imagine something like this used in a corporate environment though, it would be mostly home/hobby.

 

[Relayer]

http://www.drobo.com/products/droboelite.php

 

[mfenn]

Depends on your needs. Performance or reliability?

At work, we have a need for a high-performance disk array to serve VM images. To that end, we took 9 1U machines with 2x500GB HDDs each. Did a Linux software RAID 1 on each box and then used PVFS (http://www.pvfs.org/) to stripe across the nodes. This gives us pretty ridiculous aggregate throughput (i.e. bottlenecking at the switch!).

Due to the RAID1, we can (and have) sustained individual drive failures. Of course, if one of our boxes fail, we lose the whole array, so this isn't a great plan for critical data. In our case, we're only serving ephemeral VM images, so it's OK.

 

[Red Squirrel]

Depends on your needs. Performance or reliability?

At work, we have a need for a high-performance disk array to serve VM images. To that end, we took 9 1U machines with 2x500GB HDDs each. Did a Linux software RAID 1 on each box and then used PVFS (http://www.pvfs.org/) to stripe across the nodes. This gives us pretty ridiculous aggregate throughput (i.e. bottlenecking at the switch!).

Due to the RAID1, we can (and have) sustained individual drive failures. Of course, if one of our boxes fail, we lose the whole array, so this isn't a great plan for critical data. In our case, we're only serving ephemeral VM images, so it's OK.

Interesting, if you wanted to could you setup pvfs so it's in fact redundant across all boxes? Quickly checked it out and it does sound like an interesting project.

In your case if a box fails and you rebuild it will it boot up the array ok, or would it get corrupted?

If I was building for home I would want decent performance and some half defent fault tollerance but does not have to be super. That's what backups are for. My main needs I'd say are storage space, performance, reliability, in that order.

My existing setup is a single server running it's own MD raid and it's been running for years with no issues. Before I put production data on it I would purposely break the array and have to say it's rock solid. If something like a PSU blows I'm dead in the water, but I'm back up and running as soon as I pop a new PSU in, so I don't consider this all that bad of a risk. Hot swap dual PSU servers are cool, but it's too propitiatory for my tastes. This type of stuff tends to have very short life span as far as support then it's hard to get replacements.

The drobo san looks interesting too though, but what if the whole box fails, could you buy another and put the drives in and still be able to use the data?

I guess that would be another priority in my case, is how easy are parts to replace if they break. If I'd build this I'd want it to last 5-10+ years, and be easy to expand and repair.

 

[mfenn]

Interesting, if you wanted to could you setup pvfs so it's in fact redundant across all boxes? Quickly checked it out and it does sound like an interesting project.

In your case if a box fails and you rebuild it will it boot up the array ok, or would it get corrupted?

With PVFS, the redundancy (if any) has to be below the filesystem level (i.e. MD or hardware RAID).

As far as a box failing, assuming both HDDs are OK, there wouldn't be a problem. If something catastrophic happened, you're looking at rebuilding the array from scratch (i.e. losing the data).

PVFS isn't a great idea for the general use case, where data persistency is paramount, it's more for performance at all costs.

 

[mfenn]

The drobo san looks interesting too though, but what if the whole box fails, could you buy another and put the drives in and still be able to use the data?.

Yeah, that's what has always worried me about proprietary solutions. You are pretty much up shit creek if the manuf. doesn't support you.

 

[Red Squirrel]

With PVFS, the redundancy (if any) has to be below the filesystem level (i.e. MD or hardware RAID).

As far as a box failing, assuming both HDDs are OK, there wouldn't be a problem. If something catastrophic happened, you're looking at rebuilding the array from scratch (i.e. losing the data).

PVFS isn't a great idea for the general use case, where data persistency is paramount, it's more for performance at all costs.

Ah ok so it's basically doing a raid 0 with a bunch of machines. Still a pretty cool solution though, as long as the arrays on the machines are redundant the odds of losing all the data is still probably slim. I like the idea of it as it adds more tcp bandwidth as well. You could probably even team nics to go a step above.

Also if I was to make a huge MD raid 5 with like 8 hard drives would that be bad as opposed to smaller arrays?

I have a sudden urge to do this now, I just don't have the money.

 

[mfenn]

Ah ok so it's basically doing a raid 0 with a bunch of machines. Still a pretty cool solution though, as long as the arrays on the machines are redundant the odds of losing all the data is still probably slim. I like the idea of it as it adds more tcp bandwidth as well. You could probably even team nics to go a step above.

Yep, although in our case, the HDDs can't saturate the NICs on a per-node basis, so there is no need. If you had a bunch of SSDs or something, then bonding would be the way to go.

Also if I was to make a huge MD raid 5 with like 8 hard drives would that be bad as opposed to smaller arrays?

I have a sudden urge to do this now, I just don't have the money.

The Linux software raid code is pretty efficient in my experience, but with 8 drives you will need a decent processor to keep up with all of the parity calculations.

Also, with that many drives, you would probably also want to seriously consider RAID 6, especially if you buy the drives in a batch.

 

[Emulex]

use grid computing like the hp lefthand product which stripes entire raid systems (which can be a phyiscal or virtual machine).

so you get network and disk raid-1 across two units; also increasing throughput.

the same logic can be applied to lessor quality hardware but you'd want to increase the redundancy to more raidsets.

lefthand vsa is free for trial use if you want to play around with it in vmware

the cost of performance for raid-5/6 is huge since you need alot of cache to read/write for CRC computations.

raid-10 may not be a bad solution since there is little cost to write (no crc; no double read/write). at 8 drives the cost isn't alot more. when i say raid 10 i really mean raid-0+1 which is 2 drives in raid-1 then striped.

 

[pjkenned]

I ended up going a bit less fancy. Core i7 w/ the Asus P6T7 WS supercomputer. A decent number of onboard connectors, but tons of PCIe connectors. Right now I have a Intel Pro/1000 PT Quad plus the two Realtek (aka good enough to host RDC ) GigE ports. This isn't high bandwidth nor high storage (once I finish offloading the old server it will be about 44TiB raw capacity), but during burn-in I've been using Windows 2008 Server R2 w/ WHS (backups!!!!) in a VM.

Currently to deal with Raid 1 and Raid 6 arrays I'm using an Adaptec 5805 and an Adaptec 31605. Tomorrow my HP SAS expander arrives which will hopefully give my Adaptec 5805 28 ports worth of connectivity. With only a total of 600MB/s in 6x GigE ports (well 4 + 2 Realtek's) the expander should be fine. I would be a bit worried if I put a second quad port Intel NIC in the machine. That being said, lots of PCIe slots make life easy since I've just been adding cards whenever the mood strikes me. It isn't a SAN, but if you were looking for a single box solution, something like that may work.

 

[imagoon]

Hey Red, what do you want this thing to do? I don't agree with the 3 year comment by the way, if you get the machine from a manufacture like Netapp, Dell, or HP for example you will often have 5-7 years easy. The reverse can be said about buying a no name sata card and using that in an ABIT motherboard.

Anyway if your looking that build it yourself direction, something like Openfiler is really good for this stuff. It can be either a SAN or NAS depending on how you configure it. I did play with it in my test environment and was able to boot Vmware VMs off it with out much of an issue.

 

[MerlinRML]

One of the more interesting things I've come across recently is a SAS expander as a card from Chenbro. http://www.chenbro.com/corporatesite/products_cat.php?pos=37

You'd have to buy a SAS card and cables, but the 6Gbps SAScards are shipping now, which has probably dropped the price on a decent LSI 3Gbps card that supports expanders. Combine all this with a cheap Norco 20-drive rack case, and for approx $1000, you've got all the pieces you need to start planning a huge storage server around running whatever software you prefer.

Granted it's not what I'd call cheap, but it's not bad for a fairly decent solution.

 

[yuppiejr]

One of the more interesting things I've come across recently is a SAS expander as a card from Chenbro. http://www.chenbro.com/corporatesite/products_cat.php?pos=37

You'd have to buy a SAS card and cables, but the 6Gbps SAScards are shipping now, which has probably dropped the price on a decent LSI 3Gbps card that supports expanders. Combine all this with a cheap Norco 20-drive rack case, and for approx $1000, you've got all the pieces you need to start planning a huge storage server around running whatever software you prefer.

Granted it's not what I'd call cheap, but it's not bad for a fairly decent solution.

One point of clarification here... what you are describing sounds more like an expensive DiY NAS (network attached storage - file level) versus a SAN (storage area network - block level) which requires an entirely different method of connection (Fiber Channel HBA, iSCSI, etc..).

While there is product crossover from the "pure" definition of SAN/NAS in the way certain products are implemented the basic concept that divides the two is key:

A NAS includes both the disk resources and a management/client access capability allowing users to map to these resources. NAS devices are simpler to use and fully appropriate in the consumer and small office space for general storage, I've even seen them used for some second tier storage or disk backup solutions in small/medium business applications. NAS is effectively an all in one solution, power it up - configure volumes/shares/security via a web interface and you're up and running.

A SAN is for all intents and purposes raw disks and a high performance controller head with no direct user interface to the file structure. You connect various servers to the SAN via a dedicated interface (Fiber Channel HBA or iSCSI optimized network adapter) through a dedicated connection fabric (fiber channel or iSCSI switches) and allow the servers to access these disk resources as through they were local to the machine. SAN hardware is all over the map but is generally designed for SMB/enterprise us and thus engineered using more expensive, longer lasting components for which there is a much longer expected lifespan than traditional consumer grade hardware components.

The value of a SAN has a lot more involved than just the price of cobbling together a bunch of consumer grade components into a workable solution. Having a single throat to choke when things go south with your SAN is of key importance in the SMB hardware space as is products that are designed and validated specifically for business usage. As a previous poster stated, mainstream SAN products normally have a lifespan of 5+ years with plenty of available support outside of all but the most obscure manufacturers for 7-10.

 

[ViviTheMage]

I would build this :

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

but where to get that chassis custom built...hmmm

 

[Red Squirrel]

I would build this :

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

but where to get that chassis custom built...hmmm

That looks interesting, That's actually another thing, cases that can hold lot of drives and still have proper ventilation are not that easy to come by. I did find a few rack mounts but they're almost a grand! So that said I would probably look at a custom build for the case too. Not yet sure how I'd do it though.

I also heard of open filer, think I will have to check that out too.

 

[pjkenned]

I wanted to build something similar using the HP SAS Expanders not the Chenbro one (the new HP one has less compatability issues) or multiple raid cards.

Sadly I couldn't find a 40 drive case :-( I ended up buying a Norco 4020 and will be using a second case for the additional drive bays. It seems like if one really wanted to fit more than the standard 22 drives a 23rd drive could fit in the optical bay there, and I bet a crafty person could fit 2-3 more drives... but it isn't 40 sadly.

Also, those PSU's are not off the shelf.

Overall, it was something I wanted to recreate, but couldn't.

 

[yuppiejr]

That looks interesting, That's actually another thing, cases that can hold lot of drives and still have proper ventilation are not that easy to come by. I did find a few rack mounts but they're almost a grand! So that said I would probably look at a custom build for the case too. Not yet sure how I'd do it though.

I also heard of open filer, think I will have to check that out too.

http://blog.backblaze.com/2009/10/07/backblaze-storage-pod-vendors-tips-and-tricks/

Sounds like the outfit that makes the cases for these guys got a huge response from their Blog post and will build the enclosure in 1 unit quantities for $750 plus shipping and applicable tax. Guessing they banged out a few hundred for the original order and it's relatively easy to duplicate in almost limitless quantities.

That is some serious bang for the buck for low performance/high density second or even third tier storage. You can build two of the Backblaze boxes at 67 tb usable each for the price of a single EMC/Data Domain DD140 which does 43 tb after deduplication (a stretch from 1.5 tb actual) and is about the next cheapest second tier/disk backup storage technology I've seen recently.

I wonder if the backplane or drives give any visual indication should a drive fail - could make finding and swapping the right drive a bit of a fiasco. Otherwise it's an interesting concept, I wonder if anyone is going to pick up the design and commercialize it...

 

[ViviTheMage]

Check out backup PC, we use it for some internal back up stuff...works in linux or windows.

 

[Red Squirrel]

http://blog.backblaze.com/2009/10/07/backblaze-storage-pod-vendors-tips-and-tricks/

Sounds like the outfit that makes the cases for these guys got a huge response from their Blog post and will build the enclosure in 1 unit quantities for $750 plus shipping and applicable tax. Guessing they banged out a few hundred for the original order and it's relatively easy to duplicate in almost limitless quantities.

That is some serious bang for the buck for low performance/high density second or even third tier storage. You can build two of the Backblaze boxes at 67 tb usable each for the price of a single EMC/Data Domain DD140 which does 43 tb after deduplication (a stretch from 1.5 tb actual) and is about the next cheapest second tier/disk backup storage technology I've seen recently.

I wonder if the backplane or drives give any visual indication should a drive fail - could make finding and swapping the right drive a bit of a fiasco. Otherwise it's an interesting concept, I wonder if anyone is going to pick up the design and commercialize it...

That case is indeed a pretty cool design, and it makes sense since a traditional SAN has lot of wasted space. And yeah it's a good thing to consider to have some kind of indicator led.... for my home server I have all my drives documented by serial # and slot # so should a drive fail I know what slot, but when dealing with tons of drives there's always the possibility of human error, then you pull out the wrong drive and break a raid. Ouch.