SAS 3g vs. 6g

[equazcion]

I just put together a new server using an Areca 1222 RAID card, and was disappointed to find out I got the wrong card. I meant to get the 6gb/s version. I have three SAS 15k 6g hard drives connected via a 6g SAS backplane, in RAID-10.

My question is, if I replace the RAID card with the 6g version will there be a real difference? I can't return the RAID card I already purchased, so the replacement 6g card will cost me another $600. Is it worth it?

I'm also wondering if I'll be able to keep my data and not re-initialize the array if/when I do replace the RAID card. I'd be replacing it with an Areca 1880i, which is more or less the same card but with 6g support. Will I be able to rescue the old array after the switch?

Thanks in advance for any help.

 

[mfenn]

How are you doing RAID 10 with 3 drives?

But no it will not matter if you have 3Gb/s vs. 6Gb/s with a small number of mechanical drives. Certainly not worth spending $600 on.

I'm not sure if you would be able to reuse the array. They use a different IOP, so I doubt it, but anything is possible with the right firmware on Areca's part.

 

[equazcion]

How are you doing RAID 10 with 3 drives?

You know, that's an excellent question. I actually bought 4 drives initially, but one showed up as failed so I removed it. Then the RAID card still gave me the 1+0 option for the 3 remaining drives and said I'd have a total of 900GB (for three 600GB drives). I was confused but figured there must be a way to pull that off that I hadn't heard of before. The array initialized fine and is still showing as RAID 1+0.

In theory you could run RAID 10 on three drives if you divide each drive into two partitions, although many of the RAID 10 benefits would be lost that way. It would explain the 900GB total though. I wonder if that's what it did. Unfortunately I can't see any way in my RAID card to see the exact topology.

Either way I'm planning to add more drives. Do you think 6g would make a difference in an 8-drive RAID 10 array?

Thanks for the answers yet again mfenn

 

[mfenn]

Hmm, I suppose I do see how you could theoretically do a RAID 10 with 3 drives partitioned in 1/2, but it doesn't make too much sense to actually do it. RAID5 would give you the same level of protection and give you 1200GB usable.

As for the question of SAS bandwidth, it would depend on how many connections you have to your backplane. Each SFF-8087/88 has 4 SAS lanes, giving a bandwidth of 12Gb/s or 24Gb/s. So your 1222 has about 24Gb/s of SAS bandwidth, assuming that your backplane supports two connections.

An 8-drive RAID 10 could probably do about 9.6Gb/s flat out (best case). A dual connected backplane will thus have sufficient bandwidth for quite a few more drives, but even a single connected one still has headroom left.

 

[equazcion]

An 8-drive RAID 10 could probably do about 9.6Gb/s flat out (best case). A dual connected backplane will thus have sufficient bandwidth for quite a few more drives, but even a single connected one still has headroom left.

I'm confused. My backplane is dual connected. Are you saying with an 8-drive RAID 10, I would, or would not, be hitting the 3g bottleneck?

If you're saying an 8-drive RAID-10 only does 9.6gb/s total, divided between all the drives, and therefore I'm not hitting the 3g bottleneck with my 15k SAS drives (fastest non-fibre platter drives on the market AFAIK), then what exactly is 6g ever good for? SSD only? I was aware that 6g doesn't generally benefit most consumer systems, but I figured these top-of-the-line server drives might break 3g.

Wouldn't platter drives' cache operations see some benefit at least?

I'm also confused about where you're getting the 9.6Gb/s total from. Are you saying all 8-drive RAID-10 arrays have this approximate transfer rate, despite the performance differences between various RAID cards and hard drives?

 

[mfenn]

I got the 9.6Gb/s number by taking 150MB/s per drive and multiplying it by 8 drives.
150 x 8 x 8 = 9600. I've measured about 130MB/s out of my 10K drives, so a 15K will be able to do more that 150MB/s, but your RAID performance isn't going to scale linearly. So treat the number as a (realistic) back-of-the-envelope calculation. The RAID controller itself will matter very little for sequential I/O on a RAID10 array. Random I/O will be massively bottlenecked by the mechanical nature of the drives and thus the backplane bandwidth will matter even less.

Your backplane has 24Gb/s of bandwidth to the controller, so no you would not be bottlenecked by 3Gb/s speads. Yes, drive cache operations would benefit from the full bandwidth available, but they are so small as to be inconsequential.

You are correct that SAS 6Gb/s is not useful in small arrays unless you have SSDs. 8 drives is a small SAS array because you've got basically a 1:1 mapping between controller lanes and drives. Remember, SAS is much more like a switched Ethernet environment than the point-to-point SATA protocol. A large SAS array would be ~50 drives. That's where having SAS 6Gb/s comes in handy.

 

[equazcion]

8 drives is a small SAS array because you've got basically a 1:1 mapping between controller lanes and drives. Remember, SAS is much more like a switched Ethernet environment than the point-to-point SATA protocol. A large SAS array would be ~50 drives. That's where having SAS 6Gb/s comes in handy.

I'm still confused by this. If each drive is limited by its mechanics to only read (assuming we're talking about read speeds) data at a certain rate, 150mb/s for argument's sake, how does having more of them suddenly utilize each one's SAS connection more? Assuming we're comparing two arrays made up of the same drive type, I'm not understanding how each connection would get utilized more in the larger array.

Thanks for taking the time to continue explaining this to me

 

[mfenn]

I'm still confused by this. If each drive is limited by its mechanics to only read (assuming we're talking about read speeds) data at a certain rate, 150mb/s for argument's sake, how does having more of them suddenly utilize each one's SAS connection more? Assuming we're comparing two arrays made up of the same drive type, I'm not understanding how each connection would get utilized more in the larger array.

Thanks for taking the time to continue explaining this to me

We're talking about both sequential read and write doesn't really matter either way.

First key thing to understand is that the more drives you put into a RAID array, the faster an array as a whole is because the load is split among the drives.

The second key thing to understand is that SAS is not a point to point link. Your controller might have two SFF-8087 connectors with 4 lanes (3Gb/s or 6Gb/s depending on the controller) for a total of 8 SAS lanes going into/out of the controller. That does not limit you to 8 drives however! With the proper expanders and backplanes, you can easily have hundreds of drives attached to that controller. Each drive has its own link to the backplane (3Gb/s or 6Gb/s), but that data has to eventually make its way back over the 8 lanes to the controller. Thus, those 8 lanes will become the bottleneck.

No problem, glad to help.

 

[equazcion]

That does not limit you to 8 drives however! With the proper expanders and backplanes, you can easily have hundreds of drives attached to that controller.

Ah, so each drive isn't being utilized more in a large array. Only the SAS lanes are utilized more, by allowing you to put more drives on each lane without sacrificing bandwidth. I think I get it now.

Thanks so much for clearing that up for me, really appreciate it.

 

[mfenn]

Exactly, the controller's SAS lanes are saturated in a large array, but will be nowhere close to saturated in a small array. Glad to help