Question PCIe lanes

[Moggy]

Hi!
I have a Asus Sage 10G (motherboard has 4x PCIe 16x slots + 3x PCIe 8x slots).
I want to use the build as file server and a little gaming sometimes.

On there I want to use 1x video card AMD Vega 64 (16 PCIe lanes) and
4x Adaptec 72405 RAID cards (8 PCIe lanes each) connected to 96 x 128GB sata SSD (=a total of 12TB in RAID 0)

--> so that is a lot of SSD data traffic over these lanes to the 2x 10Gb network...
______
Question is which Intel CPU to get; 7820X (28 lanes) or 7900X (44 lanes)
Does the lane count matter when populating with 4x 8 lane hungry & fast RAID cards?
_____
Thanks!

 

[Moggy]

Just to set things straight because of people answering in my name;
This is a serious legitimate project for real world usage
If you are unfamiliar (meaning not knowledgeable enough) with the technical side does not mean someone else's project is nerd/non-sense.
In the mean time I have the whole setup running and am very happy with the results.
Thanks to all who contributed and thanks to all others who just wanted to be opinionated.

Thread closed as far as I am concerned.

 

[VirtualLarry]

What was the RAID-0 stripe size that you ended up using for the 96-drive array? I'm curious.

Can you give us some benchmarks?

 

[Moggy]

What was the RAID-0 stripe size that you ended up using for the 96-drive array? I'm curious.

Can you give us some benchmarks?

I'm curious how you think this is a 96-drive array???

 

[mpo]

I'm curious how you think this is a 96-drive array???

Your OP?
"On there I want to use 1x video card AMD Vega 64 (16 PCIe lanes) and
4x Adaptec 72405 RAID cards (8 PCIe lanes each) connected to 96 x 128GB sata SSD (=a total of 12TB in RAID 0)"

 

[Moggy]

72405=24 sata raid 0
4 raid cards hence for 4 seperate raid 0

 

[VirtualLarry]

72405=24 sata raid 0
4 raid cards hence for 4 seperate raid 0

Hint: Most "decent" RAID cards allow you to create arrays that "span" cards. Hence, based on your original statements, I assumed that you were going to go through with your project, and create a 96-drive RAID-0.

4x 24-disk RAID-0 == not nearly as impressive (or failure-prone, I admit).

 

[mpo]

Hint: Most "decent" RAID cards allow you to create arrays that "span" cards. Hence, based on your original statements, I assumed that you were going to go through with your project, and create a 96-drive RAID-0.

4x 24-disk RAID-0 == not nearly as impressive (or failure-prone, I admit).

Just some back-of-the-envelop calculations for the 24-drive RAID-0 array. Assuming a 1% annualized failure rate for any drive, there is a 24% chance the array will fail in any year (0.01 * 24). That would mean that there is a 2% chance this RAID-0 array will fail in any month.

For mission-critical data, wouldn't touch this with a 10 foot pole. I am more risk-averse than Moggy.

 

[Moggy]

Just some back-of-the-envelop calculations for the 24-drive RAID-0 array. Assuming a 1% annualized failure rate for any drive, there is a 24% chance the array will fail in any year (0.01 * 24). That would mean that there is a 2% chance this RAID-0 array will fail in any month.

For mission-critical data, wouldn't touch this with a 10 foot pole. I am more risk-averse than Moggy.

?? 1% annualized failure rate for any drive ??
you forgot the raid cards
and oh, the power supply energy spike
and oh the motherboard failure
...
If I'm using your "back-of-the-envelop calculations" i can calculate that you have never heard of MTBF and multiplying the amount of your answers by the times you were online since 2010 there is a ((375/10)-19thumbs-up*24/5) , 88.8% chance that you are "a bit" off.
MTBF of the drives is 2.1M hrs , put this in a proper risk assessment formula with all variables and get following outcome; (₃₅C₄)/(₉₀C₄) = 476/23229 ≈ 0.020
If you don't know what that means, I'm sorry

 

[sdifox]

?? 1% annualized failure rate for any drive ??
you forgot the raid cards
and oh, the power supply energy spike
and oh the motherboard failure
...
If I'm using your "back-of-the-envelop calculations" i can calculate that you have never heard of MTBF and multiplying the amount of your answers by the times you were online since 2010 there is a ((375/10)-19thumbs-up*24/5) , 88.8% chance that you are "a bit" off.
MTBF of the drives is 2.1M hrs , put this in a proper risk assessment formula with all variables and get following outcome; (₃₅C₄)/(₉₀C₄) = 476/23229 ≈ 0.020
If you don't know what that means, I'm sorry

What is that formula representing? I understood raid 0 failure rate to be 1-(1-r)^n where r is the failure rate of a single drive.

This paper is interesting

Flash Reliability in Production: The Expected and the Unexpected | USENIX

www.usenix.org

 

[mpo]

?? 1% annualized failure rate for any drive ??
you forgot the raid cards
and oh, the power supply energy spike
and oh the motherboard failure
...
If I'm using your "back-of-the-envelop calculations" i can calculate that you have never heard of MTBF and multiplying the amount of your answers by the times you were online since 2010 there is a ((375/10)-19thumbs-up*24/5) , 88.8% chance that you are "a bit" off.
MTBF of the drives is 2.1M hrs , put this in a proper risk assessment formula with all variables and get following outcome; (₃₅C₄)/(₉₀C₄) = 476/23229 ≈ 0.020
If you don't know what that means, I'm sorry

Sigh...
Yes, a priori, this is just going to be based on just the failure rate of the drives. Not the RAID cards, random events, the motherboard, etc. Consider this to be the floor for the chance of failure for the RAID-0 array. Things will only get worse with with those factor added in (hence, 'back of the envelope')

Annualized Failure Rate = 8766/MTBF (https://en.wikipedia.org/wiki/Annualized_failure_rate)

Your 2.1 million hour MTBF drives aren't going to help you as much as you think.
AFR = 8,766 / 2,100,000 ~= 0.42% chance of failure per drive per year.

Treating the SSDs independently (i.e., they each get a dip in the well at failure each year, yes a bad assumption) 0.42% ^ 24 ~= 9.5% chance of failure for the RAID-0 in any year. That is ~=0.8% chance of failure per month.
(Edit--fixed exponent for trials instead of product, both the annual and monthly failures are within 0.5%, doesn't change the overall message.)

 

[Moggy]

Sigh...
Yes, a priori, this is just going to be based on just the failure rate of the drives. Not the RAID cards, random events, the motherboard, etc. Consider this to be the floor for the chance of failure for the RAID-0 array. Things will only get worse with with those factor added in (hence, 'back of the envelope')

Annualized Failure Rate = 8766/MTBF (https://en.wikipedia.org/wiki/Annualized_failure_rate)

Your 2.1 million hour MTBF drives aren't going to help you as much as you think.
AFR = 8,766 / 2,100,000 ~= 0.42% chance of failure per drive per year.

Treating the SSDs independently (i.e., they each get a dip in the well at failure each year, yes a bad assumption) 0.42% ^ 24 ~= 9.5% chance of failure for the RAID-0 in any year. That is ~=0.8% chance of failure per month.
(Edit--fixed exponent for trials instead of product, both the annual and monthly failures are within 0.5%, doesn't change the overall message.)

sigh indeed;
your calculations are mindboggling wrong;
# "AFR is used to characterize the reliability of hard disk drives " = meaning drives with spinning platters, you know, the noisy ones... i.e. not the SSD's I'm using
# you assume 24/7/y up time (8766)

 

[VirtualLarry]

Why would "AFR" refer to ONLY HDDs, and not SSDs?

Don't SSDs have an MTBF, same as HDDs, thus allowing the same sort of calculation?

 

[mpo]

Some SSD drive manufacturers publish their MTBF specifications. We live in a world where things break.
* Kensington DC450R, MTBF 2 million hours (https://www.kingston.com/us/ssd/dc450-data-center-solid-state-drive)
* Samsung 860 EVO, MTBF 1.5 Million Hours (https://www.samsung.com/semiconductor/minisite/ssd/product/consumer/860evo/)

Yes, I assume 24/7 up time. It is a ratio, so adjust for an 8- or 12-hour work day. But, then you will have to account for power and/or thermal cycling going from ambient to work temperatures. There's no such thing as a free lunch.

 

[Moggy]

Why would "AFR" refer to ONLY HDDs, and not SSDs?

Don't SSDs have an MTBF, same as HDDs, thus allowing the same sort of calculation?

The " annualized failure rate" simplified calculations , MPO sent me (his Wiki link), takes ONLY HDD into consideration and is thus not valid....
If one would make a proper calculation one would indeed need to take thermal cycling and many other factors like the bathtub curve amongst others into count... The last part MPO wrote, "thermal cycling" being a factor, is indeed correct. (I used it in my calculations)

 

[Moggy]

Still, I want to close this thread.
My system is up and running healthy.
_
What I do take from the Anandtech site here is that I got few if any positive and helpful answers.
A lot of the replies were only criticizing what and how I was going about something, without knowing any background and putting the dot on the i, instead of answering my question.
Someone stated that is due to the fact that there are too many wild guys with impossible dreams posing impossible questions on the forum, but that is a mute argument to me.
_
I wish everyone a happy build!

 

[Markeyse]

Still, I want to close this thread.
My system is up and running healthy.
_
What I do take from the Anandtech site here is that I got few if any positive and helpful answers.
A lot of the replies were only criticizing what and how I was going about something, without knowing any background and putting the dot on the i, instead of answering my question.
Someone stated that is due to the fact that there are too many wild guys with impossible dreams posing impossible questions on the forum, but that is a mute argument to me.
_
I wish everyone a happy build!

SO do you have video or at least pictures of your build. I just read this post and I am amused. I would like to see benchmark numbers if you have them.

Myself, I wouldn't dream of adding all those drives (And I like extra), especially since considering a 128GB drive won't have an endurance of even a 512GB drive due to controllers and wear on the chip. Power requirements will pull a more considering if each SSD takes roughly about 5W of power, you looking at 480 just from the drives themselves. And RAID zero MAN!!! That is a HUGE dream of yours! I would like to see this.

I'll have to ask you @Moggy, why not spread those drive to multiple computers? Maybe upgrade some old computers with HDD's still to SSD's? I'm just curious. I like more extreme stuff but you say this is real world, but for a file server even that isn't in an enterprise environment. As well as you point out your experience. Have you done this before? This seems like a mad scientist build (Again, I like the stuff too). But this is stuff I would like to see answers now I read thru this whole thread.