Building up a beefy server for a client

[ViviTheMage]

I have not built a server of this calibur before, usually it's just a 1U/2U with minor disk space, so software raid5 was OK. I've been scowering the web for a good server/hardware forum, but nothing worthwhile came up.

The basic setup is, that he runs a rather large site like rapidshare. We plan to use CentOS, and cPanel/WHM-Apache for now (might go to litespeed if his script can support it).

server needs to have 10+ hot swappable drives
2 quad core, intel CPU's
32GB ram capable, 64 capable is preferred
1u-4u, so we can rack mount it

So, here are my questions :



1) How to recover from raid failure, is software/hardware raid better for this?

2) We plan to have either raid 5, or 6, with 5+ consumer grade WD/Seagate 2TD drives. What type of raid is better? SW/HW?

3) What brand server works the best? We need a 1u-4u server that can hold 10+ hot swappable drives, and have the ability to expand if we needed, to another attachable cage, so we can expand on the current RAID - is this possible?

4) What RAID card should I get that can handle 5+ drives, or more? In a raid 6?

5) What server can handle more drives that we can attach to this server, and expand on the RAID?

6) If a drive fails, can it rebuild on the fly, while the servers still up and running?

7) What can we use to monitor each disk, and prepare for the drive failure, so we can send a new drive to the datacenter, and hot swap it?

8) Is it possible to increase an already running raid5/6? If we have 5 drives, and want to add to it? What would be the best way to go about it? With, or without downtime?


I like supermicro servers, but I am open to another brand of server. He has a 3,000-4,000$ budget for this as well.

 

[mc866]

How much total storage are you looking for? You are planing on using SATA not SAS?

 

[ViviTheMage]

Originally posted by: mc866
How much total storage are you looking for? You are planing on using SATA not SAS?

He wants to use SATA drives, we are planning on using WD2001FASS drives, or if we have to, WD20EADS.

 

[mc866]

DL 370 G6? It has the new Nehalem Xeon's and optional drive cages you can add, supports SAS and SATA from 2.5 up to 3.5. It appears the largest drive it supports is 1TB though.

 

[ViviTheMage]

Originally posted by: mc866
DL 370 G6? It has the new Nehalem Xeon's and optional drive cages you can add, supports SAS and SATA from 2.5 up to 3.5. It appears the largest drive it supports is 1TB though.

Dang, definitely need 2TB drives.

Wat about my other questions, I am very curious about those. We really need expandability, that's for sure.

 

[imagoon]

Originally posted by: ViviTheMage

Originally posted by: mc866
DL 370 G6? It has the new Nehalem Xeon's and optional drive cages you can add, supports SAS and SATA from 2.5 up to 3.5. It appears the largest drive it supports is 1TB though.

Dang, definitely need 2TB drives.

Wat about my other questions, I am very curious about those. We really need expandability, that's for sure.

Easy. iSCSI / FC SAN will fit the bill.

If you need that much space, trying to mash it in to a server is going to be a bit messy. Most (all?) real servers run SAS, running SATA only in encapsulation, this reduces the performance the SATA drives, sometimes significantly due to overhead costs.

SAN's typically have 15+ slots. Allowing you to buy a smaller, much cheaper server to run stuff in.

 

[ViviTheMage]

That is an option imagoon, I was thinking of getting a nice 1U server that would house just two drives for the OS, and my 2 quad core CPU's, and my loads of ram.

Then attach a SAN/iSCSI box, but I cannot think of any that do SATA drives. Do you have any links? I have never set one up either, is it pretty easy when going through the centos setup?

How about my other, more technical questions?

 

[imagoon]

Off the top of my head, the Dell MD3000i and AX4 will take SATA disks. They SAS units though so they do it via SATA encapsulation. I know Dell mentions that support Linux. They specifically mention Redhat so I would expect CentOS to work as well.

---

1) How to recover from raid failure, is software/hardware raid better for this?

Hardware all the way. You want in most cases the XOR engine on the card.

2) We plan to have either raid 5, or 6, with 5+ consumer grade WD/Seagate 2TD drives. What type of raid is better? SW/HW?

Hardware all the way. You want in most cases the XOR engine on the card. Another issue to note is that large arrays can take very long times to rebuild. This can leave you open to multi disk failure.

3) What brand server works the best? We need a 1u-4u server that can hold 10+ hot swappable drives, and have the ability to expand if we needed, to another attachable cage, so we can expand on the current RAID - is this possible?

Honestly this pure choice. We use Dell happily, but HP / IBM are also good products.

4) What RAID card should I get that can handle 5+ drives, or more? In a raid 6?

Cards like the Highpoint RocketRAID 3560 can handle 24 disks. Use this only if your planning to "roll your own."

5) What server can handle more drives that we can attach to this server, and expand on the RAID?

6) If a drive fails, can it rebuild on the fly, while the servers still up and running?

On nearly all cards yes. Check the specs to make sure.

7) What can we use to monitor each disk, and prepare for the drive failure, so we can send a new drive to the datacenter, and hot swap it?

Depending on the service... a SAN will email you when a disk is suspected of being iffy.

8) Is it possible to increase an already running raid5/6? If we have 5 drives, and want to add to it? What would be the best way to go about it? With, or without downtime?

Some can do this but it is pretty rare on the cheaper cards.

I like supermicro servers, but I am open to another brand of server. He has a 3,000-4,000$ budget for this as well.

That budget sounds pretty low for what he is planning.

 

[ViviTheMage]

That MD3000i is already 4k, so I guess that'd be out of his price range!

Thank you for clarifying everything for me. I just have to pick out the proper hardware for him now.

 

[ViviTheMage]

Ok, this is what I was thinking :

HP DL180G5
2x E5420
16GB RAM
HP StorageWorks 60 Modular Smart Array 2U Sotrage System

I can get this for $2800 locally. I assume this should work for what we/he needs? I haven't used HP servers before, but this sounds promising. 90 day warranty on the server, 3 year on the modular storage unit.

1800 for the server + CPU/RAM, 1,000 for the attached storage system.

This should work, yes? He also said that this would allow me to build upon the raid, if we wanted to add another HP StorageWorks 60 modular Smart Array.

 

[dartworth]

take a look at http://www.aberdeeninc.com

 

[ViviTheMage]

Nice, thanks dart

But what would I use to attach a huge 2u raid to though, what card, and what storage setup?

 

[VinylxScratches]

Server + Coraid of some sorts?

 

[ViviTheMage]

It'd be nice to have something attach direct to the server via fiber/raid card, not via ethernet.

 

[ViviTheMage]

Anyone know a cheap place to pick up a HP StorageWorks 60, and trays? Seems my connection ran out of them. I am also open to something simliar. From what i've been looking at, I can use ANY ol 1U server and pop in an HP RAID card (p800) and hook it up to the StorageWorks.

Anyone verify for me? I am also open to suggestions on another attached storage, the only thing is it needs to have the ability to expand on the array if it fills up in a 'daisy chain' sort of way.

 

[ViviTheMage]

Just opening this up to discussion, but maybe I could get this :


http://www.newegg.com/Product/...x?Item=N82E16816101259


A supermicro SYS-6026T-NTR+, all the specs I want, and it's a 2U, holds 8 drives. I will set two up as raid 1, for the OS.

Now the tricky part is what can I do create a raid with the other 6 drives, AND have the ability to attach a SAS drive to build up on the RAID, on the fly? what RAID card can I use for that? What kind of attach setup can I use that would work with this supermicro server/the RAID card/linux ?

 

[Viper GTS]

From the questions you're asking it sounds like you really don't have experience in this level of server. I would avoid the DIY route & get something that is pre-packaged and fully supported in one place.

Have you looked at the Dell PowerEdge R510?

There's an 8 drive chassis available, plus order it with a PERC 6/E add-on to direct-attach to an MD1000 later (up to 15x15K SAS).

1) How to recover from raid failure, is software/hardware raid better for this?

Hardware. PERC 6 or better.

2) We plan to have either raid 5, or 6, with 5+ consumer grade WD/Seagate 2TD drives. What type of raid is better? SW/HW?

Avoid the consumer level drives. Get the RAID editions. Saving $50 or $100 a drive isn't worth it.

3) What brand server works the best? We need a 1u-4u server that can hold 10+ hot swappable drives, and have the ability to expand if we needed, to another attachable cage, so we can expand on the current RAID - is this possible?

Not a lot of variation between them. Finding 10+ drives internal is going to be somewhat hard, but direct attached storage is easy and cheap(ish).

4) What RAID card should I get that can handle 5+ drives, or more? In a raid 6?

Essentially any hardware SAS RAID card will do any of this. With the slow drives you're using you're not likely to be controller limited in any situation so spending big money on Areca or similar with huge cache is almost certainly a waste.

5) What server can handle more drives that we can attach to this server, and expand on the RAID?

This is what direct-attached storage is for. MD1000 holds 15 drives.

6) If a drive fails, can it rebuild on the fly, while the servers still up and running?

Of course.

7) What can we use to monitor each disk, and prepare for the drive failure, so we can send a new drive to the datacenter, and hot swap it?

For Dell - Open Manage. Other vendors have their own system. Better RAID controllers (like the Areca) have web management.

8) Is it possible to increase an already running raid5/6? If we have 5 drives, and want to add to it? What would be the best way to go about it? With, or without downtime?

Typically you would just setup another RAID set, but depending on the controller migration/expansion is possible.

Viper GTS

 

[ViviTheMage]

thanks Viper.

Dells servers are rather pricey, and that supermicro seems to come with more power then the Dell does for hundreds less.

I haven't had any issues with consumer grad hard drives, but if you think about it like this, if I purchased 6 2TB drives @ 200 a pop, compared to $300 a pop, the money I saved I could have 3 extra drives on site to swap if one fails.

Is a drive like the WD RE4 (http://www.newegg.com/Product/...?Item=N82E16822136365) really all that much better?

The thing that I REALLY need to crack down on is how I can expand on the array, because his script only runs on ONE folder,so it will not work with multiple ARRAYS/folders. That's my biggest dilemma, I can attach an ol 1u/2u server, but I need a specific setup, so I can expand in the future, into a SAS or something, but I have never done a SAS before. I've setup a few 1U/2U servers, just not with that specific requirement. I have also never used a heavy duty RAID card, software RAID has always been suffecient for a few drives.

So what I REALLY need to find out is :

.5) Is that WD RE4 drive really better then Seagates 2TBconsumer drive, or WD consumer grade drive? It seems the failure rate is about the same on all of them. The specific use for this server is web hosting -> one site, it's not going to be thrashed like a database would do to it.

1) What RAID CARD can I use to build up an array with 6 SATA Hard Drives.

1.5) RAID card needs to be able to rebuild the arry if a drive fails, on the fly. Needs to be able to add HDD's and build the arry, on the fly as well. Web Portal for HDD management is a HUGE plus (worth the extra 100-200$ IMO, since we wont be at the data center).

2) What Storage system can I use to expand ON that ARRAY, and RAID CARD, so that the folder/mount gets larger.

3) I assume all storage systems are interbrand usable, but I want to verify it works with that SuperMicro server as well.

 

[bigshooter]

If you are using CentOS, why do you have to expand the array itself? Can't you just add another array later and expand the LV?

 

[bigshooter]

What is this used for btw... if it's at all important, wouldn't your client rather dish out an extra grand for reliability than to cut costs?

 

[ViviTheMage]

Originally posted by: bigshooter
What is this used for btw... if it's at all important, wouldn't your client rather dish out an extra grand for reliability than to cut costs?

One, large, file sharing website. Think of it like megaupload.com.

Extending the LVM would allow me to add disk to the current Directory? That is KEY!

 

[Gunbuster]

Did I just see you write about buying a server with a 90 day warranty?

 

[ViviTheMage]

Originally posted by: Gunbuster
Did I just see you write about buying a server with a 90 day warranty?

I am not getting that one anymore, not when supermicro sells a new one, same caliber for the same price and a 3 year warranty.

What decent RAID cards have nice web management apps? So that I can check the health of the RAID/SAS via the web?

Anymore ideas on building upon the array after we fill up our 8 drive limit in the server?

 

[ViviTheMage]

My current plan is to get this chassis :

SUPERMICRO CSE-846TQ-R900B

this motherboard :

SUPERMICRO MBD-X8DAH+-O

Two E5504 quad core xeons

12GB ram, (4gbx3)

2TB drives x 7, and a smaller 320gbx2 in a raid 1 for just the OS.


With that CHASSIS we have room for a lot of space when needed.

Now I just need to figure out the RAID card, OR if we are just going to use software raid. And what sort of management/alerts/monitoring I can use for the drive health.

 

[Viper GTS]

You should populate both CPUs with RAM (6 DIMMs). I'm not even sure the second CPU will work with no memory installed, I can't find anything conclusive either way. At a minimum for performance reasons if you're going to have two CPUs give it 6 DIMMs.

On second thought I have a dual socket nehalem server sitting here let me pull one CPU's RAM & see what happens...

Looks like it will work. During POST it bitches about an unsupported memory configuration but it does let you continue and I have both CPUs active. Still not a recommended configuration though.

Viper GTS