How many of you IT guys build your own work servers (IE Lenovo)?

[LokutusofBorg]

Long story... bear with me.

I'm a programmer at a small company where our CTO has let our infrastructure decay to the point where it's threatening our business. We've had our production environment collocated for several years and it's been a very good setup with no major problems. But it is getting old, and our CTO decided we needed to move to managed/shared hosting because he thinks managing our own servers is a mistake (and he bends over for the latest buzzword, this month it's Cloud Computing).

We used to have two IT guys -- one was laid off almost a year ago, and the other one quit a month ago in complete disgust (he should have been laid off instead of the other guy, but that's a different story). So it fell to us programmers to try to make this server move. Well, despite help from a full consulting team of IT pros and DBAs at the new hosting company, our system was down for three days while we struggled to make it work. We finally reverted back to our old environment (ask me how I feel about having to stay up for nearly 50 hours straight to get us back online...). So now we're back to square one.

As mentioned, our old environment is too old to stay with for much longer (system slowdowns). I personally think we should stick with a collocation solution since we have some hardware that can continue to be used. We just need to pony up some dough for the right hardware upgrades and find a new provider that can give us better service (our current one is less than ideal). Our CTO, however, is convinced it will take well over $100k to go this route. I think he's either clueless, or he's using this high dollar amount to rule out this option from the first. I've decided I either need to start speaking up, or I need to find another job, because I can't stand to put up with the situation any longer. So I've been looking into servers and pricing, and just want some perspective.

Our system is pretty database intensive. Right now we have a dual P4 DB server with 4GB of RAM and it's getting us by, but it seems to be the major cause of most of the slowness our users are seeing. (Either that, or our SAN is causing it. DBs are stored on the SAN along with lots of other stuff the system uses heavily, and DB IO wait times are in hundreds of milliseconds when users are reporting problems.) We have corollary servers (web, process/notifications) but they aren't showing any signs of overload.

I'm not very good at capacity planning, but we have two 50GB databases; we have an internal production team of around 40 people that use the system heavily all day; and we have several thousand customers, so maybe another 100 external users hitting the system at any one time. I had some numbers for IO throughput and transactions/second from earlier today, but I can't remember them now. But I think a back-of-the-napkin look at things is sufficient, given the caliber of hardware you can buy now (dual Nehalem/Opteron servers, SAS RAID) and the fact that our antiquated system is just now borderline insufficient. I think we'll be fine with a modestly robust DB server with some dedicated storage, and the SAN can be retained for non-DB data and backups, etc.

Looking at Dell's server offerings, prices I'm seeing are upwards of $20k for a dual socket server (and that's still not counting a storage solution). When I look at options like the Lenovo servers you can buy and piece together, it seems ludicrous to pay that much.

How many of you IT guys are paying the Dell (or HP or whatever) premium? How many of you are piecing your own servers together to save on costs? What are your thoughts on each approach and how does the size of your company affect your opinion? What ballpark do I need to be looking at to get a pretty noticeable upgrade from a dual P4 box? Any general advice based on the story I've outlined above?

TIA.

 

[Fullmetal Chocobo]

In my area, we currently pay the Dell premium, but are looking to build our own. It's a matter of licensing and going through all the red tape to do so. It wasn't really considered before, as no one had any experience in doing so, but since I've came in the shop, it is more likely a possibility.

 

[KentState]

I have never worked for a company that would build their own equipment. Secondly, I would never go the Dell route and would stick with HP. They provide the best support, hardware, and tools to run a data center. We are actually replacing the last remaining Dell boxes with HP eBay specials. Part of what you are paying for is a company testing and piecing together a very stable machine. Unlike at home where downtime and messing with components is part of the hobby, at work it means loss of revenue.

Speaking of eBay, that is a great way to find slightly used HP servers for a fraction of the cost of buying new. There are a couple resellers on their that specialize in used HP equipment.

 

[TheKub]

Originally posted by: LokutusofBorg
..(and he bends over for the latest buzzword, this month it's Cloud Computing).

Well, it really might be a solution. I dont think you are in a position where you should be managing your servers. There are a number of companies that will offer you managed computing power based on what ever $$ you want based on a SLA. You will get the servers you need without buying the hardware (they spin you up virtual servers).

Using this model you have flexabiliy. Say around the holidays your demand for computing doubles for a few weeks. Rather than being slow those few weeks or wasting money by spec-ing for the high load you can simply add additional capacity by calling your provider and paying the negotiated premium.

Dont get me wrong I hate clueless managers that spout buzz words but it sounds like it is truely a viable option for you.

 

[bjlockie]

I've worked in shops that used Dell or shops that used Suns.
The thing is real servers have dual power supplies and hot swappable disk drives.
I would also recommend hardware raid cards.
I absolutely loved the remote management (ILOM on Suns) and that is usually a 'server' feature.

With no system administrators of your own, I'd look at the cost of outsourcing.

 

[LokutusofBorg]

Originally posted by: TheKub

Originally posted by: LokutusofBorg
..(and he bends over for the latest buzzword, this month it's Cloud Computing).

Well, it really might be a solution. I dont think you are in a position where you should be managing your servers. There are a number of companies that will offer you managed computing power based on what ever $$ you want based on a SLA. You will get the servers you need without buying the hardware (they spin you up virtual servers).

Using this model you have flexabiliy. Say around the holidays your demand for computing doubles for a few weeks. Rather than being slow those few weeks or wasting money by spec-ing for the high load you can simply add additional capacity by calling your provider and paying the negotiated premium.

Dont get me wrong I hate clueless managers that spout buzz words but it sounds like it is truely a viable option for you.

I have my doubts about the "universal applicability" of cloud computing, but I held my tongue while this move was in the planning stage because I'm no expert by any means, but I was a network admin for a few years and I've been a computer enthusiast for more than a decade. I don't think our system fits the general profile for cloud computing, in that we have a fairly constant demand, we need high availability, and we have *very* DB-intensive logic. We will hardly ever double our demand for a short while, like you're saying. Like I said in the OP, we have a team of production personnel that work on the system all day every day, and they are the bulk of the load the system has to endure. Our ops people have said that our customer-utilization rate is something like 10%, which means most of them are sending us all their paperwork and we're doing the work in the system for them, if that makes sense.

 

[LokutusofBorg]

Originally posted by: bjlockie
I've worked in shops that used Dell or shops that used Suns.
The thing is real servers have dual power supplies and hot swappable disk drives.
I would also recommend hardware raid cards.

The Lenovo systems that I mentioned do have these features. They just free you up to not have to buy Dell-certified hard drives that carry a 1000% premium, etc.

With no system administrators of your own, I'd look at the cost of outsourcing.

Most of the reason for my extremely low opinion of our CTO has less to do with the botched move I explained in the OP and more to do with his general handling of his duties. He seems oblivious to the fact that he's paying programmers to take care of IT duties, even though we've told him multiple times that that's the case. He's constantly bemoaning how we can't get any of the larger programming projects done, and is always talking about hiring new programmers. Quite a few of us get near livid when he brings it up. We've even taken it over his head. Anyway, hiring a good IT guy (or getting an outsourcing company to dedicate a guy to us) is something we're pushing for very heavily as well. Like I said, he just doesn't seem to get it.

 

[TheKub]

Originally posted by: LokutusofBorg
I have my doubts about the "universal applicability" of cloud computing, but I held my tongue while this move was in the planning stage because I'm no expert by any means, but I was a network admin for a few years and I've been a computer enthusiast for more than a decade. I don't think our system fits the general profile for cloud computing, in that we have a fairly constant demand, we need high availability, and we have *very* DB-intensive logic. We will hardly ever double our demand for a short while, like you're saying. Like I said in the OP, we have a team of production personnel that work on the system all day every day, and they are the bulk of the load the system has to endure. Our ops people have said that our customer-utilization rate is something like 10%, which means most of them are sending us all their paperwork and we're doing the work in the system for them, if that makes sense.

Cloud computing may not have "universal applicability" but if your system works as a colocation there is no doubt in my mind that it would work "in the cloud". I mean the principle is the same its just you dont have physical servers. As for your *very* db-intensive logic a reputable company will be able to meet your needs. When you look at doing it you create an SLA with the company, in that you can state your the maximum IO wait time is as welll as many other performance points. If you think you are more then they can handle you can likely think again. Thats thier job if they couldnt do it well they wouldnt be in business. The hardware they are going to be running is likely leaps and bounds over what your self-build would be.

I can understand if there is some hesitation but I have seen virtualization in action, we have consolidated most of our data center down to 7 servers. And your really never going to have issue with down time unless you guys do something wrong becasue with virualization and properly configured redundancy (which you wouldnt have to worry about) hardware issues will have zero impact on you.

 

[LokutusofBorg]

Originally posted by: TheKub
Cloud computing may not have "universal applicability" but if your system works as a colocation there is no doubt in my mind that it would work "in the cloud". I mean the principle is the same its just you dont have physical servers. As for your *very* db-intensive logic a reputable company will be able to meet your needs. When you look at doing it you create an SLA with the company, in that you can state your the maximum IO wait time is as welll as many other performance points. If you think you are more then they can handle you can likely think again. Thats thier job if they couldnt do it well they wouldnt be in business. The hardware they are going to be running is likely leaps and bounds over what your self-build would be.

I can understand if there is some hesitation but I have seen virtualization in action, we have consolidated most of our data center down to 7 servers. And your really never going to have issue with down time unless you guys do something wrong becasue with virualization and properly configured redundancy (which you wouldnt have to worry about) hardware issues will have zero impact on you.

I'm not sure how you can say that collocation is the same principle as cloud/virtualization. VMWare has put out data that shows that with only a few virtual machines on a server, you will see something on the order of 20% reduction in performance compared to bare metal. The "not having physical servers" *is* the whole principle.

This article gives a very good viewpoint that sums up my position. Cloud computing is nothing more than a new version of Grid computing, and to say that every system out there is just as good off on virtual (or distributed) machines as it is running on bare metal (or a single beefy system) is rather naive or propogandistic (salesman-ish?), IMO.

I don't mean that insultingly. Like I said in the OP, we had a team of IT guys and DBAs trying to help us figure out how to get our system working on the cloud servers we tried to move to. I'm not going to call out the host by name, but this isn't a small company, and their guys knew what they were talking about. We even had a local consultant on our end that we brought in to try to help us out, and he is one of the leading experts on virtualization in our area (he worked for HP for a number of years). They worked for two days straight, tried every trick in their book, moved us to their newest cluster that had no other clients yet. And after all their efforts, their consensus was, "looks like you need a dedicated box". I had people look at me and give me that look, "that's what you've been saying all along."

If your response to that is that those guys did not, in fact, know what they were doing, that's fine and I'll concede that. After several conference calls I was left shaking my head. I'll just say that I firmly believe not every application is appropriate for the cloud, and we can agree to disagree.

EDIT:

Originally posted by: TheKub
The hardware they are going to be running is likely leaps and bounds over what your self-build would be.

I'm also not sure how you can claim this. The Lenovo server I've been eyeing is the RD210, which is a dual Nehalem-EP box. We can buy mid-range CPUs (like 5520s) and put 32GB of RAM in there for not too much money. The (newer) blade servers they put us on had last gen Xeons and 32GB of RAM. And we'd have to share that box with other clients or pay a high premium for a *gasp* dedicated box.

 

[TheKub]

If I sound like a salesman its only because I have seen first hand how virtualization can work. We have virtualized just about everything we can here and have had zero perceptible performance loss (and many performance increases). Now if a host dies before we can even get alerted the system is comming up on another box. Need to perform maintenance on the host no problem click a few buttons and we can rip the server out of the rack during business hours with zero downtime\impact. We have a cement truck drive through our datacenter? no problem 2-4 hours we will be up and running in our DR site (as opposed to 1-2 weeks rebuilding).

I cant comment on your situation but if your experts say it cant be done (personally Id get a second opinion) so be it, but it would be one of a very small list of things that cant be virtualized. We have met and spoken with shops that have massive VM environments and I suspect that they put far more demand and stress on thier equipment and it handles it great. I cant fathom why a load that can be handled (even poorly) by a dual p4 with 4GB of memory cant be virtualized. Then again I woudn't even declare myself a VM expert, but I have seen what it can do and Im a believer.

Originally posted by: LokutusofBorg
EDIT:

Originally posted by: TheKub
The hardware they are going to be running is likely leaps and bounds over what your self-build would be.

I'm also not sure how you can claim this. The Lenovo server I've been eyeing is the RD210, which is a dual Nehalem-EP box. We can buy mid-range CPUs (like 5520s) and put 32GB of RAM in there for not too much money. The (newer) blade servers they put us on had last gen Xeons and 32GB of RAM. And we'd have to share that box with other clients or pay a high premium for a *gasp* dedicated box.

As for that it would depend on your provider, I dont have specific knowledge of the environment they may offer. There are large 64+ CPU boxes that run VMWare with massive amounts of memory. Which most assuredly beats your server. Do you need that much, certainly not. Same with their storage system its likely to be several steps higher than one that is sized soley for your use.

Given the 5 min intro to your environment and situation I offered my input if you have experience otherwise so be it. As for the lenovo servers, I dont have any experience with them, nor have I ever seen a datacenter that has them. I do know that when we ever have an issue with our HP servers we have the spare part or a technition at the datacenter the next day.

 

[LokutusofBorg]

Yeah, the support contracts that come with the premium hardware are compelling. But for an SMB like us it's hard sometimes to stomach the price. We aren't building a datacenter, just a system with a half dozen servers. A competent mid-level systems administrator should be able to handle that kind of environment. That's what I'm angling for here. We need an IT guy to take the load off us programmers, so why not get one that's competent, and build a collocation system that he can manage? That just seems to make the most sense for us from my point of view. Aside from the (idealistic) argument about virtualization that you and I are getting into here, we already have an investment in hardware that can continue to be used, and to abandon it in order to move to a hosted solution makes little sense to me. I don't make the decisions, I just feel the need to represent this course more accurately so that the people that do make the decisions here can be more informed.

Thanks for the input. I don't mean to bat you down. I'm just not convinced virtualization is the right solution for us.

 

[sonoma1993]

Originally posted by: LokutusofBorg

How many of you IT guys are paying the Dell (or HP or whatever) premium? How many of you are piecing your own servers together to save on costs? What are your thoughts on each approach and how does the size of your company affect your opinion? What ballpark do I need to be looking at to get a pretty noticeable upgrade from a dual P4 box? Any general advice based on the story I've outlined above?

TIA.

When i was working at my old job, we built all of our servers and our client servers. We were using supermicro cases and supermicro motherboard.

 

[Axon]

What would you guys recommend for a law office of ten people that generally needs to store word documents and pdf scans on a shared hard disk?

 

[Winterpool]

Axon, you might want to spin off a new thread for your query, as the sort of infrastructure the others are discussing is much more hardcore.

That said, you might want to start by looking at Microsoft's Small Business Server (presuming your office is a Windows shop). If it's merely a matter of storage, WHS might almost be usable had it a licence that supported a few more users (I think it maxes at 10). Punters online will also inevitably suggest Linux / Samba, various NAS concoctions, etc, but I don't know if this office can run that sort of tech (is this your workplace or a friend's?).

Lokutus, you probably already know this, but I'd strongly recommend you consult the 'pro' forums at ArsTechnica (I think they're called 'The Server Room'?). I love AnandTech and post here more than any other place online, but the knowledge and experience on the ArsTechnica forums are formidable.

 

[lxskllr]

Originally posted by: Winterpool
Axon, you might want to spin off a new thread for your query, as the sort of infrastructure the others are discussing is much more hardcore.

That said, you might want to start by looking at Microsoft's Small Business Server (presuming your office is a Windows shop). If it's merely a matter of storage, WHS might almost be usable had it a licence that supported a few more users (I think it maxes at 10). Punters online will also inevitably suggest Linux / Samba, various NAS concoctions, etc, but I don't know if this office can run that sort of tech (is this your workplace or a friend's?).

For such simple needs, I think Linux would be a good choice. They require almost nothing, and then wouldn't have to worry about licensing fees. Assuming this is Axon's office, he should be able to learn all he needs to know over a weekend. It would then be fairly simple for him to administer*

All assuming he's technically inclined, and not afraid of the unknown.

 

[glugglug]

If you are currently running on a dual P4, it is a VERY safe bet that you do not need a dual socket server.

I would wager an i7 desktop with 6GB RAM and an SSD will outperform the $20K Dell as a DB by a large margin, mostly because of the faster disk.

If you must go the HP/Dell route (almost all enterprises do, smaller companies tend to build their own), I would go with one of these: HP DL 380 G6 starting at $2289 It is dual socket, but we currently use only 1 in each box because Win2K3 licenses don't support more than 4 cores.

Replace the drives that come with it with Intel SSDs in RAID1 for your DB (or just add them in the extra drive bays). They are cheaper than the 15K RPM drives from HP and literally dozens of times faster. Contrary to popular myth, while the RAID controller in there comes with SAS drives on it, it is actually a hybrid SAS/SATA controller, so it will work with the SATA drives.

Edit: Just realized you said you are using a SAN. What kind? If it is from the same era as your servers you can almost be certain a single SSD on a fileserver will outperform it. Even if it is more modern, there is still a good chance of that.

 

[LokutusofBorg]

Originally posted by: Winterpool
Lokutus, you probably already know this, but I'd strongly recommend you consult the 'pro' forums at ArsTechnica (I think they're called 'The Server Room'?). I love AnandTech and post here more than any other place online, but the knowledge and experience on the ArsTechnica forums are formidable.

Awesome, thanks. I read Ars via their RSS feed, but I never realized they had a forum. I had a hard time deciding if I should post this thread on AT, and then an even harder time trying to decide which forum to put it in. I'll definitely be hitting up the Ars forums.

 

[LokutusofBorg]

glugglug, our SAN is an HP unit the company bought just a few years ago. So thanks for the link to the HP server, that's definitely something I'm going to take a hard look at. Not sure why I haven't ever even perused their site yet. I had an assumption that they were really expensive. Definitely had the wrong idea there, didn't I? $3k for a mid-range model of that server, and the only thing really missing from it is storage? That's pretty good.

Buying a dual socket server but only using one CPU for now is a great idea, I was getting a bit ahead of myself and didn't have that option in mind. Definitely a good plan to help save some money. We're pretty sure storage IO for the DBs is our bottleneck at this point.

I'm a little wary of SSDs at this point still. But those charts in the AT article comparing them to SAS *are* pretty damn compelling, I do have to say. Does that HP server you linked restrict you to proprietary hard drives, or are you free to install whatever you want?

In that same vein, does Dell restrict you to using their hard drives in their servers?

Thanks, you've got me thinking along all kinds of new lines now.

 

[RebateMonger]

Originally posted by: LokutusofBorg
In that same vein, does Dell restrict you to using their hard drives in their servers?

No. Even when the subject of RAID arrays comes up in a support call, Dell won't ask where the drives came from. Obviously they won't pay for the replacement of a disk they didn't supply.

 

[LokutusofBorg]

Originally posted by: RebateMonger

Originally posted by: LokutusofBorg
In that same vein, does Dell restrict you to using their hard drives in their servers?

No. Even when the subject of RAID arrays comes up in a support call, Dell won't ask where the drives came from. Obviously they won't pay for the replacement of a disk they didn't supply.

I just had all kinds of wrong assumptions. Good to know...

I see now I chose a bad subject line for this thread, because by "build your own server" I wasn't referring to buying a separate motherboard, and piecing it all together like you do when you build a home PC. I really meant buying a barebones kit that already has all the main pieces, and then just putting in your own drives, and maybe a controller or whatever. I had thought that the Dell and HP servers were hardware-restricted and they only let you run their hard drives.

 

[HappyCracker]

A poorly configured drive array can cause all sorts of problems. We had a Symmetrix 8830 where growth had been poorly planned, leading to bad performance and hotspots. What HP unit are you using? Is it an MSA? If you only have two 50GB databases, what are your expected growth rates on those? You may be able to pick up a DL380/385 and setup three arrays. One for the OS volume, the other for the database data, and the third for the transaction logs. They have 146GB/15K drives in the 2.5" size these days, so you could build something with pretty good disk performance locally and ditch your HP storage unit. To me, the cost of SSD in the enterprise is too great currently. The only drives that would be acceptable are the SLC flash drives and those carry a premium. SAS is a solid performer.

I can attest to my love of HP servers. Though storage is my area these days, I dealt with HP and Sun boxes for some time. iLO is a solid remote management system, HP's RAID cards are good and offer battery-backed write caching. Their support is usually pretty good, though sometimes you have to flex some muscle to get them to back off and just send out the replacement parts. They were always doubting the fact a hard drive needed to be replaced when the system had marked it as bad.

 

[ViviTheMage]

Why not rent servers monthly? And have the DC take care of everything for you? Grab 2 servers, one for your large database that everyone connects up to, and another, and throw on VMWare ESX and virtualize your other little servers that don't seem to take up much load.

It doesn't sound like you use a lot of space, so why not just get a RAID5/10/SAS/SAN and attach both systems to it? Or have them separate if you want.

 

[LokutusofBorg]

HappyCracker, I'm not sure what model our HP SAN is. It has a few TB and we're not really close to filling it up. Our system is just really disk intensive (via the database, as well as how the web side was coded), and so it looks like putting the DB on a new server with dedicated storage is our best option at the moment. Ditching the SAN altogether isn't something I'd really thought about, but maybe we can sell it used and use that money for other things. I'll have to see what's possible there.

ViviTheMage, renting servers is certainly one choice. We just already have an investment in quite a few servers that we don't need to replace right now, and so changing the whole setup doesn't make much sense to me. Maybe just renting the DB server is something to look into. However, owning our own servers has quite a few added benefits in my mind. When we upgrade production servers we can pull the old server down into the office and use it for in-house stuff (we need a pretty robust dev/test environment, we host internal web apps, etc.). Even if we change our production environment over to a non-owned solution, we'd still have to figure out a hardware upgrade policy for these internal servers. It just makes sense to wrap the whole thing up into one plan and manage it all together. The only other way that makes the most sense to me is to outsource *all* of it. And that's actually the direction our CTO has been trying to take us. But I think it's an ill-advised move for our company at this time.

I'm merely hoping/trying to represent the path that makes the most sense to me because he doesn't understand it, and the people above him are not getting the complete picture. They need to know how viable this option really is, then they can make the right choice -- from their perspective -- for our company.

EDIT: BTW, thanks for everybody's feedback. This has been very worthwhile for me.

 

[Axon]

Originally posted by: lxskllr

Originally posted by: Winterpool
Axon, you might want to spin off a new thread for your query, as the sort of infrastructure the others are discussing is much more hardcore.

That said, you might want to start by looking at Microsoft's Small Business Server (presuming your office is a Windows shop). If it's merely a matter of storage, WHS might almost be usable had it a licence that supported a few more users (I think it maxes at 10). Punters online will also inevitably suggest Linux / Samba, various NAS concoctions, etc, but I don't know if this office can run that sort of tech (is this your workplace or a friend's?).

For such simple needs, I think Linux would be a good choice. They require almost nothing, and then wouldn't have to worry about licensing fees. Assuming this is Axon's office, he should be able to learn all he needs to know over a weekend. It would then be fairly simple for him to administer*

All assuming he's technically inclined, and not afraid of the unknown.

I'll look into it. I'm technically inclined and have at least intermediate skill, I've just never done a server before. It's my workplace. Thanks for the advice guys!

 

[jdogg707]

What kind of a SAN is it? Fiber Channel or iSCSI? If FC, do you have a 1, 2 or 4GB connection? If iSCSI, are you sharing a switch for access? Are you using hardware or software initiators? What kind of arrays are you configuring on the SAN? For a DB server, I would recommend the Databases be located on a RAID 10 LUN for fastest access time.

While I agree that your server needs to be replaced, ignoring your storage and network infrastructure could be a pretty big mistake.

As for the idea of building a server, the biggest question mark to me would be service support. If a part failed, how long would the turn around time be until I got a replacement? That is one of the reasons you pay more for Dell or HP, they provide you with service and support that helps get you up and running sooner after a problem occurs. I would say that if your company doesn't want to invest in at least a part time Sys Admin to maintain a solution, there really isn't any reason for you to fight this battle.