Spidey's net project

[spidey07]

Well its time to re-do the "core" of our network so this should be fun if not incredibly difficult and a tremendous amount of work.

I'll start out by saying the campus is a typical core-distribution-access model with about 290 servers all connected to the "core". The core consists of two cisco 6509 switches loaded up with 10/100 blades and 1000-Base GBIC based blades. Core's main funtion is server connectivity, data center routing, multicast routing/control and providing connectivity to all other buildings/distrubtion closets.

So rather than impossibly trying to upgrade these boxes with new supervisors, cards, etc I'm getting two other 6509s and loading them up with sup720 and dCEF 10/100/1000 blades. Figured now cisco has a real data center solution for 1000-Base-T (finally). These two switches will be the "server module" and will connect all servers. They'll also route and will be a fully redundant configuration with all the bells and whistles.

The old "core" will remain just that and get new horse power via sup720s and dCEF cards. Its new function will be that of L3 routing as fast as it can - connecting the distribution switches and other modules of the network like the security module (firewalls, dmzs, intrusion detection, etc)

Lots of work but maybe this thread can be a project long journal. If you have any ideas please let me know.

 

[BS911]

Sounds very difficult but a fun learning expierence (Would be for me anyway )

 

[spidey07]

Originally posted by: BS911
Sounds very difficult but a fun learning expierence (Would be for me anyway )

Its also a way for some of the heavy hitters on this board (you know who you are) to get in and get down to some serious net engineering.

One unusual thing I learned about the 6500 platform - the 6513. Only slots 9-13 have two 20 Gbs switch fabric connections, slots 1-8 have one. Serious architecture issue that is forcing me into 6509 (all slots have 2x20 Gbs switch fabric connections.

 

[ScottMac]

Sounds like a hoot. I wish I was in your neighborhood so I could "watch."

I know you're committed to Cisco, but just for S&G, check out the Extreme 10800 series switch:

1.5 Terrabit backplane
1.2 Million routes (IPv4 & IPv6)
128,000 ACLs each way, per port
48 TEN Gig ports
480 Gig ports
Stateful inspection at wire speed PER PORT (regardless of speed)

(and the list goes on for quite a while ... this is a really juicy, net-geek-woody kinda switch)

All of the above on and applied to each port, with a guaranteed latency of NO GREATER than 9 MICROseconds.

Slurp, drool, slurp.... If only it wasn't so damn purple.....

Like I said, I know you're "stuck" with Cisco, but check out the wild side to see what you're missing.....

FWIW

Scott

 

[spidey07]

certainly sounds like an awesome product Scott.

But yeah its cisco only. We use them for reasons other raw performance.

 

[bgroff]

Originally posted by: spidey07
certainly sounds like an awesome product Scott.

But yeah its cisco only. We use them for reasons other raw performance.

Heh, its EIGRP isn't it? They've got ya locked in, don't they?

On a more serious note, I like the idea of the "upgrade journal." Its been a while since I've got to play with the 6500 series platform. Last time I played it was still Sup2 with the CatOS/IOS.

 

[ScottMac]

Yeah yeah, I know.

I was brushing up on my Extreme stuff for a class I'm presenting and there it was ...

BGroff: There are many reasons outside of performance for going with a particular vendor's product suite. In this case, I believe we'd be talking support mostly (and that's a biggie), guaranteed compatibility, industry leadership ... scope of products ... Cisco products are rarely (if ever) the best performance within a given class of products, but their support (especially to Enterprise customers) is very hard to beat.

EIGRP ... Yep, I'm sure that's probably it in this case though :beer: : It makes all that routing crap so much easier ....Spidey being the lazy slacker that he is .... he's gotta take the easy out.

(Ducking)


Not that Extreme doesn't also provide excellent service and support, they do, I love 'em. But when nearly everything in your Enterprise has a little bridge medallion on it's face, you pretty much gotta hang with it.

FWIW

Scott

 

[Buddha Bart]

Is there a dark chance in hell you/your-department are hiring? I just graduated, will have my ccna in two weeks, and drool over this kinda work.

 

[bgroff]

Originally posted by: ScottMac
Yeah yeah, I know.

I was brushing up on my Extreme stuff for a class I'm presenting and there it was ...

BGroff: There are many reasons outside of performance for going with a particular vendor's product suite. In this case, I believe we'd be talking support mostly (and that's a biggie), guaranteed compatibility, industry leadership ... scope of products ... Cisco products are rarely (if ever) the best performance within a given class of products, but their support (especially to Enterprise customers) is very hard to beat.

EIGRP ... Yep, I'm sure that's probably it in this case though :beer: : It makes all that routing crap so much easier ....Spidey being the lazy slacker that he is .... he's gotta take the easy out.

(Ducking)


Not that Extreme doesn't also provide excellent service and support, they do, I love 'em. But when nearly everything in your Enterprise has a little bridge medallion on it's face, you pretty much gotta hang with it.

FWIW

Scott

True dat on Cisco support. As long as you're forking over the buck$ for smartnet, support is top notch. And the CCO site is hard to beat. After working with other brands of networking gear, you find that the shear amount of documentation available on CCO is nearly impossible to beat (and it only gets better with the more access privledges available).

 

[spidey07]

Originally posted by: Buddha Bart
Is there a dark chance in hell you/your-department are hiring? I just graduated, will have my ccna in two weeks, and drool over this kinda work.

Buddha, unfortunately no.

To scott and bgroff...
How dare you insult me by saying I run EIGRP?

Next on the list and we can keep this thread going as it truly will give others the chance to see what "real network engineering" is all about.

Its not about the product or the manufacturer. Its not about the best performer. Its all about "as a director/architecht will I get called in the middle of the night?" For me the answer better be "hell no". I have a life and want to live it.

So we can go back to industry stats and total cost of ownership analysis and realize that support of a global network is by far the biggest expense there is. People aren't cheap and dammit they don't depreciate (as much as I wish they would)

So NO, the network is OSPF with about 9 areas. Can EIGRP converge faster? Yep. Does EIGRP have the support of being an open routing protocol that scales/summarizes really well? Nope.

Next - on the whole cisco thing. Here's my view:
Cisco makes good products that do what they are supposed to do backed by a pretty darn good support staff (they're getting worse though ) And sometimes you have to go with the market. Imagine you have 12-15 level 1/2 support personell that have lived and breathed cisco for 3-7 years. Do you think I would pick another vendor? Does anyone honestly think I want to live with the hassle of multiple maintenance contracts, added training, prolonged mean time to resolution and mixed network management platforms? No...I don't.

So I'm like most enterprise customers. Cisco is to networks what MS is to operating systems. It might not be the greatest but if you look at the big picture in terms of support/training/flexibility they are the right choice. It is sometimes amusing reading the posts like "well I can make linux do anything a cisco router can do".

Well sure you can. But I have a business to take care of and the last thing I want my people doing is taking 12 hours to fix something they mucked up because they can't pick up the phone and get the guys who wrote the software on the phone. The internet is NOT the end all be all source of information - just because you read it somewhere on the net does not make it true nor does it make it the right choice.

Oh - update. I had a vendor pow-wow at lunch and we'll be seriously looking at 10 gig connections between the real core of the net and the server module. It only makes sense.

-off soapbox- :beer:

 

[Garion]

OK, I'll bite..

Why not consider combining the fabric-enabled 6500's that you use for L3 with your server block - Any reason to use them as core only? Seems like a waste of a big chassis and high dollars if you're not going to get ANY connectivity out of it.

At my last job (where I actually RAN the network, as opposed to now where I just consult on networks and Internet/web/proxies/load balancers to customers) in each data center, we put our L3's at the core, then connected the most important or very high volume servers directly to them. From there, we ran tagged interfaces to child L2 switches that held the rest of the data center.

This saves you from having to go 10GB to the distribution switches from the core - There's really no need for it, IMHO, unless you have some HUGE apps that talk to the clients. Proper two/three tier designs should have all of the active devices on one switch, and secondary devices on a second switch. That way, traffic doesn't have to exit the local fabric. The ONLY place I can see needing 10GB is between the L3's - Gotta love HSRP.

Another thing to be very careful of is how you connect your distribution switches to the core - If Switch A and B are L3, and C and D are L2 only, what's the L1/2 connectivity? A->B is obvious, but should A->C, B->D AND C->D and let spanning tree kick in?

Something that I've found is that it's very, very tempting to go way overboard on the bandwidth and connectivity within the data center. The amount of bandwidth you need within an environment directly correlates to the amount of bandwwidth you have going OUT of the data center. For example, we have a MAN in the Northwest that connects each building to the DC - The MAN ring is built from redundant OC3's. There's simply not enough MAN bandwidth for us to need 10GE between switches.

Of course, there are things that can throw a wrench into this and make my opinion moot. For example, if you're using physical distribution of switches to different parts of the data center to manage cabling, it can change things considerably. You might end up needing to distribute a 2/3-tier app among lots of different switches. At the last job, we had this happen a lot. One switch served one row in the data centers, and a row was dedicated to Intel racks, Sun racks or other devices. If you have an app that has a LOT of bandwidth between Web/App/DB tiers, you might need more than a channeled gig trunk. Backups are the same way (or used to be - SAN attached backups have changed things quite a bit lately).

I'm sure a lot of this is driven by the "My servers all have gig copper ports and I want to use them, so that will be a lot more bandwidth". But do you really need it? Most apps nowadays are designed to minimize bandwidth as opposed to the behemoths of yesteryear - File-based databases, running apps off of servers, etc.

In my current company, our data center network is actually fairly similar to what you describe. We have a pair of heavy-hitter L3's at the core. All they do is distribute out to various "modules". Those "modules" include two classic L3/distribution switch environments for internal apps (one wasn't enough, and we ran out of ports after about 8 6509's), many firewall pairs for Internet-facing environments, as well as connectivity to the MAN - Think of them as TRUE core switches, that don't have anything connected to them other than other L3 devices. Even for a 40,000+ square foot data center with thousands of servers, we still only have FEC'ed gigabit (2Gb/s per channel) coming from the core switches to the other modules and haven't had ANY problems. Of course, we do most of our server L3 on other devices, but all in/outbound traffic goes through these beasts. This might be confusing so...

Buildings -> MAN -> Data center core switches -> module L3 switches -> module L2 switches -> servers.


For reference purposes, what kind of utilization are you seeing on your current trunks, and what's the growth pattern? How many servers do you expect will have gigabit? Do you really expect that this will be a net growth in traffic, vs just getting it through to the core faster? How much bandwidth do you have OUT of the data center? What are your apps like? Efficient client/server or old bandwidth-intensive junk?

One other thought.. You could always change your strategy a bit - Get one set of L3's to handle all traffic outside of the data center and aggregate it all together - Access switches, etc. and leave one set as the data center L3's and put a fat trunk between them. It would help ease the load on the L3's and provide some more control points for ACL's, etc.

- G

 

[spidey07]

Garion,

Spoken like a true net geek. You're thoughts are what have been going thru my mind. you're coming to town for derby right?

 

[Garion]

I wish.. It's a long ways from Seattle to Kentucky, even for a chance at the good stuff you sneak out the back door at work. Got a new baby coming in a few days, too. Wife might not appreciate me hopping on a plane when she's at home with a 3-year-old and a 3-month-old. *grin*

- G

 

[cmetz]

spidey07, have you lab tested the sup720s? Scouting reports are that they're disappointing, especially for the $$. Then again, it's par for the course on the 6500.

Go buy an entry level Extreme switch (48si) and Juniper router (M5) just for play. Then make sure to give your Cisco sales rep a guided tour where he just happens to pass them buy. Your discount increase will pay for those toys immediately

 

[spidey07]

Originally posted by: cmetz
spidey07, have you lab tested the sup720s? Scouting reports are that they're disappointing, especially for the $$. Then again, it's par for the course on the 6500.

Go buy an entry level Extreme switch (48si) and Juniper router (M5) just for play. Then make sure to give your Cisco sales rep a guided tour where he just happens to pass them buy. Your discount increase will pay for those toys immediately

LOL! 50% is pretty decent but I'm sure the rep would have a heart attack. I really have to go with the sup720s given the sheer density of 1000 Base-T. We're a large SAP shop so there is a ton of purple back there.

I think I've made an executive decision. I will be using 10-gig to connect the server module to the "core" price wise it is about the same as a 24 port gig card and I like the cleanliness of only having a few links between switches.

Any thoughts?

 

[n0cmonkey]

Is this how normal people feel when I start talking about Unix?

 

[spidey07]

Originally posted by: n0cmonkey
Is this how normal people feel when I start talking about Unix?

Networking and Medicine are by far the most jargon/acronym filled professions.

 

[knighthawkaz]

Originally posted by: spidey07

Originally posted by: n0cmonkey
Is this how normal people feel when I start talking about Unix?

Networking and Medicine are by far the most jargon/acronym filled professions.

Couldn't agree more myself. This sounds like an interesting project. I am going to be going starting to study soon for my CCNA. Then hopefully I will know what you all are talking about.

 

[SaigonK]

Nice project Spidey!
Of course I am in an all Nortel arena, they do have some great units, the 8600 series chassis is pretty damn sweet, and their Layer-7 Alteon products are the best out there (IMHO).
You gonna upgrade all of your Novell servers to 6.5?

 

[spidey07]

Originally posted by: SaigonK
Nice project Spidey!
Of course I am in an all Nortel arena, they do have some great units, the 8600 series chassis is pretty damn sweet, and their Layer-7 Alteon products are the best out there (IMHO).
You gonna upgrade all of your Novell servers to 6.5?

Novel is gone. We're MS now.

 

[cmetz]

spidey07, check how much throughput Cisco's 10Gb/s card has. If memory serves, it's a good bit less than 10Gb/s (like 6). Just make sure that's okay for your application. (if you're using them to interconnect core switches, be aware of the real throughput when you plan densities)

10Gb/s is clearly the long-term best way to interconnect things. It's just wicked expensive right now, partly because of market position and partly because of the optics.

Only having a few links between the switches is a big win. Trying to do link aggregation between switches is a headache.

Lab test aggressively and make sure your sales guy knows you're doing it. Cisco's quite bold sometimes in stretching the truth on their performance claims. You need to know what their equipment's really going to do for your own network engineering purposes.
And knowing where it doesn't live up to the marketing will help you negotiate with your sales rep. Cisco's margins are very fat, and you should squeeze out as much of that as you possibly can. It's not that Cisco makes products that are bad, it's that their price/performance is bad; if you can get the price down low enough, that's no longer a problem.

If you're getting a pretty standard 50% off list, you're starting with a good discount schedule.

 

[AFB]

Sounds fun ! :beer:

 

[SaigonK]

Originally posted by: spidey07

Originally posted by: SaigonK Nice project Spidey! Of course I am in an all Nortel arena, they do have some great units, the 8600 series chassis is pretty damn sweet, and their Layer-7 Alteon products are the best out there (IMHO). You gonna upgrade all of your Novell servers to 6.5?

Novel is gone. We're MS now.

NOOOOOOOOOOOOO!

 

[Mucman]

Cool stuff! Glad to see the titans of the Networking forum battling again

I haven't kept up to date with networking since I'm thinking of going the DBA route. I'm a sucker for designing 8 table join SQL queries. I think I'll keep this networking thing as a hobby (what else am I supposed to do with the 10 computers I got at home )

 

[spidey07]

Mucman,

I was hoping we could keep a running thread of progress/decisions. I'm sure all the heavy hitters are salivating and were all friends here so we can argue all we want!

:beer: