Seeing how I'm just a lowly CCNA working towards my CCSP and CCNP and should have both by the time I'm done with college (about 2-3 more years), I'm really enjoying this conversation. I dont have any experience in a big datacenter...a 15-server, 100-user, 4-site network is the biggest I've worked with. However, I do want to throw something out for the sake of curiosity as I've been thinking about it lately.
Spidey, you use Spanning-Tree on your network right? If you have multiple VLANs, couldnt you loadbalance the VLANs across Spanning-Tree primary and secondary links? Example....I've got Gig0/1 on my switches setup as the primary link, Gig0/2 is in Alternate-Blocking mode and I've got 4 VLAN's on the network. Couldn't I setup VLAN 1 and 3 to use Gig 0/1 as the primary and Gig 0/2 as the alternate, then setup VLAN 2 and 4 to use Gig0/2 as primary and Gig0/1 as alternate? That would keep both links forwarding, and would load balance your traffic, but still give you redundancy. I know that doesnt help you with your L3 issues in your datacenter...but I just wanted to throw that out there. It seems to me that it might give you more efficient use of some of your bandwidth. You might not even be using VLANs...I dunno. I've heard from a few people that Cisco is going to get rid of VLANs by 2006. Supposedly VLAN's arent even covered on the new multilayer switching exam that's in testing right now.
I agree that you should put pressure on your Cisco rep. Even in our relatively small business, I told our rep that we were evaluating Dell and 3Com switches when we were testing for a "big" network upgrade and that we already had a Dell, had tested it extensively and were very happy with it. That alone got them to bump up our discount. We were getting around 38% before, after I told him that, we ended up getting 4 C2950's, and 6 2940's for the price of 2 C2950's. You do the math...but they basically gave them away. Apparently Cisco REALLY REALLY has it in for Dell.