Help! - Switch Issues

[Tarrant64]

I'm in need of some help. I have a group of switches that are all dropping connections. More specifically everything "pauses" if you're accessing it remotely.

One of the techs said one of the switches was getting "pause" frames due to too much traffic, but nothing's changed this week. All of the sudden every 12 hours things are getting worse.

Starting Monday all of the sudden traffic went up slightly across all interfaces on all switches. It seems every port is using the same bandwidth at times (where they would typically vary, they all sometimes say 1 megabit/second or 2-3 megabit/second). It started with just one application being affected, and has now grown to our entire network being effected. Seems to get worse as these days go on. Like something is flooding the switches? We can't narrow it down.

It was just losing remote desktop connections before, but now we're seeing anywhere from 4-10 ports drop off between switches now (only 4 switches total).

example - One of them is really bad - may replace this switch tonight:
Switch - ending in .53
G8(.53) - Dropped 86 packets
G12(.53) - Dropped 217 packets
G13(.53) - Dropped 196099 packets
G15(.53) - Dropped 1474853 packets
G17(.53) - Dropped 27 packets
G18(.53) - Dropped 13282 packets
G19(.53) - Dropped 7462 packets
G20(.53) - Dropped 50441 packets
G22(.53) - 113 Collions, no drops
G23(.53) - 127 collions, no drops
G24(.53) - Dropped 2658296 packets (uplink)

Stats on all the other ones are actually looking OK, no drops/collisions.

Anyone seen madness like this before?

 

[spidey07]

That has all the signs of a broadcast storm or bridging loop. Make sure to clear the counters on the ports, that many dropped packets is a pretty bad sign, you normally won't have any on a LAN unless there is a design flaw or a big bottleneck.

And fiber used to connect switches? Unidirectional links can cause bridging loops where only one strand of the fiber stays connected. If you've got a flaky strand or two that could cause this.

How are they all connected? Make sure to pay real close attention to spanning-tree. Make and model of the switches would help as well.

Also, is there any wireless? Bridging the wireless and wired card can and frequently does cause such bridging loops.

 

[RadiclDreamer]

I concur with spidey, however ive seem some odd code bugs cause similar issues even on gear that has been running for years without issue. For example i had a 3550 that after years of running would almost max out the cpu causing very slow terminal access but switching occurred as normal. Reboot wouldnt fix, replaced code and voila!

 

[Tbirdkid]

Ive seen this happen one time, and it was caused by one of two things. A faulty power supply in the switch, or the incoming ethernet cable had a bad connection. You could try flashing the rom on the switch, or reloading the config on the switch, but i am not 100 percent sure thats going to fix it. It could, dont get me wrong...

 

[Pheran]

Bridging loop was my first thought too, but there's not enough info to tell for sure. You don't say anything about what model switches these are. If Cisco, I highly recommend portfast on all host ports and "spanning-tree portfast bpduguard default" in the global config. Make sure you know where your spanning-tree root is and that the topology makes sense. Also, if these are Cisco and you have proper logging enabled, seeing MACFLAP messages is another hallmark of a loop.

 

[Tarrant64]

Thanks for everyone's input.

These are all cisco switches (a 540 and I think 2024's - I'll have to double-check). And I don't think STP is configured on them (this is a remote data center, I haven't gotten too involved in the setup and it's been the way it is long before I came in - just fyi).

We found the issue. sorry it took me forever to get back to post. I came back Friday evening to update and the forum was down for maintenance. There was a loop being created with a web filter we had there. We're still trying to figure out why it started to freak out as it's been there for well over a month now with out issues - but I think it's always been a problem and finally just bubbled up to be noticeable.

We're still seeing some drops periodically on some ports, but it's difficult to say whether it's still a problem or it's always been like that (unfortunately no past metrics to compare, i know i know -sucks). But the "freezing" issue is completely gone, and none of the switches spike to 100% CPU usage anymore. Anywhere from 2-6 hours and go by without any errors (meaning dropped ports, if that's "truly" what's happening"), then maybe 10 minutes of some random ports between 4 switches drop for a second, then everything looks great for awhile. Weird. Again, can't really go back yet and say for the past 30 days it's been doing that. But I do know it's not constant like it was.

 

[spidey07]

Glad it got resolved. But you really shouldn't have any drops on LAN interfaces unless there is severe oversubscription going on.

 

[Tarrant64]

Glad it got resolved. But you really shouldn't have any drops on LAN interfaces unless there is severe oversubscription going on.

Yeah, that's what I was saying. I have a meeting to discuss it with some people today. We did narrow it down to just 1 switch still getting drops on the interfaces (thousands of them) - but we're not "experiencing" big issues anymore. So it could be a bad switch, or a particular server(s) is causing the issue. It's coming to light we have problems with servers using both public/private IPs, makes my head spin looking at this mess. We are putting a plan together to fix it though.

 

[Pheran]

These are all cisco switches (a 540 and I think 2024's - I'll have to double-check). And I don't think STP is configured on them (this is a remote data center, I haven't gotten too involved in the setup and it's been the way it is long before I came in - just fyi).

Ugh, those are Linksys switches. Don't believe the Cisco moniker for a minute. I wouldn't dream of putting those in a data center.

Regardless, I'm glad you found the problem.