Odd 6500 switch failure and UDLD

[spidey07]

This is a core data center switch in a redundant pair, so there is a secondary. This is considered primary and has mostly fiber connection out to dozens of other distribution closets. It's an older design and isn't using routed ports to distribution layer, but individual layer2 vlans for each individual link and vlan interfaces. So it's layer3 routing on each link, but the port is still a switch port. So universal link detection (UDLD) is used just as a precaution. UDLD ensures there is two way communication to avoid spanning-tree problems that can happen when you have only one way communication.

Well the supervisor (brains, does everything) failed and the secondary took over just fine. But when the primary was repaired all the fiber links were showing down.

Every switch connected to this 6509, there was no link on the 6500 ports. Turns out UDLD had shut down the ports on the distribution switches. Very strange as the only way that could happen is if control plane was totally down but yet there was still link. So the 6500 couldn't respond to UDLD frames, the other side had link, as such it's going to err-disable the ports.

I'm thinking I should set up automatic recovery of UDLD after this, never seen it happen. It's like data plane was up, control plane non existent on 6500.

 

[Cooky]

When the sup failed, were the linecards online or offline?

I've seen similar issue before, when the switches were inter-connected via 1q tunnels, so the other side didn't detect physical link-down.

 

[spidey07]

The only way this could happen is if link was still up on the far end, but didn't receive UDLD frames repeated back to it. Imagine if there were 100 switches connected and they all err-disabled their uplink.

So when the core 6500 was brought back up, no link on it's fiber ports and you had to manually go to every other switch and shut/no shut the port.

 

[ScottFern]

How did you determine UDLD was the culprit? Sys logs of some kind or just experience?

 

[Railgun]

So you had a failover and all was good, but when what was the primary sup came back online you had an issue?

Did the primary just reset itself or was it down hard? When it came back, did you fail back over to it? If so, how?

Or does that primary switch only have a single sup? If so, I think we've seen similar issues, but rarely as we don't typically run UDLD as our environment doesn't need it.

 

[Nuwave]

It would be handy to be able to have UDLD auto-recover. I dont' think my switches support that, and it would have been time savers in the past!


UDLD = UniDirectional Link Detection

scott,

Spidey would have scene the ports on the far end of the 6500 err-disabled with a cause of UDLD violation

 

[spidey07]

So you had a failover and all was good, but when what was the primary sup came back online you had an issue?

Did the primary just reset itself or was it down hard? When it came back, did you fail back over to it? If so, how?

Or does that primary switch only have a single sup? If so, I think we've seen similar issues, but rarely as we don't typically run UDLD as our environment doesn't need it.

When primary switch came back up, it's fiber ports showed no link. Since everything was running through the secondary it wasn't too big of a deal, just surprising. No redundant sups, I always use redundant chassis instead.

It looks like you can set up err-disable recovery for certain features/reasons, I wouldn't want to do it globally. If a port gets disabled, it's normally for a very good reason and you don't want it to come back up until you find out what's wrong. In this case, I do want it to come back up. If it is indeed a unidirectional link then it will just get shut back down.