OSPF Ring Questions

[cpals]

We've never had a ring setup before and it's always been a hub-spoke type of design. Recently we've been fortunate enough to obtain our own dark fiber to create a ring of six main locations. We're looking at doing this by using OSPF to allow for multipathing, etc. My coworker and I recently took a 1 week course on OSPF but unfortunately it didn't dive deep enough for what I'm trying to accomplish.

Also, the two connections on each path (10Gb SFPs) are in a LACP group.

Here are a couple designs we're playing with...
http://dl.dropbox.com/u/95425426/OSPF_1.pdf
http://dl.dropbox.com/u/95425426/OSPF_2.pdf

What we're trying to accomplish:

• Full redundancy between sites. We are also using four-strands of fiber in each direction to achieve 20Gb/s and also redundancy within the optical equipment.
• We have a DR location that I plan on bringing up a vSphere cluster... needs to have the server vlan spanned to it for VMware SRM to function the best. (We don't want to have to mess with re-IPing everything during a failover).
• Not cause any loops
• Main site has the Internet for all sites, so the default route would have to go back to the main site.
• Main site also has MetroE connections that are not participating in the OSPF, but route through the main site to get to the other sites on the ring.

Questions I had:

• Right now everything is in Area 0 because most of the sites are just a /24 for the users and I couldn't think of a good reason to make more than one area?
• We were debating about a /30 (Diagram 1) between the sites on the ring or put them all in the same vlan on a /24 (Diagram 2). I seem to remember for OSPF it talks via multicast which to me means it won't get past to the other locations if everything is a /30? We did do a test setup of the /30 ring and each segment has it's own DR/BDR selected.. so essentially my ring has 6 DR and BDR routers?
• If we put everything in a /24 vlan I would think there would be only one DR/BDR and all the sites would be able to talk to each other via multicast... not sure if this is better or not since you would also have more broadcast traffic, etc. Also, if we put all in the same vlan wouldn't we have spanning tree issues?
• We were seeing some weird problems with the /30 diagram where if I pulled one fiber the location would have no traffic for around a minutes and then it would finally 'notice' the other direction and start routing... in our class they mentioned turning off spanning tree on the OSPF ports and once I did that on all the routers it started working very fast for cutovers.

Thanks for any help or ideas!

 

[Gryz]

Over a WAN, you really don't want to do bridging (aka level-2 switching).
So consider all the point-to-point physical links between your sites as point-to-point links in your logical design. You *really* do not want to run STP over your expensive WAN links.

Put everything in area-0. That keeps it simple. Don't bother with other areas.

 

[freegeeks]

I would probably run mpls on top and use L2 or L3 services a needed

 

[drebo]

I would use OSPF only to establish routing between loopback addresses on each router. From there, I would use an iBGP mesh, which is much, MUCH more customizable. MPLS isn't really necessary, though wouldn't hurt. So, your site-to-site links should be /30s (in Area 0) with a /32 loopback (in a different area, unique per location) on each router, and then an iBGP mesh between the loopbacks on all of the routers to exchange the actual routes.

OSPF only supports equal cost load balancing, so you wouldn't be able to use both directions of the ring at the same time. iBGP will allow you to do quite a lot more with path selection and is, overall, a much more robust protocol. It would also allow you to migrate easily to MPLS transport if you needed to for any reason.

 

[freegeeks]

I would use OSPF only to establish routing between loopback addresses on each router. From there, I would use an iBGP mesh, which is much, MUCH more customizable. MPLS isn't really necessary, though wouldn't hurt. So, your site-to-site links should be /30s (in Area 0) with a /32 loopback (in a different area, unique per location) on each router, and then an iBGP mesh between the loopbacks on all of the routers to exchange the actual routes.

OSPF only supports equal cost load balancing, so you wouldn't be able to use both directions of the ring at the same time. iBGP will allow you to do quite a lot more with path selection and is, overall, a much more robust protocol. It would also allow you to migrate easily to MPLS transport if you needed to for any reason.

I agree that implementing mpls is a bit more work but the size and capacity of his network are already ideally suited for mpls services. You get powerfull mpls features like fast reroute and mpls traffic engineering
and I would use is-is instead of ospf for the underlying igp

second best option is what drebo described :-;

 

[drebo]

Meh...for 6 sites, IS-IS doesn't get you anything substantial over OSPF...especially when your routes are all handled via BGP/MPLS anyway.

MPLS is a lot of fun, though.

 

[cpals]

That all sounds nice, but unfortunately I have no familiarity with MPLS or iBGP.

Also, drobo, that is what I have setup now for testing is the /30 for connectivity and the loopback for the router-id of the ospf. What would be the purpose of the /32 in a different area?

 

[drebo]

That all sounds nice, but unfortunately I have no familiarity with MPLS or iBGP.

Also, drobo, that is what I have setup now for testing is the /30 for connectivity and the loopback for the router-id of the ospf. What would be the purpose of the /32 in a different area?

Expandability. If you were to ever set up other routers at those sites, you'd want them in a different area anyway.

Also, it's a useful way to identify what's where.

Just because your network is simple doesn't mean you can avoid best practices!

 

[cpals]

Hmmm. I didn't really say it cause didn't think it was pertinent? Off of the main site we also have two juniper srx1400 that terminate around 15 site to site vpns in a MetroE cloud. Would I want those participating in ospf and area even if they pretty much only have one route out?

 

[Cooky]

Here's what I recommend:
1. use routed /30 (or /31) ports instead VLAN/SVI

2. run UDLD, or equivalent if platform supports it

3. Ethernet defaults to broadcast as network type.
Change interfaces to point-to-point.
This will eliminate the step where neighbors have to elect DR/BDR

4. if fast re-convergence is desired, reduce timers from the default 10/40 to 1/4.
I'd avoid sub-second timers, as it usually puts a dent on your CPU.

5. The interfaces directly connected on the actual ring can participate in area 0.
For the local subnets behind each node though, you can put them in separate areas, so that you can summarize them.
This will make your routing table clean, and easy to troubleshoot.

6. If your platform still reference any interface greater or equal to 100M as 1, make sure you change it accordingly.
This is so that if later on you add additional nodes w/ only single 10G, or 1G links, the nodes would go through the path w/ higher bandwidth.
========

Now, to fulfill your L2 requirement (using same IP, or expand a VLAN between two sites):
You can run MPLS, and utilize EoMPLS, but this will introduce additional layer of complexity.
It's pretty darn cool though; we run our own MPLS on our IP core, which was one of the coolest things I've done so far.

Or, you can use OTV, or equivalent, which has its own complexity, but much easier to deal w/ than MPLS.

BGP could be very beneficial, which provides a lot of granularity, but convergence time is much slower than IGP.

 

[cpals]

Cool. Thanks for the detailed reply Cooky. In regards to number one... Our new equipment is Enterasys and I do believe in my readings that they only route on the vlan and not the port. They stress that you cannot put an ip on a port and it must be in a vlan. Not sure if that complicates things.

 

[Cooky]

Not being able to do routed port is a big drawback, but I suppose it's not end of the world.
Just make sure segment vlan's (100 - 105) only exist on the ring, and nowhere else.
Also set them up as access ports, instead of trunk. (unless you really have a need to extend vlan's between sites, which cannot be achieved using any other solution)

Not sure if you'd want them to be portfast / edge ports...on one hand you can achieve fast re-convergence, but on the other, you may end up w/ bad SPT issue.

 

[eljustinoid]

Interestingly I'm working on the same dilemma - one /30 per link, or a /24 or similar for the whole OSPF area.

The layer 2 ring uses a proprietary loop-prevention protocol rather than spanning tree.

Would be interested in discussing privately as well.

Thanks!

 

[alkemyst]

Over a WAN, you really don't want to do bridging (aka level-2 switching).
So consider all the point-to-point physical links between your sites as point-to-point links in your logical design. You *really* do not want to run STP over your expensive WAN links.

Put everything in area-0. That keeps it simple. Don't bother with other areas.

I agree. This should be simple P2P links and doing routes. Using STP on everything inside each of those endpoints.

Ring topologies usually aren't needed with a WAN/Cloud/et al.

For most folks AREA 0 is all they need. Only the architect will know if multiple areas is proper and also if areas not directly connected to AREA 0 is more proper.

A lot of folks over-design their routing topologies.

 

[cpals]

Interestingly I'm working on the same dilemma - one /30 per link, or a /24 or similar for the whole OSPF area.

The layer 2 ring uses a proprietary loop-prevention protocol rather than spanning tree.

Would be interested in discussing privately as well.

Thanks!

Just wanted to update this a little. We're still doing the /30 per link on the OSPF ring, but instead of connecting layer 2 all the way around for our vlans we are going to do a GRE tunnel on top of the OSPF. This way there is no spanning tree issues and we use the OSPF for the traffic routing.

We haven't turned it on just yet as it's an extra license cost from our switch vendor.