Layer 3 separation with OSPF instances – conceptual overview

The main idea is that each provider router will be aware of all customer routes while each customer router will only learn routes relevant to their network. Separate OSPF instances are running for each customer, and are enabled on provider facing interface. One OSPF instance number 1 collects routes from all customers through redistribution. We will call it the super instance. Then, the routes are spread throughout the OSPF domain. The super instance is enabled only on provider-to provider links.

All links of the OSPF super instance are in area 0. To distinguish different customer routes when they are imported into the super instance, the routes are tagged. In this scenario, I’m tagging the routes with the OSPF instance ID they are imported from. To keep customer routes from leaking, each OSPF instance (except the super instance) imports only routes with a tag equal to the instance ID. If differently tagged routes are imported it will cause route leaking which may be desired or not. As a consequence, the IDs of OSPF instances running for the same customer, but on different sites, must be equal in order to receive OSPF updates.

This model solves the problem with route separation, but the problem with overlapping customer or provider networks remains. Since provider routers use only the main routing table to store routes, customer and provider subnets need to be unique. To circumvent this restriction we will be using the traditional NAT technique. The provider may allocate a separate subnet for that purpose, with each customer network mapped to one IP. It is configured as a Loopback IP address at the customer edge (CE) router, where NAT is performed. While customers still receive dynamic updates from other sites, for outgoing traffic passing through the provider network the source IP is swapped with the Outside Global address (loopback0). This process closely resembles MPLS behavior. Control plane information, e.g. routing is using non-NAT-ed addresses and data plane traffic is transported using NAT-ed addresses.

Summary of configuration steps

Create and configure OSPF super instance number 1 on all provider routers. It collects and carries routes from all customer sites. It is enabled on links connecting provider routers. All links are in area 0.

on all provider routers. It collects and carries routes from all customer sites. It is enabled on links connecting provider routers. All links are in area 0. Create and configure per-customer OSPF instances on every provider edge (PE) router. Sites belonging to the same customer are running OSPF with the same instance IDs. The instances are enabled on the links facing the CE routers. The links may be in any area, even area 0, but for the sake of clarity I use the same OSPF areas on different sites.

Import routes from all other OSPF instances into the OSPF super instance. Tag the imported routes with a tag equal to the instance ID they are imported from.

Import routes from the OSPF super instance into other OSPF instances only when the route tag is equal to the instance ID.

Map on-site customer networks to a unique Loopback interface IP address at the CE router through NAT. This allows overlapping IP addressing schemes in provider and customer networks. The Loopback interface should be enabled for OSPF routing, so that it will be advertized throughout the provider and customer networks.

Note: No default route should be advertized by the PE routers. Otherwise all customer networks will have connectivity between each other due to classless IP routing. For the same reason customer OSPF areas cannot be stub or NSSA areas. For Internet connectivity another link should be used and default route should not be redistributed in OSPF.

Scenario network topology diagram

We assume that all routers are configured according to the above topology diagram, IP connectivity is established and OSPF instances are running. OSPF instance 1 is the super instance, OSPF instance 2 is servicing customer A and OSPF instance 3 is servicing customer B. We start the example by configuring route-maps and redistribution.

Router configuration and verification

PE1 !Configure route-map to tag when redistributing into OSPF !super instance from instance 2 ! PE1(config)#route-map TagCEA1 permit 10 PE1(config-route-map)#set tag 2 PE1(config-route-map)#exit ! !Configure route-map to filter when redistributing into OSPF !instance 2 from super instance ! PE1(config)#route-map MatchCEA1 permit 10 PE1(config-route-map)#match tag 2 PE1(config-route-map)#exit !configure redistribution PE1(config)#router ospf 1 PE1(config-router)#redistribute ospf 2 route-map TagCEA1 subnets PE1(config-router)#exit PE1(config)#router ospf 2 PE1(config-router)#redistribute ospf 1 route-map MatchCEA1 subnets ! !Configure route-map to tag when redistributing into OSPF super !instance from instance 3 ! PE1(config)#route-map TagCEB1 permit 10 PE1(config-route-map)#set tag 3 PE1(config-route-map)#exit ! !Configure route-map to filter when redistributing into OSPF !instance 3 from super instance ! PE1(config)#route-map MatchCEB1 permit 10 PE1(config-route-map)#match tag 3 PE1(config-route-map)#exit !configure redistribution PE1(config)#router ospf 1 PE1(config-router)#redistribute ospf 3 route-map TagCEB1 subnets PE1(config-router)#exit PE1(config)#router ospf 2 PE1(config-router)#redistribute ospf 1 route-map MatchCEB1 subnets

Continue with PE2 configuration which is similar:

PE2 PE2(config)#route-map TagCEA2 permit 10 PE2(config-route-map)#set tag 2 PE2(config-route-map)#exit PE2(config)#route-map MatchCEA2 permit 10 PE2(config-route-map)#match tag 2 PE2(config-route-map)#exit PE2(config)#router ospf 1 PE2(config-router)#redistribute ospf 2 route-map TagCEA2 subnets PE2(config)#router ospf 2 PE2(config-router)#redistribute ospf 1 route-map MatchCEA2 subnets PE2(config)#route-map TagCEB2 permit 10 PE2(config-route-map)#set tag 3 PE2(config-route-map)#exit PE2(config)#route-map MatchCEB2 permit 10 PE2(config-route-map)#match tag 3 PE2(config-route-map)#exit PE2(config)#router ospf 1 PE2(config-router)#redistribute ospf 3 route-map TagCEB2 subnets PE2(config)#router ospf 3 PE2(config-router)#redistribute ospf 1 route-map MatchCEB2 subnets

Verify routing table and connectivity on CEA2:

CEA2 CEA2#sh ip route 10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks O E2 10.0.1.0/24 [110/1] via 10.0.4.1, 00:00:15, FastEthernet0/0 O E2 10.0.0.1/32 [110/2] via 10.0.4.1, 00:00:15, FastEthernet0/0 C 10.0.4.0/24 is directly connected, FastEthernet0/0 C 10.0.0.5/32 is directly connected, Loopback0 O E2 192.168.0.0/24 [110/2] via 10.0.4.1, 00:00:15, FastEthernet0/0 C 192.168.1.0/24 is directly connected, FastEthernet0/1 CEA2#ping 192.168.0.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.0.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 720/916/1448 ms CEA2#

All routes are present and connectivity is established.

An unexpected problem

However, when we check CEB2’s routing table we stumble on an issue we didn’t anticipate:

CEB2 CEB2#sh ip route O E2 192.168.10.0/24 [110/2] via 10.0.5.1, 00:00:58, FastEthernet0/0 10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks O E2 10.0.2.0/24 [110/1] via 10.0.5.1, 02:27:56, FastEthernet0/0 O E2 10.0.0.2/32 [110/2] via 10.0.5.1, 02:27:56, FastEthernet0/0 C 10.0.0.6/32 is directly connected, Loopback0 C 10.0.5.0/24 is directly connected, FastEthernet0/0 C 192.168.2.0/24 is directly connected, FastEthernet0/1 CEB2#

The route to the overlapping network 192.168.0.0/24 is missing. Let’s first check if CEB1 is advertizing the 192.168.0.0/24 subnet:

CEB1 CEB1#sh ip ospf database router self-originate OSPF Router with ID (10.0.0.2) (Process ID 1) Router Link States (Area 2) LS age: 507 Options: (No TOS-capability, DC) LS Type: Router Links Link State ID: 10.0.0.2 Advertising Router: 10.0.0.2 LS Seq Number: 80000015 Checksum: 0x153 Length: 60 Number of Links: 3 Link connected to: a Stub Network (Link ID) Network/subnet number: 192.168.0.0 (Link Data) Network Mask: 255.255.255.0 Number of TOS metrics: 0 TOS 0 Metrics: 1 ...................... some parts skipped ...................... CEB1#

Indeed, the route is in the link-state database. Then, let’s check whether PE1 is receiving the update:

PE1 PE1#sh ip ospf database router 10.0.0.2 OSPF Router with ID (10.0.2.1) (Process ID 3) Router Link States (Area 2) LS age: 708 Options: (No TOS-capability, DC) LS Type: Router Links Link State ID: 10.0.0.2 Advertising Router: 10.0.0.2 LS Seq Number: 80000015 Checksum: 0x153 Length: 60 Number of Links: 3 Link connected to: a Stub Network (Link ID) Network/subnet number: 192.168.0.0 (Link Data) Network Mask: 255.255.255.0 Number of TOS metrics: 0 TOS 0 Metrics: 1 ...................... some parts skipped ......................

PE2 is receiving the update but it is not redistributed into OSPF instance 1:

PE1 PE1#sh ip ospf database | begin Process ID 1 ...................... some parts skipped ...................... Type-5 AS External Link States Link ID ADV Router Age Seq# Checksum Tag 10.0.0.1 10.0.0.3 1006 0x80000005 0x00563E 2 10.0.0.2 10.0.0.3 845 0x80000001 0x006630 3 10.0.0.5 10.0.0.4 1893 0x80000009 0x00206B 2 10.0.0.6 10.0.0.4 395 0x80000009 0x002861 3 10.0.1.0 10.0.0.3 489 0x8000000C 0x003D51 2 10.0.2.0 10.0.0.3 1006 0x80000009 0x004A45 3 10.0.4.0 10.0.0.4 141 0x8000000A 0x001A72 2 10.0.5.0 10.0.0.4 395 0x80000009 0x002368 3 192.168.0.0 10.0.0.3 1014 0x80000005 0x00AC74 2 192.168.1.0 10.0.0.4 1900 0x80000009 0x001818 2 192.168.2.0 10.0.0.4 402 0x80000009 0x001F0F 3 PE1#

There is one tagged entry for 192.168.0.0/24 (with a tag of 2) from the other customer routing instance. It is the OSPF multi-instance behavior to blame for the missing route. More details will be given at the end of the scenario. For now it is enough to remember that the route, which is received first is installed in the routing table. And in case you wonder if load balancing is possible, the answer is “no”.

Solution and consequences

Obviously, some information is lost. The route appears only once in the main routing table and once in the OSPF super instance link-state database. So the other customer (B) , which imports only routes with a tag equal to 3 will not receive it.

To resolve the issue, we use a VRF technique. The overlapping subnet is tagged with a different tag and then it is imported in both customer routing instances.

The necessary changes on the PE1 router:

PE1 ip access-list standard Net192.168.0.0 permit 192.168.0.0 0.0.0.255 route-map TagCEA1 permit 5 match ip address Net192.168.0.0 set tag 23 ! route-map TagCEA1 permit 10 set tag 2 !

The necessary changes on the PE2 router:

PE2 route-map MatchTagCEA2 permit 10 match tag 2 23 route-map MatchTagCEAB2 permit 10 match tag 3 23

In order to verify the changes:

CEB2 CEB2#sh ip ospf database | begin External Type-5 AS External Link States Link ID ADV Router Age Seq# Checksum Tag 10.0.0.2 10.0.4.1 1848 0x80000004 0x005041 3 10.0.2.0 10.0.4.1 1342 0x8000000C 0x003456 3 192.168.0.0 10.0.4.1 140 0x80000008 0x009685 23 CEB2#

CEA2 CEA2#sh ip ospf database | begin External Type-5 AS External Link States Link ID ADV Router Age Seq# Checksum Tag 10.0.0.1 10.0.5.1 661 0x80000002 0x00454F 2 10.0.1.0 10.0.5.1 661 0x80000002 0x003A5B 2 192.168.0.0 10.0.5.1 43 0x80000001 0x009D84 23 CEA2#

Now, the route to 192.168.0.0/24 appears at both sites with a tag of 23.

Note: Interesting behavior is observed when 192.168.0.0/24 is down. If it is the 192.168.0.0/24 network on CEA1 that is down, then the 192.168.0.0/24 route from CEB1 is immediately redistributed into the OSPF super instance and gets a tag 3. CEA2‘s instance is not importing this route and connectivity to the subnet is lost, as it should be. CEB2‘s OSPF instance is importing routes tagged with 3 and the route stays. But when 192.168.0.0/24 on CEA1 is up again it will not be redistributed into OSPF 1 and connectivity to network from CEA2 will not be restored until CEB1 fails. If we tag 192.168.0.0/24 from both customers with tag 23, then the route will be advertized until there is at least one customer network up. This is the punishment for routing information being lost.

To implement the second scenario, I tag the route for 192.168.0.0/24 (tag 23) from CEB1 as well:

On PE1:

PE1 PE1(config)#route-map TagCEB1 permit 5 PE1(config-route-map)#match ip address Net192.168.0.0 PE1(config-route-map)#set tag 23

No changes are necessary on PE2. At the end of the article we’ll extend the scenario to account for the missing information. Now, we leave the control plane and continue with data plane. As seen, we’ve assured routing information delivery, but data traffic is still using overlapping subnets. To solve the data plane problem, NAT is employed. If now we try a ping from CEB1 to CEB2’s IP address 192.168.2.1 with source 192.168.0.10 the result is:

CEB1 CEB1#ping 192.168.2.1 source 192.168.0.10 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds: Packet sent with a source address of 192.168.0.10 ..... Success rate is 0 percent (0/5) CEB1#

Let’s make the following changes to CEB1’s configuration:

CEB1 CEB1(config)#ip access-list standard NAT CEB1(config-ext-nacl)#permit ip 192.168.0.0 0.0.255.255 CEB1(config-ext-nacl)#exit CEB1(config)#int fa0/1 CEB1(config-if)#ip nat inside CEB1(config-if)#int fa0/0 CEB1(config-if)#ip nat outside CEB1(config-if)#exit CEB1(config)#ip nat inside source list NAT interface loopback 0 overload

If we try try the above ping command again, the result will be a 100% success.

To verify that NAT is employed:

CEB1 CEB1#sh ip nat translations Pro Inside global Inside local Outside local Outside global icmp 10.0.0.2:9 192.168.0.10:9 192.168.2.1:9 192.168.2.1:9 CEB1#

Connectivity is established but for good design practice we need to setup NAT on all customer routers (not shown).

Back to the missing route problem

Below is presented one possible solution to the problem. May be it is not the best, but it is a way out and will complement the model presented. Since routing information for the overlapping subnet is not always correct, we choose a unique piece of information for each customer to track. Of course, this is the Loopback0 interface address, which participates in customer OSPF routing. We still cannot track the state of each route in the customer network, but at least we can be sure that the CE router is up and OSPF is running. The mechanism acts as follows. On CEA2 we track reachability to CEA1‘s Loopback0 interface by a track object. If the Loopback0 interface is reachable, CEA2 will use the OSPF external route to 192.168.0.0/24. If the track object is DOWN, a static route to Null is installed for the 192.168.0.0/24 network. It will be preferred over the OSPF route and all packets will be locally dropped and not transit the provider network. The inverse behavior of the static route installation is achieved by using a second track object. It is of list type and negates the state of the first object.

Summary of the steps to perform on CEA2:

Configure track object No 1 to track reachability to CEA1 Loopback0 interface (10.0.0.1).

Loopback0 interface (10.0.0.1). Configure track object No 2 which is a Boolean list containing only one item. Add object No 1 to the list and negate its state. Because the list object has only one item the state of the object is determined by the state of that item which is the opposite of object No 1.

Configure static route to Null for 192.168.0.0/24 depending on track object No 2.

Implementation and verification steps:

CEA2 !first track object ! CEA2(config)#track 1 ip route 10.0.0.1/32 reachability CEA2(config-track)#delay down 1 up 1 CEA2(config-track)#exit !second track object CEA2(config)#track 2 list boolean and CEA2(config-track)#object 1 not CEA2(config-track)#delay up 1 down 1 !speed up convergence CEA2(config)#track timer ip route 1 !static route for 192.168.0.0/24 CEA2(config)#ip route 192.168.0.0 255.255.255.0 null0 track 2 CEA2(config)#exit CEA2#

Verify state of track object No 1:

CEA2 CEA2#sh track 1 Track 1 IP route 10.0.0.1 255.255.255.255 reachability Reachability is Up (OSPF) 1 change, last change 00:03:36 Delay up 1 sec, down 1 sec First-hop interface is FastEthernet1/0 Tracked by: Track-list 2 CEA2#

The route to 10.0.0.1 is present, state is up.

Verify the state of track object 2:

CEA2 CEA2#sh track 2 Track 2 List boolean or Boolean AND is Down 3 changes, last change 00:12:27 object 1 not Up Delay up 1 sec, down 1 sec Tracked by: STATIC-IP-ROUTING 0 CEA2#

State is down.

Verify routing table:

CEA2 CEA2#sh ip route 10.0.0.0/8 is variably subnetted, 4 subnets, 2 masks O E2 10.0.1.0/24 [110/1] via 10.0.4.1, 02:34:35, FastEthernet1/0 O E2 10.0.0.1/32 [110/2] via 10.0.4.1, 00:13:43, FastEthernet1/0 C 10.0.4.0/24 is directly connected, FastEthernet1/0 C 10.0.0.5/32 is directly connected, Loopback0 O E2 192.168.0.0/24 [110/2] via 10.0.4.1, 00:13:40, FastEthernet1/0 C 192.168.1.0/24 is directly connected, FastEthernet1/1 CEA2#

The route to 192.168.0.0/24 is obtained through OSPF.

Now we shut down CEA1 Loopback0 interface and issue the same commands:

CEA2 CEA2#sh track 1 Track 1 IP route 10.0.0.1 255.255.255.255 reachability Reachability is Down (no route) 4 changes, last change 00:00:08 Delay up 1 sec, down 1 sec First-hop interface is unknown Tracked by: Track-list 2 CEA2#

The state is DOWN.

CEA2 CEA2#sh track 2 Track 2 List boolean or Boolean OR is Up 4 changes, last change 00:00:54 object 1 not Down Delay up 1 sec, down 1 sec Tracked by: STATIC-IP-ROUTING 0 CEA2#

The state of the second object is UP and a static route to null for 192.168.0.0/24 is installed:

CEA2 CEA2#sh ip route Gateway of last resort is not set 10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks O E2 10.0.1.0/24 [110/1] via 10.0.4.1, 02:39:51, FastEthernet1/0 C 10.0.4.0/24 is directly connected, FastEthernet1/0 C 10.0.0.5/32 is directly connected, Loopback0 S 192.168.0.0/24 is directly connected, Null0 C 192.168.1.0/24 is directly connected, FastEthernet1/1 CEA2# CEA2#sh ip ospf database | begin Type-5 Type-5 AS External Link States Link ID ADV Router Age Seq# Checksum Tag 10.0.0.1 10.0.5.1 156 0x80000001 0x00474E 2 10.0.1.0 10.0.5.1 156 0x80000001 0x003C5A 2 192.168.0.0 10.0.5.1 166 0x80000001 0x009D84 23 CEA2#

OSPF route selection in multi-instance OSPF

OSPF route selection dictates that intra-area routes are preferred over inter-area routes, which are preferred over external routes. However ,this rule applies to routes learned via the same OSPF instance (Ask Cisco). Routes from different instances are compared their administrative distances. By default OSPF instances will have the same administrative distances. And if a route is installed via one OSPF instance, it will NOT be overwritten by another OSPF instance unless the route is first deleted from the routing table by the instance that installed it. So the route selection becomes non-deterministic as observed in the above example. The first to come is installed, preventing all routes for the same prefix, which are received from other OSPF instances, from being installed. In order to achieve load balancing, multiple routes to the same prefix should be received via the same OSPF instance (which is not the case in this scenario). This behavior introduces a great potential for rooting loops. External routes form one OSPF instance may be installed in the routing table instead of internal routes, provided they are received earlier. This is not a concern in this scenario for customer addresses are only used in the customer control plane. Routing and data traffic in provider network is using NAT-ed IP addresses.