A few months ago I bought some wireless gear to play with the idea of starting a WISP (see here). Nothing really ever came of that experiment, but I did learn a lot about Mikrotik RouterOS and how to do some cool things for managing end clients. It was a really great experiment involving what goes into a last-mile deployment, but doesn’t really touch on what goes into the core network of an ISP. That’s where MPLS really shines, it’s fast and works really well for creating “transparent” networks (read VPNs) across large geographical distances.

I’ve only known about MPLS for a few years now, I knew about BGP and a bit about OSPF, but I didn’t really know how large service providers bridge the gaps across continents. BGP works pretty well for large meshes of smaller ISPs, but say there is a large company that has big offices on the east coast, and on the west coast as well. BGP would be a lot of overhead to connect the two offices. A better and more elegant solution is to use MPLS to create a VPN between the two sites. Then the company doesn’t need to do complicated peering on both ends, instead they can simply “bridge” the two sites.

So what even is MPLS? It stands for Multiple Protocol Label Switching. Wait, switching? Isn’t MPLS some advanced provider routing protocol?? Well the answer is kinda. MPLS is in reality a switching protocol, it involves placing a very simple label (or stack of labels) on the packet and then forwarding it to the next MPLS router. Further down the chain, the routers look at the label on the incoming packet, do the lookup, and then “switch” the label on the packet with the new label for the next hop determined in the lookup. This goes on until the packet reaches it’s destination where the label is removed (or popped) and the packet is forwarded as whatever protocol was sent in the first place. MPLS is really a layer 2.5 protocol, its capable of encapsulation several different protocols within it at layer 3.

I said MPLS is used to create VPNs, but there’s a piece of this puzzle missing to make that happen. MPLS on it’s own can be used to quickly get packets from point A to point B, but VPNs usually imply that the paths are kept isolated, and MPLS doesn’t do that on it’s own and that’s where VRFs come into the picture. VRF stands for Virtual Routing and Forwarding, and its essentially like VLANs on steroids. VRFs allow for multiple isolated routing tables to exist on one router. This is where the magic happens because it allows the same IP addresses to be used more than once on a router and across the MPLS fabric. This is super useful for when a service provider has multiple clients that want to use, say private IP space, across an MPLS VPN to stitch two sites together.

So for this trial I ended up with 11 virtual routers. This ended up being a pretty big lab because its so easy to add devices into this fabric once the core is set up. Even the core network was really easy to setup and I was actually surprised. All of this was done with Mikrotik CHR images on ESXi to make things simple (although the networking in ESXi was a pain for this particular lab).

I started with just 5 routers and expanded into the above setup after some experimentation. I began by following the Mikrotik guide on MPLS layer 3 VPN here, and expanded to test the capabilities of the VRFs and to make the route reflector a little more obvious. I also used OSPF to distribute information about each of the MPLS routers to avoid the static routing nightmare that would have been with this many devices.

Each device in the MPLS fabric was setup with a loopback adapter for stability and easy identification. The loopback addresses are printed at each router in the image above. Every black link in the diagram is a point to point link with a /30 network to conserve addresses. On top of that each router has OSPF running on the loopback interface, and the point to point interfaces which allows all of the routers in the network to learn the loopback IPs of every other router. I used area 0 for all of the links just for simplicity, but a more appropriate OSPF configuration could be used for larger networks.

Once the OSPF network is converged MPLS and LDP can be configured on each of the routers (LDP stands for Label Distribution Protocol and it’s an easy way to distribute MPLS labels across a network for easy path creation). Each router has MPLS enabled with the transport address being the loopback address, and then LDP is enabled on all interfaces between the MPLS routers. It’s important to note that LDP is NOT enabled on the links between the MPLS fabric and C1 -C3 and CE4 and CE5. Again C1, C2, C3, CE4 and CE5 are NOT MPLS routers.

The next piece of the puzzle is the router called VRF-RR, this is a BGP route reflector and is a critical piece for moving VRF traffic across the MPLS fabric. It also has OSPF running on it, but it is not part of the MPLS fabric either because it does not need to be. It runs a BGP route reflector and will peer with all of the routers on the edge of the MPLS network. For every edge router in the MPLS network there needs to be a peer defined on this router with the route-reflect client flag enabled.

The last few pieces of the MPLS network are the actual edge routers themselves. In MPLS terminology these are called Provider Edge routers (or PE routers) because they are part of the provider network but sit on the edge between the provider core and the customers (the routers inside the core are simply called Provider routers, or P routers). With OSPF and MPLS already set up and running on the PE routers, there’s really not too much to do to get things going. First the VRFs need to be set up in the routing config to actually create the separate routing table. It’s very important to get the routing mark, and the route distinguisher right here or client traffic could leak into other clients. Once the VRFs are set up then BGP needs to be configured to peer with VRF-RR set up in the previous step. First the VRFs need to be added to the BGP instance, in my lab I used the default instance but you can get as creative as you want. It’s important when configuring the actual peering on the PE routers to change the protocol from ip to vpnv4 to advertise the appropriate routing information. All that’s left is to add the appropriate interfaces to the VRFs and configure the links between the PE and Customer routers.

For the Customer to Provider Edge connections the example on the wiki used OSPF to transfer routing information between the customer device and the VRF on the PE router and I used the same method. On the PE routers OSPF must be configured to use VRFs properly and to redistribute BGP routes as type 1. Setting up OSPF to use a VRF on a mikrotik device isn’t very hard but there is a trick that caught me up to start with. Each OSPF instance can be configured to use a specific routing table (read specific VRF), but the problem with that then becomes that the areas for the default instance will not work with any other instance. For each OSPF instance completely new areas must be set up for that instance. In my case I create a backbone area for each instance and kept it at that. If you do not create the new areas, even if you add the networks properly the OSPF instance will always show as not running. Once the new areas are configured the instances will run just like any other OSPF instance, only they will use the specified routing table to keep routes in their specified VRF.

On the client side it’s even easier to set up. All the clients need to do is set up OSPF on the link between the client router and the provider edge router and MPLS and BGP will take care of the rest automagically. A quick look at the routing tables and MPLS bindings on the PE routers should show all the routes and their respective route distinguishers, and all the label bindings as well.

So then to test the whole system I defined some loopbacks on the different client routers with completely different addresses across all the clients. I was then able to only ping the addresses of the routers within the appropriate VRF and not the ones outside of the VRF they were in. Then to fully test the VRFs I defined the same IP in two different VRFs, and lo and behold when checking the routing tables the two IPs show up, but with different route distinguishers!

By far my favorite aspect of this setup is how easy it really is to add new edge routers. When you look at the diagram, VRF-RR, Core-East, and Core-West are in blue, where as Edge-East, Edge-West, and Edge-SouthWest are colored in red and green. This is because the blue routers actually have no concept of which VRFs are passing through them, they don’t even have a single VRF defined on them, even VRF-RR. My favorite aspect of this is that the VRFs only need to be installed on the PE routers that need them! Notice that Edge-SouthWest is actually colored blue and red instead of green and red. That’s because VRF1 is the only VRF defined on that router, VRF2 doesn’t exist on it, but VRF2 still works just fine.

Well this write up is getting much longer than I expected so I’ll cut it here. This has been a pretty awesome lab and I feel like I have a better understanding of MPLS and VRFs. There’s still a lot to MPLS that I don’t know but that’s for another day.

The configurations for my setup as the currently stand, use at your own risk:

VRF-RR: https://pastebin.com/wKBs1VH4

Core-West: https://pastebin.com/SnXA3jnW

Core-East: https://pastebin.com/J9n9s10N

Edge-West: https://pastebin.com/U2MBVDw0 (this is called core west but its actually edge-west, oops)

Edge-East: https://pastebin.com/Xt5bKP9V

Edge-SouthWest: https://pastebin.com/ZDWj5B8c

C1: https://pastebin.com/Rp4rN5LC

C2: https://pastebin.com/912j1GFq

C3: https://pastebin.com/gmaWZASP

CE4: https://pastebin.com/bV5W8iqu (theres an route filter in this config to show how you would prevent routes from being broadcast across the fabric)

CE5: https://pastebin.com/geygE9Ps

Beware of the strange interface naming, doing this on ESXi had some weird side effects like adding a second ether1 when the original ether1 had already been renamed. It shouldn’t be a problem but keep in mind that for each interface that is named, it is actually a separate interface. This would have been easier with real hardware, but I made do with what I had.

Enjoy!