So, I ended my last blog post with a wish – “hopefully someday I can get a real switch running Cumulus to play with ;-)” Well, as it turns out, that post was somewhat popular, and caught the attention of some folks at Cumulus Networks (who kindly RT’d my tweet publicizing the post – thanks!) A day later, I was informed that I’d been placed on the list for one of their demo switches that was making the rounds. And about a month later, I received an equipment travel case with a Delta Networks 6448r switch bolted inside:

Of course, it was quite similar to many other OEM-sourced switches I’ve seen from the likes of Dell, SuperMicro, et al… on the front, 48 RJ-45 Gig-Ethernet ports, 4 SFP ports (shared with RJ-45 ports 45-48), a RJ-45 serial port and management Ethernet port; in the rear, bays for two power supplies (only one inserted into the unit), two rear-exhaust fans, and two uplink modules (again, only one came with the unit, a 2 x 10Gb SFP+ module.)

But enough about the hardware – the real meat lays inside… I took it out of the road case, put it in my lab rack, and cabled it up to the assortment of switches I have there (the DNI whitebox switch is on the shelf at the bottom):

I fired it up, and as it booted, I could already see that it was Linux… At the end of the boot messages, I was greeted by a simple login prompt:

cumulus login: 1 cumulus login :

I typed in ‘ root ‘ and then entered the default Cumulus Linux password, and there I was – at a bash prompt. I tried a few commands I use to orient myself to a new Linux environment, and all worked just as you’d expect them to on a Linux system:

root@cumulus:~# uname -a Linux cumulus 3.2.46-1cl1.5.131118 #1 SMP Mon Nov 18 16:40:12 PST 2013 ppc powerpc GNU/Linux root@cumulus:~# cat /etc/debian_version 7.3 root@cumulus:~# df -h Filesystem Size Used Avail Use% Mounted on dev 10M 0 10M 0% /dev /dev/sysroot1 68M 68M 0 100% /mnt/root-ro /dev/sda3 3.1G 104M 2.8G 4% /mnt/root-rw overlayfs:root 3.1G 104M 2.8G 4% / tmpfs 10M 96K 10M 1% /run tmpfs 10M 0 10M 0% /run/lock tmpfs 100M 0 100M 0% /tmp tmpfs 100M 0 100M 0% /var/tmp tmpfs 473M 0 473M 0% /run/shm initramfs 473M 2.1M 471M 1% /mnt/initramfs /dev/persist 124M 16K 118M 1% /mnt/persist root@cumulus:~# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 root @ cumulus : ~ # uname -a Linux cumulus 3.2.46 - 1cl1.5.131118 #1 SMP Mon Nov 18 16:40:12 PST 2013 ppc powerpc GNU/Linux root @ cumulus : ~ # cat /etc/debian_version 7.3 root @ cumulus : ~ # df -h Filesystem Size Used Avail Use % Mounted on dev 10M 0 10M 0 % / dev / dev / sysroot1 68M 68M 0 100 % / mnt / root - ro / dev / sda3 3.1G 104M 2.8G 4 % / mnt / root - rw overlayfs : root 3.1G 104M 2.8G 4 % / tmpfs 10M 96K 10M 1 % / run tmpfs 10M 0 10M 0 % / run / lock tmpfs 100M 0 100M 0 % / tmp tmpfs 100M 0 100M 0 % / var / tmp tmpfs 473M 0 473M 0 % / run / shm initramfs 473M 2.1M 471M 1 % / mnt / initramfs / dev / persist 124M 16K 118M 1 % / mnt / persist root @ cumulus : ~ #

OK, the output of that last command looks a bit different from what you’d see on your typical Linux server, but hey – it’s a switch! So then I used the iproute2 command ip with the “link show” (show layer-2 details) subcommand to see what interfaces I had:

root@cumulus:~# ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 44:38:39:00:19:97 brd ff:ff:ff:ff:ff:ff 3: swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link/ether 44:38:39:00:19:98 brd ff:ff:ff:ff:ff:ff 4: swp2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link/ether 44:38:39:00:19:99 brd ff:ff:ff:ff:ff:ff 5: swp3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link/ether 44:38:39:00:19:9a brd ff:ff:ff:ff:ff:ff ...output snipped... 51: swp49: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link/ether 44:38:39:00:19:c8 brd ff:ff:ff:ff:ff:ff 52: swp50: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link/ether 44:38:39:00:19:c9 brd ff:ff:ff:ff:ff:ff 53: swp51: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link/ether 44:38:39:00:19:ca brd ff:ff:ff:ff:ff:ff 54: swp52: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link/ether 44:38:39:00:19:cb brd ff:ff:ff:ff:ff:ff 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 root @ cumulus : ~ # ip link show 1 : lo : & lt ; LOOPBACK , UP , LOWER_UP & gt ; mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT link / loopback 00 : 00 : 00 : 00 : 00 : 00 brd 00 : 00 : 00 : 00 : 00 : 00 2 : eth0 : & lt ; BROADCAST , MULTICAST , UP , LOWER_UP & gt ; mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link / ether 44 : 38 : 39 : 00 : 19 : 97 brd ff : ff : ff : ff : ff : ff 3 : swp1 : & lt ; BROADCAST , MULTICAST & gt ; mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link / ether 44 : 38 : 39 : 00 : 19 : 98 brd ff : ff : ff : ff : ff : ff 4 : swp2 : & lt ; BROADCAST , MULTICAST & gt ; mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link / ether 44 : 38 : 39 : 00 : 19 : 99 brd ff : ff : ff : ff : ff : ff 5 : swp3 : & lt ; BROADCAST , MULTICAST & gt ; mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link / ether 44 : 38 : 39 : 00 : 19 : 9a brd ff : ff : ff : ff : ff : ff . . . output snipped . . . 51 : swp49 : & lt ; BROADCAST , MULTICAST & gt ; mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link / ether 44 : 38 : 39 : 00 : 19 : c8 brd ff : ff : ff : ff : ff : ff 52 : swp50 : & lt ; BROADCAST , MULTICAST & gt ; mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link / ether 44 : 38 : 39 : 00 : 19 : c9 brd ff : ff : ff : ff : ff : ff 53 : swp51 : & lt ; BROADCAST , MULTICAST & gt ; mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link / ether 44 : 38 : 39 : 00 : 19 : ca brd ff : ff : ff : ff : ff : ff 54 : swp52 : & lt ; BROADCAST , MULTICAST & gt ; mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 500 link / ether 44 : 38 : 39 : 00 : 19 : cb brd ff : ff : ff : ff : ff : ff

OK, 52 switch ports present and accounted for (48 RJ-45 + 4 (possible) SFP+ ports on the rear modules.) So I made up a lab plan, and then over the next few days, I tested out different scenarios involving grouping ports into VLANs (in Linux, attaching ports to software bridges, which I named br-vlan<#> ), implementing a trunk link to a Cisco switch (involving using a interface, in this case swp1 , which was the “native” [i.e. untagged] interface, and making two tagged sub-interfaces for VLAN2 and VLAN3 traffic, which are named swp1.2 and swp1.3 respectively), and a two-link LACP LAG (in Linux, a “bond”) between the CL switch, and a PC running Debian Linux (which was the testbed for my prior blogpost.) I also devoted a interface on the CL switch to my server running GNS3, which has a Cisco router lab, and I set up a OSPF area between the virtual Cisco routers in GNS3, and the CL switch (using the open-source routing suite known as “Quagga” which Cumulus ships with their OS.) I then tied in a few laptops to test connectivity, and the end result looked like this:

As we can see, there are three VLANs (VLAN1 = green = 10.1.1.0/24, VLAN2 = blue = 10.1.2.0/24, VLAN3 = yellow = 10.1.3.0/24.) The bridges on the CL switch each had an IP applied to them (in Linux, bridges can be addressed like interfaces – the equivalent of a “SVI” in Cisco-speak), which were 10.1.1.6 for VLAN1, 10.1.2.6 for VLAN2, and 10.1.3.6 for VLAN3, and were all advertised in OSPF. The routers in GNS3 had remote networks 210.1.1.0/24, and 210.1.2.0/24, which were also advertised in OSPF. So the resulting configuration file for the switch ports (which in Debian Linux, which Cumulus is based off of, lives in /etc/network/interfaces ) was thus:

root@cumulus:~# cat /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The management network interface auto eth0 iface eth0 inet dhcp # Link to Cisco SW1-3560 fa0/1 auto swp1 iface swp1 inet manual up ip link set $IFACE ip down ip link set $IFACE down # VLAN 2 sub-int on swp1 auto swp1.2 iface swp1.2 inet manual up ip link set $IFACE up down ip link set $IFACE down # VLAN 3 sub-int on swp1 auto swp1.3 iface swp1.3 inet manual up ip link set $IFACE up down ip link set $IFACE down # Link to GNS3 routers auto swp2 iface swp2 inet manual up ip link set $IFACE ip down ip link set $IFACE down # Link to Juniper EX2200 ge-0/0/1 auto swp25 iface swp25 inet manual up ip link set $IFACE up down ip link set $IFACE down # Link to Juniper EX2200 ge-0/0/2 auto swp26 iface swp26 inet manual up ip link set $IFACE up down ip link set $IFACE down # LAG of swp[47,48] auto bond0 iface bond0 inet manual up ip link set $IFACE up down ip link set $IFACE down pre-up ip link set up dev swp47 pre-up ip link set up dev swp48 bond-slaves swp47 swp48 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-lacp-rate 1 bond-xmit_hash_policy layer3+4 # Bridge for "VLAN 1" (i.e. untagged) auto br-vlan1 iface br-vlan1 inet static address 10.1.1.6 netmask 255.255.255.0 mstpctl_ports swp1 swp2 swp25 mstpctl_stp on # Bridge for "VLAN 2" auto br-vlan2 iface br-vlan2 inet static address 10.1.2.6 netmask 255.255.255.0 mstpctl_ports swp1.2 swp26 mstpctl_stp on # Bridge for "VLAN 3" auto br-vlan3 iface br-vlan3 inet static address 10.1.3.6 netmask 255.255.255.0 mstpctl_ports swp1.3 bond0 mstpctl_stp on root@cumulus:~# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 root @ cumulus : ~ # cat /etc/network/interfaces # This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The management network interface auto eth0 iface eth0 inet dhcp # Link to Cisco SW1-3560 fa0/1 auto swp1 iface swp1 inet manual up ip link set $ IFACE ip down ip link set $ IFACE down # VLAN 2 sub-int on swp1 auto swp1 . 2 iface swp1 . 2 inet manual up ip link set $ IFACE up down ip link set $ IFACE down # VLAN 3 sub-int on swp1 auto swp1 . 3 iface swp1 . 3 inet manual up ip link set $ IFACE up down ip link set $ IFACE down # Link to GNS3 routers auto swp2 iface swp2 inet manual up ip link set $ IFACE ip down ip link set $ IFACE down # Link to Juniper EX2200 ge-0/0/1 auto swp25 iface swp25 inet manual up ip link set $ IFACE up down ip link set $ IFACE down # Link to Juniper EX2200 ge-0/0/2 auto swp26 iface swp26 inet manual up ip link set $ IFACE up down ip link set $ IFACE down # LAG of swp[47,48] auto bond0 iface bond0 inet manual up ip link set $ IFACE up down ip link set $ IFACE down pre - up ip link set up dev swp47 pre - up ip link set up dev swp48 bond - slaves swp47 swp48 bond - mode 802.3ad bond - miimon 100 bond - use - carrier 1 bond - lacp - rate 1 bond - xmit_hash_policy layer3 + 4 # Bridge for "VLAN 1" (i.e. untagged) auto br - vlan1 iface br - vlan1 inet static address 10.1.1.6 netmask 255.255.255.0 mstpctl_ports swp1 swp2 swp25 mstpctl_stp on # Bridge for "VLAN 2" auto br - vlan2 iface br - vlan2 inet static address 10.1.2.6 netmask 255.255.255.0 mstpctl_ports swp1 . 2 swp26 mstpctl_stp on # Bridge for "VLAN 3" auto br - vlan3 iface br - vlan3 inet static address 10.1.3.6 netmask 255.255.255.0 mstpctl_ports swp1 . 3 bond0 mstpctl_stp on root @ cumulus : ~ #

And the routing table looked like this:

root@cumulus:~# ip route show default via 192.168.1.254 dev eth0 10.1.1.0/24 dev br-vlan1 proto kernel scope link src 10.1.1.6 10.1.2.0/24 dev br-vlan2 proto kernel scope link src 10.1.2.6 10.1.3.0/24 dev br-vlan3 proto kernel scope link src 10.1.3.6 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.102 210.1.1.0/24 via 10.1.1.10 dev br-vlan1 proto zebra metric 20 210.1.2.0/24 via 10.1.1.10 dev br-vlan1 proto zebra metric 21 root@cumulus:~# vtysh Hello, this is Quagga (version 0.99.21). Copyright 1996-2005 Kunihiro Ishiguro, et al. quagga# sh ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, A - Babel, > - selected route, * - FIB route S 0.0.0.0/0 [1/0] via 10.1.1.10, br-vlan1 K>* 0.0.0.0/0 via 192.168.1.254, eth0 O 10.1.1.0/24 [110/10] is directly connected, br-vlan1, 2d01h58m C>* 10.1.1.0/24 is directly connected, br-vlan1 O 10.1.2.0/24 [110/10] is directly connected, br-vlan2, 2d01h58m C>* 10.1.2.0/24 is directly connected, br-vlan2 O 10.1.3.0/24 [110/10] is directly connected, br-vlan3, 2d01h58m C>* 10.1.3.0/24 is directly connected, br-vlan3 C>* 127.0.0.0/8 is directly connected, lo C>* 192.168.1.0/24 is directly connected, eth0 O>* 210.1.1.0/24 [110/20] via 10.1.1.10, br-vlan1, 00:02:38 O>* 210.1.2.0/24 [110/21] via 10.1.1.10, br-vlan1, 00:01:45 quagga# exit root@cumulus:~# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 root @ cumulus : ~ # ip route show default via 192.168.1.254 dev eth0 10.1.1.0 / 24 dev br - vlan1 proto kernel scope link src 10.1.1.6 10.1.2.0 / 24 dev br - vlan2 proto kernel scope link src 10.1.2.6 10.1.3.0 / 24 dev br - vlan3 proto kernel scope link src 10.1.3.6 192.168.1.0 / 24 dev eth0 proto kernel scope link src 192.168.1.102 210.1.1.0 / 24 via 10.1.1.10 dev br - vlan1 proto zebra metric 20 210.1.2.0 / 24 via 10.1.1.10 dev br - vlan1 proto zebra metric 21 root @ cumulus : ~ # vtysh Hello , this is Quagga ( version 0.99.21 ) . Copyright 1996 - 2005 Kunihiro Ishiguro , et al . quagga # sh ip route Codes : K - kernel route , C - connected , S - static , R - RIP , O - OSPF , I - IS - IS , B - BGP , A - Babel , & gt ; - selected route , * - FIB route S 0.0.0.0 / 0 [ 1 / 0 ] via 10.1.1.10 , br - vlan1 K & gt ; * 0.0.0.0 / 0 via 192.168.1.254 , eth0 O 10.1.1.0 / 24 [ 110 / 10 ] is directly connected , br - vlan1 , 2d01h58m C & gt ; * 10.1.1.0 / 24 is directly connected , br - vlan1 O 10.1.2.0 / 24 [ 110 / 10 ] is directly connected , br - vlan2 , 2d01h58m C & gt ; * 10.1.2.0 / 24 is directly connected , br - vlan2 O 10.1.3.0 / 24 [ 110 / 10 ] is directly connected , br - vlan3 , 2d01h58m C & gt ; * 10.1.3.0 / 24 is directly connected , br - vlan3 C & gt ; * 127.0.0.0 / 8 is directly connected , lo C & gt ; * 192.168.1.0 / 24 is directly connected , eth0 O & gt ; * 210.1.1.0 / 24 [ 110 / 20 ] via 10.1.1.10 , br - vlan1 , 00 : 02 : 38 O & gt ; * 210.1.2.0 / 24 [ 110 / 21 ] via 10.1.1.10 , br - vlan1 , 00 : 01 : 45 quagga # exit root @ cumulus : ~ #

And from the Cisco router side:

The sharp-eyed might notice that there is a default route static entry in the Quagga routing table to next-hop to 10.1.1.10 (the “R1” Cisco router in GNS3) that is not making it into the FIB (and hence the Linux routing table.) This is because the management ethernet port on the CL switch ( eth0 ) is getting a default route via DHCP, and this is overriding the static route I entered in Quagga. I had the pleasure of having support from Cumulus Networks for the term of my demo, and so I opened a ticket on this, and worked with an awesome guy named Barnaby, who is their main customer support tech. As it turns out, the management port can not interoperate with the fabric ports (which is expected, and a good thing), but Linux doesn’t “know” that, and is throwing the management port default route into the (single) Linux routing table (which also is consumed by Quagga, showing up as a “K”ernel route there, which trumps the equivalent static route.) For now, this is a drawback of the current implementation, but they do have VRF (Lite) on their roadmap, which should be available in a forthcoming release (later this year, I was told.) Everything else I labbed worked as expected, without problem.

The best part of all for me is that Cumulus Linux is basically straight Debian Linux, with all the UNIX-y goodness I have come to know and love. Due to the fact that the switches they support are based on Broadcom chipsets, and those chipset APIs are covered under NDA (as well as the fact that basically to this point, Linux hasn’t been envisioned by its creators as a switch OS), there are some “closed-source” bits in the OS – mainly, as I understand it, the “ switchd ” daemon that presents the fabric ports to the kernel as “regular” ethernet ports (actually, tap devices are used, and that’s why they are named “ swp<#> ” instead of “ eth<#> ” – “swp” makes more sense in this case than the usual “tap” designator) and that ties the Linux kernel networking structures down to the Broadcom chipset ASICs that do the actual packet forwarding. Sources at Cumulus tell me that the OS is “as open as possible”, and that any code that Cumulus Networks writes to extend regular Linux commands or applications (such as Quagga, for instance) is provided upstream, as GPL requires, and other programs that Cumulus authored are made freely available on their GitHub repo at https://github.com/cumulusnetworks.