Linux Containers with OpenWrt

by Craig Miller



Virtual Network in the Palm of your hand

In Linux Containers on the Pi I described how to run LXC/LXD on SBCs (Small Board Computers), including the Raspberry Pi.

Linux Containers Part 2

Although you can turn your Pi into an OpenWrt router, it never appealed to me since the Pi has so few (2) interfaces. But playing with LXD, and a transparent bridge access for the containers, it made sense that it might be useful. But after creating a server farm on a Raspberry Pi, I can see where there are those who would want to have a firewall in front of the servers to reduce the threat surface.

Docker attempts this, by fronting the containers with the dockerd daemon, but the networking is klugy at best. If you choose to go it on your own, and use Docker's routing, you will quickly find yourself in the 90s where everything must be manually configured (address range, gateway addresses, static routes to get into and out of the container network). The other option is to use NAT44 and NAT66, which is just wrong, and results in a losing true peer to peer connectivity, limited server access (since only 1 can be on port 80 or 443), and the other host of brokenness of NAT.

OpenWrt is, on the other hand, a widely used open-source router software project, running on hundreds of different routers. It includes excellent IPv6 support, including DHCPv6-PD (prefix delegation for automatic addressing of the container network, plus route insertion), an easy to use Firewall web interface, and full routing protocol support (such as RIPng or OSPF) if needed.

Going Virtual

The goal is to create a virtual environment which not only has excellent network management of LXC, but also an easy to use router/firewall via the OpenWrt web inteface (called LuCI), all running on the Raspberry Pi (or any Linux machine).

Motivation

OpenWrt project does an excellent job of creating images for hundreds of routers. I wanted to take a generic existing image and make it work on LXD without recompiling, or building OpenWrt from source.

Additionally, I wanted it to run on a Raspberry Pi (ARM processor). Most implementations of OpenWrt in virtual environments run on x86 machines.

If you would rather build OpenWrt, please see the github project https://github.com/mikma/lxd-openwrt (x86 support only)

Installing LXD on the Raspberry Pi

Unfortunately the default Raspian image does not support name spaces or cgroups which are used to isolate the Linux Containers. Fortunately, there is a Ubuntu 18.04 image available for the Pi which does.

UPDATE: Sept 2019: Raspian Buster now supports Linux Containers (LXD) by using snapd . To install LXD on Raspian:

sudo apt-get install snapd bridge-utils sudo snap install core lxd

Add your userid to the lxd group, and run lxd init . Done!

If you haven't already installed LXD on your Raspberry Pi, please look at Linux Containers on the Pi blog post.

Creating a LXD Image

NOTE: Unless otherwise stated, all commands are run on the Raspberry Pi

Using lxc image import an image can pulled into LXD. The steps are:

Download the OpenWrt rootfs tarball Create a metadata.yaml file, and place into a tar file Import the rootfs tarball and metadata tarball to create an image

Getting OpenWrt rootfs

The OpenWrt project not only provides squashfs and ext4 images, but also simple tar.gz files of the rootfs. The current release is 18.06.1, and I recommend starting with it.

The ARM-virt rootfs tarball can be found at OpenWrt

Download the OpenWrt 18.06.1 rootfs tarball for Arm.

The x86 rootfs is here

Create a metadata.yaml file

Although the yaml file can contain quite a bit of information the minimum requirement is architecture and creation_date. Use your favourite editor to create a file named metadata.yaml

architecture: "armhf" creation_date: 1544922658

The creation date is the current time (in seconds) since the unix epoch (1 Jan 1970). Easiest way to get this value it to find it on the web, such as the EpochConverter

Once the metadata.yaml file is created, tar it up and name it anything that makes sense to you.

tar cvf openwrt-meta.tar metadata.yaml

Import the image into LXD

Place both tar files (metadata & rootfs) in the same directory on the Raspberry Pi. And use the following command to import the image:

lxc image import openwrt-meta.tar default-root.tar.gz --alias openwrt_armhf

Starting up Virtual OpenWrt

Unfortunately, the OpenWrt image won't boot with the imported image. So a helper script has been developed to create devices in /dev before OpenWrt will boot properly.

The steps to get your virtual OpenWrt up and running are ( init.sh is not required if you are using OpenWrt 19.07 or later, omit steps 3-6):

Create the container Adjust some of the parameters of the container Download init.sh script from github Copy the init.sh script to /root on the image Log into the OpenWrt container and execute sh init.sh Validate that OpenWrt has completed booting

Create the OpenWrt Container

I use router as the name of the OpenWrt container

lxc init local:openwrt_armhf router lxc config set router security.privileged true

In order for init.sh to run the mknod command the container must run as privileged.

Adjust some parameters for the OpenWrt container

Since this is going to be a router, it is useful to have two interfaces (for WAN & LAN), and therefore a profile for this network config must be created. Create the profile, and edit to match the config below (assuming you have br0 as a WAN and lxdbr0 as LAN).

lxc profile create twointf lxc profile edit twointf config: {} description: 2 interfaces devices: eth0: name: eth0 nictype: bridged parent: lxdbr0 type: nic eth1: name: eth1 nictype: bridged parent: br0 type: nic root: path: / pool: default type: disk name: twointf

And then edit the router container to have 2 interfaces. The only line you need to add is the eth1 line, and be sure to have a unique MAC address (or just increment the eth0 MAC).

lxc config edit router architecture: armv7l config: image.architecture: armhf image.description: 'OpenWrt 18.06.1 from armvirt/32 ' image.os: openwrt image.release: 18.06.1 ... volatile.eth0.hwaddr: 00:16:3e:72:44:b5 volatile.eth1.hwaddr: 00:16:3e:72:44:b6 ...

Now assign the twointf profile to the router container, and remove the default profile (which only has one interface)

lxc profile assign router twointf lxc profile remove router default

Download init.sh from the OpenWrt-LXD open source project

The init.sh script is open source and resides on github. To download it on your Pi, use curl (you may have to install curl )

curl https://raw.githubusercontent.com/cvmiller/openwrt-lxd/master/init.sh > init.sh

UPDATE: Sept 2019: OpenWrt release 19.07 no longer requires the use of init.sh . Skip down to Managing the Virtual OpenWrt router

Copy the init.sh to the OpenWrt container

In order to use the lxc push command the container must be running, so we'll start the router.

lxc start router

Then copy the `init.sh script to the container

lxc file push init.sh router/root/

Log into the OpenWrt container and execute the init.sh script

With the container started, the OpenWrt container boot will stall after running procd (think init in linux). By running init.sh the boot process will continue, and OpenWrt should be up and running.

Log into the router container using the lxc exec command, and run the init.sh script.

lxc exec router sh # # sh init.sh

Validating OpenWrt is up and running

You can see if OpenWrt is up and running by looking at the processes. An unhappy container will only have three. A happy container will have about 12. Type ps inside the container should look like this:

~ # ps PID USER VSZ STAT COMMAND 1 root 1324 S /sbin/procd 78 root 1064 S sh 107 root 1000 S /sbin/ubusd 196 root 1016 S /sbin/logd -S 64 213 root 1328 S /sbin/rpcd 322 root 1512 S /sbin/netifd 357 root 1228 S /usr/sbin/odhcpd 409 root 828 S /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 467 root 820 S odhcp6c -s /lib/netifd/dhcpv6.script -Ntry -P0 -t120 eth1 469 root 1064 S udhcpc -p /var/run/udhcpc-eth1.pid -s /lib/netifd/dhcp.script -f -t 0 -i eth1 -x hostname:router 508 root 1116 S /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0. 850 dnsmasq 1152 S /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pi 1191 root 1064 R ps

Additionally, if you have connected the router up the right way (e.g. WAN=eth1/br0 LAN=eth0,lxdbr0) then the WAN and LAN should have addresses. Use the ip addr to view them. (note the ip address of the WAN interface for management later)

Managing the Virtual OpenWrt router

The LuCI web interface by default is blocked on the WAN interface. In order to manage the router from the outside, a firewall rule allowing web access from the WAN must be inserted.

The standard way it to add the following to bottom of the /etc/config/firewall file within the OpenWrt container.

lxc exec router sh # vi /etc/config/firewall ... config rule option target 'ACCEPT' option src 'wan' option proto 'tcp' option dest_port '80' option name 'ext_web'

Save the file and then restart the firewall within the OpenWrt container.

/etc/init.d/firewall restart

Now you should be able to point your web browser to the WAN address (see output of ip addr eth1 address). and login, password is blank.

http://[2001:db8:ebbd:2080::93b]/

Follow the instructions to set a password, and configure the firewall as you like.

Step back and admire work

Type exit to return to the Raspberry Pi prompt. By looking at some lxc output, we can see the virtual network up and running.

$ lxc ls +---------+---------+------------------------+-----------------------------------------------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +---------+---------+------------------------+-----------------------------------------------+------------+-----------+ | docker1 | RUNNING | 192.168.215.220 (eth0) | fd6a:c19d:b07:2080:216:3eff:fe58:1ac9 (eth0) | PERSISTENT | 0 | | | | 172.17.0.1 (docker0) | fd4b:7e4:111:0:216:3eff:fe58:1ac9 (eth0) | | | | | | | 2001:db8:ebbd:2080:216:3eff:fe58:1ac9 (eth0) | | | +---------+---------+------------------------+-----------------------------------------------+------------+-----------+ | router | RUNNING | 192.168.215.198 (eth1) | fd6a:c19d:b07:2084::1 (br-lan) | PERSISTENT | 1 | | | | 192.168.181.1 (br-lan) | fd6a:c19d:b07:2080::8d1 (eth1) | | | | | | | fd6a:c19d:b07:2080:216:3eff:fe72:44b6 (eth1) | | | | | | | fd4b:7e4:111::1 (br-lan) | | | | | | | fd4b:7e4:111:0:216:3eff:fe72:44b6 (eth1) | | | | | | | 2001:db8:ebbd:2084::1 (br-lan) | | | | | | | 2001:db8:ebbd:2080::8d1 (eth1) | | | | | | | 2001:db8:ebbd:2080:216:3eff:fe72:44b6 (eth1) | | | +---------+---------+------------------------+-----------------------------------------------+------------+-----------+ | www | RUNNING | 192.168.181.158 (eth0) | fd6a:c19d:b07:2084:216:3eff:fe01:e0a3 (eth0) | PERSISTENT | 0 | | | | | fd4b:7e4:111:0:216:3eff:fe01:e0a3 (eth0) | | | | | | | fd42:dc68:dae9:28e9:216:3eff:fe01:e0a3 (eth0) | | | | | | | 2001:db8:ebbd:2084:216:3eff:fe01:e0a3 (eth0) | | | +---------+---------+------------------------+-----------------------------------------------+------------+-----------+

The docker1 container is still running from Part 1, and still connected to the outside network br0. You can see this by the addressing assigned (both v4 and v6).

The router container (which is running OpenWrt) has both eth1 (aka WAN) and br-lan (aka LAN) interfaces. The br-lan interface is connected to the inside lxdbr0 virtual network. And OpenWrt routes between the two networks.

Lastly the www container is just another instantiation of the web container (created in Part 1), but connected to the inside network. It was started with the following command:

lxc launch -p default local:web_image www

Managing the OpenWrt Firewall

In order to permit access to webservers, a firewall rule must allow the traffic.

Example 1: Create lots of webservers

After creating two, I wrote a little shell script to create the rest, called start_webs.sh

#!/bin/bash MAX=10 HTMLFILE="/var/www/localhost/htdocs/index.html" for i in $(seq 3 $MAX) do echo "starting web container: $i" free -h lxc launch -p default local:web_image w$i # update webserver home page lxc exec w$i -- sed -i "s/Container/Container w$i/" $HTMLFILE done # show the containers lxc ls

Managing the OpenWrt Firewall

In order to permit access to webservers, a firewall rule must allow the traffic. Add a new rule to allow port 80 traffic to pass to any host on the inside network (the lxdbr0 bridge).

Now point your browser at one (or more) of the webservers to see the content.

Example 2: Running a webcam in a container

This is an example from a previous VicPiMakers.ca talk, but now running inside a container.

Earlier, I created a container based on Debian 10 (Buster), the next version of Debian. Since we are running the webcam inside a container, the webcam device (/dev/video0) must be mapped into the container.

lxc launch images:debian/10 webcam # map webcam into container with 'audio' GID lxc config device add webcam video0 unix-char path=/dev/video0 gid=29 # install apps to run webcam lxc exec webcam bash apt-get install fswebcam curl python3 openssh-server sudo

Once that is done, step inside the container, and install the software we need to run the webcam and the IPv6 enabled python webserver. This takes a while because of the dependencies.

Now start fswebcam which will take a photo every 5 seconds, and -b places the program in background.

Then start the Python webserver:

# start webcam taking photos every 5 seconds $ fswebcam -r 640x480 --jpeg 85 -S 10 -l 5 -b web-cam-shot.jpg # start the personal webserver $ ~/bin/ipv6-httpd.py Listening on port:8080

Point your web browser to the webcam container on port 8080, and see the files, click on web-cam-shot.jpg. Note: the firewall will require a rule for port 8080 as well.

As you can see the firewall prevents unauthorized probing of the inside network.

Limitations of Virtual OpenWrt

There are some limitations of the virtual OpenWrt. Please see the github project for the most current list. Most notably, ssh although it works, needs improving (fixed with OpenWrt 19.07).

Address Stability

Because all of this is running on LXC, there is address stability. Not matter how many times you reboot the Raspberry Pi, or restart containers in different order, the addresses remain the same. This means the addresses above can be entered into your DNS server with out churn. Something Docker doesn't provide.

Running a Virtual Network

LXC is the best at container customization, and virtual networking (IPv4 and IPv6). With LXCs flexibility, it is easy to create templates to scale up multiple applications (e.g. a webserver farm running in the palm of your hand). OpenWrt is one of the best Open source router projects, and now it can be run virtually as well. Now you have a server farm in the palm of your hand, with excellent IPv6 support and a firewall! Perhaps the Docker folks will take note.



3 Jan 2019

23 Sept 2019, Updated 14 Jan 2020

Palm Photo by Alie Koshes