A complete tutorial to get your hands on lxc-net and give your LXC containers a proper bridge network via DHCP

After having played around a bit with LXC and discovered its main features, you may want to have a proper network setup for your containers.

There are multiple network setups possible and multiple ways to implement them. In this post, we are going to setup a bridge, using lxc-net. It requires very little configuration and should be enough for a simple LXC architecture.

More details about this bridge setup:

Containers will have an IPv4 within their own subnet

Containers will be able to access each other within this subnet

The host will be able to access the containers trough this subnet

Containers will have access to the internet thanks to the bridge interface

Note that I'm using Debian 9 for this tutorial. Also, if you're using LXD to manage your LXC containers, this isn't necessary as it does everything automatically.

Table of content

Install lxc-net

That's pretty easy, as lxc-net is a part of LXC, it's already installed.

lxc-net uses dnsmasq to manage DHCP and DNS.

To install it, run:

apt install dnsmasq-base

Do not install the dnsmasq package. Indeed, dnsmasq-base contains the binary and the doc, whereas dnsmasq also contains the service.

However, lxc-net spawns its own dnsmasq process, so if you install dnsmasq , it will run on its own and cause a conflict with lxc-net, for example :

systemctl restart lxc-net systemctl status lxc-net .... failed to create listening socket for 10.0.3.1: Address already in use Failed to setup lxc-net. ...

We need the dnsmasq binary and not the service, so we only install dnsmasq-base .

If you're using a distribution without dnsmasq-base , stop and disable the service:

systemctl stop dnsmasq systemctl disable dnsmasq

(This is basically the same thing as uninstalling dnsmasq on Debian)

I'm adding this in the post in case someone else encounters this problem, which I did because I installed dnsmasq in the first place.

That being said, back to the tutorial!

Configure the bridge

The bridge interface will not be configured by default.

Replace the line by this:

lxc.network.type = veth lxc.network.link = lxcbr0 lxc.network.flags = up lxc.network.hwaddr = 00:16:3e:xx:xx:xx

lxcbr0 is the name of our bridge on the host.

Now, tell LXC to use the bridge. Create /etc/default/lxc-net and put this in it:

USE_LXC_BRIDGE="true"

Restart lxc-net and make sure it's running:

systemctl restart lxc-net systemctl status lxc-net

The lxcbr0 interface should be up:

[email protected] ~ # ip -4 -o a show lxcbr0 3: lxcbr0 inet 10.0.3.1/24 scope global lxcbr0\ valid_lft forever preferred_lft forever

Our newly created containers will now be getting an address within the 10.0.3.1/24 subnet.

Test your bridge

Let's use our bridge!

lxc-create -t debian -n c1 lxc-create -t debian -n c2 lxc-create -t debian -n c3

lxc-start -n c1 lxc-start -n c2 lxc-start -n c3

[email protected] ~ # lxc-ls --fancy NAME STATE AUTOSTART GROUPS IPV4 IPV6 c1 RUNNING 0 - 10.0.3.32 - c2 RUNNING 0 - 10.0.3.200 - c3 RUNNING 0 - 10.0.3.77 -

There you have it! Do some tests with your containers:

[email protected] ~ # ping -c2 10.0.3.32 PING 10.0.3.32 (10.0.3.32) 56(84) bytes of data. 64 bytes from 10.0.3.32: icmp_seq=1 ttl=64 time=0.043 ms 64 bytes from 10.0.3.32: icmp_seq=2 ttl=64 time=0.087 ms --- 10.0.3.32 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1008ms rtt min/avg/max/mdev = 0.043/0.065/0.087/0.022 ms

[email protected] ~ # ping -c2 10.0.3.200 PING 10.0.3.200 (10.0.3.200) 56(84) bytes of data. 64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.071 ms 64 bytes from 10.0.3.200: icmp_seq=2 ttl=64 time=0.067 ms --- 10.0.3.200 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1024ms rtt min/avg/max/mdev = 0.067/0.069/0.071/0.002 ms

[email protected] ~ # lxc-attach -n c1 [email protected]:~# ip -4 -o a 1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 15: eth0 inet 10.0.3.32/24 brd 10.0.42.255 scope global eth0\ valid_lft forever preferred_lft forever [email protected]:~# ping -c2 10.0.3.200 PING 10.0.3.200 (10.0.3.200): 56 data bytes 64 bytes from 10.0.3.200: icmp_seq=0 ttl=64 time=0.077 ms 64 bytes from 10.0.3.200: icmp_seq=1 ttl=64 time=0.088 ms --- 10.0.3.200 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.077/0.082/0.088/0.000 ms

[email protected]:~# curl -I https://angristan.xyz/ HTTP/2 200 server: nginx date: Thu, 15 Feb 2018 23:32:51 GMT content-type: text/html; charset=utf-8 content-length: 17443 vary: Accept-Encoding x-powered-by: Express cache-control: public, max-age=0 etag: W/"4423-aJyAbkYavS/1P5xFFFgdnme4hCQ" vary: Accept-Encoding x-cache-status: EXPIRED strict-transport-security: max-age=31536000; includeSubDomains; preload

Neat, right?

Use another subnet

Here is the default subnet and DHCP range used by lxc-net:

LXC_ADDR="10.0.3.1" #Address of lxcbr0 on the host LXC_NETWORK="10.0.3.0/24" LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"

To use something else, define these options in /etc/default/lxc-net .

For example:

LXC_ADDR="10.0.42.42" LXC_NETWORK="10.0.42.0/24" LXC_DHCP_RANGE="10.0.42.100,10.0.42.200"

Restart lxc-net:

systemctl restart lxc-net

Then you have to restart the concerned containers for them to get their new IPs:

lxc-stop -n c1 && lxc-start -n c1 lxc-stop -n c2 && lxc-start -n c2 lxc-stop -n c3 && lxc-start -n c3

Check the result:

[email protected] ~ # ip -4 -o a show lxcbr0 3: lxcbr0 inet 10.0.42.42/24 scope global lxcbr0\ valid_lft forever preferred_lft forever [email protected] ~ # lxc-ls --fancy NAME STATE AUTOSTART GROUPS IPV4 IPV6 c1 RUNNING 0 - 10.0.42.185 - c2 RUNNING 0 - 10.0.42.153 - c3 RUNNING 0 - 10.0.42.117 -

Use static IPs

As you may have noticed, by default, new containers will get a random IP address within the subnet.

If you want to define your own IPs, it's simple.

Create /etc/lxc/dhcp.conf , and define you CTs as following:

dhcp-host=<ct-name>,<ip>

Example:

dhcp-host=c1,10.0.3.11 dhcp-host=c2,10.0.3.12 dhcp-host=c3,10.0.3.13

To tell lxc-net to load the configuration, add this line in /etc/default/lxc-net :

LXC_DHCP_CONFILE=/etc/lxc/dhcp.conf

Restart lxc-net:

systemctl restart lxc-net

Then you have to restart the concerned containers for them to get their new IPs:

lxc-stop -n c1 && lxc-start -n c1 lxc-stop -n c2 && lxc-start -n c2 lxc-stop -n c3 && lxc-start -n c3

[email protected] ~ # lxc-ls --fancy NAME STATE AUTOSTART GROUPS IPV4 IPV6 c1 RUNNING 0 - 10.0.3.11 - c2 RUNNING 0 - 10.0.3.12 - c3 RUNNING 0 - 10.0.3.13 -

Success! 🎉

Define a domain

A quick tip: you can define a default domain for you containers using lxc-net.

Add this in /etc/defalt/lxc-net

LXC_DOMAIN="domain.tld"

For example:

LXC_DOMAIN="angristan.xyz"

If I create a new container, it now has a FQDN:

systemctl restart lxc-net lxc-create -t debian -n ct lxc-start -n ct lxc-attach -n ct

Route a host port to a container

With our bridge, containers can access to the internet, but cannot be accessed.

We can use the iptables NAT routing table to map a host's port to a container's port, with the following command:

iptables -t nat -A PREROUTING -i <host_nic> -p tcp --dport <host_port> -j DNAT --to-destination <ct_ip>:<ct_port>

To be more specific, we're mapping a port from the host's public interface to the container's IP. Obviously, if you want your container to be accessible from the internet, use the interface ( host_nic ) where you public IPv4 is mounted.

As an exemple, I want the web server of my c1 container to be publicly accessible.

host_nic : eth0

: ct_ip : 10.0.3.11

: host_port : 80

: ct_port : 8080

We'll use:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 10.0.3.11:8080

You can now reach your container's port 8080 trough your host's 80 port (via its public IP).

IPv6

Sadly, I have not succeeded in setting up IPv6 with lxc-net, with both my public and private IPv6 blocks. Even tough the feature has been added in 2015, there is not documentation whatsoever and I have not found any tutorial.

It seems another solution is to set up our own bridge manually.

Well, another subject for another tutorial?

In the meantime, I hope my lxc-net tutorial will be useful. The post is somewhat long, but in the end it's basic IPv4 addressing, so it's easy to understand.

I'm still learning networking though, so if you have any correction/suggestion to make, I'm all ears!

Sources: