Over the past few months, I’ve been working on Bee2. It’s a provisioning system for automating the process of building, running and maintaining my own websites and web applications. In previous tutorials, I had gone over provisioning servers using APIs, configuring those servers for remote Docker administration via a VPN, and automating LetsEncrypt and HAProxy in containers. Bee2 has gotten mature enough that I’ve finally migrated many of my production sites and web applications to it, including this website. In this post, I’ll go over some of the additional challenges I faced with IPv6, and refactoring containers to allow live HAProxy refreshes.

More IPv6 Woes

When enabling IPv6 in Docker, the Docker Engine requires an IPv6 subnet that it uses to assign each container a public IPv6 address. Some providers give each server a fully routable /64 IPv6 network. Although Vultr does provide a /64 subnet to each server, these subnets are not fully routed. In these situations, the default gateway will send a neighbor solicitation request to incoming packets bound for that subnet, and the Docker host will need to run an NDP Proxy Daemon to automatically respond to these requests.

I created the following Ansible role to install and setup the ndppd daemon:

- name: Install ndppd action: apt name=ndppd - name: Configure ndppd template: src=ndppd.conf.j2 dest= owner=root group=root mode=0644 - name: Enable proxy_ndp for public adapter sysctl: name=net.ipv6.conf.ens3.proxy_ndp value=1 state=present - name: Restart ndppd Service service: name=ndppd state=restarted

The Ansible template for the ndppd.conf configuration file is fairly straightforward as well, using the same /80 subnet we previously assigned to the Docker daemon:

proxy ens3 { timeout 500 ttl 30000 rule /80 { static } }

With this configuration, my containers could receive incoming requests from exposed Docker ports, and they could send/receive ICMP requests over IPv6 (ping6) to and from the outside world. However, they could not establish TCP connections over IPv6 to any public address. This particular issue stumped me for a while, and I eventually discovered that my ufw firewall needed the default forward policy to be set to ACCEPT, in order for IPv6 TCP connections to be established in both directions. I added the following task to my firewall role in Ansible to to set the default forward policy:

- name: set UFW default forward policy to ACCEPT lineinfile: dest: /etc/default/ufw line: DEFAULT_FORWARD_POLICY="ACCEPT" regexp: "^DEFAULT_FORWARD_POLICY\\="

This brings us to the last IPv6 issue I encountered, and which required the most refactoring of Bee2. HAProxy has configuration options to inject an X-Forwarded-For header to the backend servers, indicating the real IP address a web browser is connecting from. For IPv4 connections, this header was getting populated with a real public IP address from the Internet. However for IPv6 connections, this header would contain a 172.17.x.x address from the Docker bridge adapter. It turns out that for IPv4, exposed ports are mapped to the listening container and translated in such a way that the container does get the actual public IP of whatever is connecting to it. However since containers are given an actual public IPv6 address, incoming requests to the host’s IPv6 address are converted to IPv4 in a translation layer!

The end result is that all IPv6 connections to my website will seem to be coming from one IPv4 address, that of a private Docker address, making any type of log analysis impossible. I could use the public IPv6 address of the container itself, opening firewall rules as appropriate and adjusting DNS records, but in my original implementation I used Docker’s default bridge for networking. It’s impossible to assign a static IPv6 address on the default bridge.

The default bridge network, as well as container linking, are not recommended for production. I avoided user defined networks in the past, but with this limitation, I had to finally bite the bullet and setup my networking correctly. First I split my /80 subnet into two /96 ranges. One would be assigned to the default bridge network (which would go unused, and which must exist because it cannot be deleted in Docker), and the other would go to my user defined network. I added a section in settings to allow for three additional Docker/IPv6 settings, a suffix for the bridge network, a suffix for the user defined network and the trailing part of the static IP to use for the HAProxy/web load balancer.

servers: web1: plan: 202 # 2048 MB RAM,40 GB SSD,2.00 TB BW os: 241 # Ubuntu 17.04 x64 private_ip: 10.10.6.10 ipv6: docker: suffix_bridge: 1:0:0/96 suffix_net: 2:0:0/96 static_web: 2:0:a

The Vultr provisioner uses these settings to establish a static web IPv6 address which is maintained in the Bee2 state file:

def web_ipv6 @servers.select { |name, s| s['dns'].has_key?('web') }.each { |name,cfg| if not @state['servers'][name].has_key?('static_web') ipv6 = @state['servers'][name]['ipv6']['subnet'] + cfg['ipv6']['docker']['static_web'] @log.info("Creating IPv6 Web IP #{ipv6} for #{name}") @state['servers'][name]['ipv6']['static_web'] = ipv6 end } save_state end

I adjusted Bee2 to check if a user defined network exists and, if not, create it using the second /96 defined by suffix_net , before running any additional Docker commands.

def establish_network(server) ipv6_subet = state['servers'][server]['ipv6']['subnet'] ipv6_suffix = @config['servers'][server]['ipv6']['docker']['suffix_net'] ipv6 = "#{ipv6_subet}#{ipv6_suffix}" if Docker::Network.all.select { |n| n.info['Name'] == @network }.empty? @log.info("Creating network #{@network} with IPv6 Subnet #{ipv6}") Docker::Network.create(@network, {"EnableIPv6" => true, "IPAM" => {"Config" => [ {"Subnet" => ipv6} ]} }) end end

Previously, the following code in the create_containers function in docker.rb was used to link containers defined by the links in the configuration file:

... 'HostConfig' => { 'Links' => (link.map{ |l| "#{@prefix}-#{cprefix}-#{l}" } if not link.nil?), 'Binds' => (volumes if not volumes.nil?), 'PortBindings' => (ports.map { |port| { "#{port}/tcp" => [{ 'HostPort' => "#{port}" }]} }.inject(:merge) if not ports.nil?) ...

Linking containers in Docker has been deprecated, so I removed the configuration option for defining links and instead connected each container to the user defined network mentioned previously. The following code also allows for setting a static IPv6 address for a container if it has static_ipv6 defined:

... 'NetworkingConfig' => {'EndpointsConfig' => {@network => { 'IPAMConfig' => {'IPv6Address' => static_ipv6 }.reject{ |k,v| v.nil? } } } }, 'ExposedPorts' => (ports.map { |port| {"#{port}/tcp" => {}}}.inject(:merge) if not ports.nil?), 'HostConfig' => { 'Binds' => (volumes if not volumes.nil?), ...

Changes were also made in vultr.rb to use the static web address for all AAAA (IPv6) DNS records, as well as the Ansible firewall rules to allow incoming 80/443 requests to that IPv6 address as well. This completes the loop and allows native IPv6 and translated IPv4 address to reach the HAProxy container and pass on the original client-IP to the backend web services. This is necessary for tools such as log analyzers like AWStats.

HAProxy refreshing and Job containers

Previously, I had extended the official HAProxy container to run a script which generated the HAProxy configuration file. With this setup, I’d have to rebuild the HAProxy container for any configuration changes. The point of HAProxy is to be highly available, hence its name. So instead I created the concept of job containers and moved the HAProxy configuration to a job. The directory containing the configuration file is shared between the hasetup job container and the haproxy app container. The docker socket is also shared with the job container so hasetup can send a kill/reload signal to HAProxy to force it to reload the updated configuration file.

jobs: hasetup: server: web1 build_dir: HAProxySetup env: haproxy_container: $haproxy certbot_container: $certbot awstats_container: $awstats domains: all volumes: - letsencrypt:/etc/letsencrypt:rw - haproxycfg:/etc/haproxy:rw - /var/run/docker.sock:/var/run/docker.sock

Job containers can be used for several other things, such as regenerating static web sites or configuring database containers like so:

jobs: rearviewmirror: server: web1 git: git@gitlab.com:djsumdog/rearviewmirror_web.git volumes: - rvm-web:/www/build:rw dbsetup: server: web1 build_dir: DBSetup env: database_json: _dbmap mysql_host: $mysql postgres_host: $postgres

Job containers can be run using the run subcommand on a docker server like so:

./bee2 -c conf/settings.yml -d web1:run:hasetup

The container is not deleted once the task has completed, and can be run again with docker start . However running the job using ./bee2 will cause the container to be deleted and recreated before it is run. So long as a job container has been run once and exists in a stopped state on the Docker host, it can be scheduled to run at regular intervals using the JobScheduler container. The + symbol is used to reference a job container and scheduling is handled using cron syntax.

scheduler: server: melissa build_dir: JobScheduler env: run_logrotate: +logrotate when_logrotate: 1 2 */2 * * run_awstats: +awstats-generate when_awstats: 10 */12 * * * run_mastodoncleaner: +mastodon-remote-media-cleanup when_mastodoncleaner: 5 3 * * * volumes: - /var/run/docker.sock:/var/run/docker.sock

Closing Remarks

Dealing with IPv6 has certainly been one of the more challenging aspects of working with Docker, Bee2 and containers. It wasn’t until I finished the implementation I described that I discovered an IPv6 NAT daemon for Docker that would have given me the same flexibility with IPv6 that I had with IPv4. I am glad I implemented clean configuration and restarts for HAProxy as well. The next installment in this series will cover automated database setup and configuration for some basic Docker applications.