Intro

This 3rd article is going to cover post-installation process, where we’ll install golang + docker +fleet + etcd. As most of these tools is made for 64bit systems we will have to modify them a bit. The set of tools is used in coreos for managing containers in a cluster, so that is a perfect tool for us!

Post-install: ResizeFS & move configs

In previous article: #Building ARM cluster Part 2: Create and write system image with goback! we have managed to write system image to sd/emmc memory via network. Now it is time to make system to serve some services.

Installed image is 2GB size, probably your memory stick is much larger than that, so you’ll need to resize system partition. You may do it by this simple script:

#!/bin/sh parted /dev/mmcblk0 <

After reboot execute and run e2fsck: resize2fs /dev/mmcblk0p2

Where 13,184,18 is where partition starts, 1005,63,31 partition end. Check your configuration by: parted /dev/mmcblk0 unit chs print

Installing configuration (/etc) files

After re-image each file you have modified disappears. In order to recover them after image is written I’m using the simplest possible approach = store /etc/* in repository.

Here are some files from my private repository:

fleetd.service

[Unit] Description=fleetd After=etcd.service [Service] ExecStart = / usr / bin / fleetd Restart=always RestartSec=10s [Install] WantedBy=multi-user.target

etcd.service

[Unit] Description=etcd Wants=network.target network-online.target After=network.target network-online.target cloudinit.service [Service] ExecStart = / usr / bin / DCE Restart=always RestartSec=10s [Install] WantedBy=multi-user.target

cloudinit.service

[Unit] Description=cloudinit [Service] Type = oneshot ExecStart=/usr/bin/coreos-cloudinit --from-file /root/cluster/cloud-init/odroid.conf [Install] WantedBy=multi-user.target

odroid.conf

#cloud-config Core: etcd: name: $host data_dir: /var/lib/etcd/ discovery: https://discovery.etcd.io/da3221e6c34261892a8c89f1845631b6 addr: $private_ipv4:4001 peer-addr: $private_ipv4:7001 fleet: public-ip: $private_ipv4 # used for fleetctl ssh command

fleet.conf

verbosity = 2 etcd_servers=["http://$private_ipv4:4001"] etcd_request_timeout=30.0 public_ip="$private_ipv4"

Custom postinstall commands:

echo "[Step] Drop iptables" systemctl disable firewalld.service systemctl stop firewalld.service echo "[Step] Run yum" yum check-update yum install -y python-pip vim parted-devel mercurial git-core net-tools tar gcc wget nfs-utils pip install reparted

Install go, docker, fleet, etcd, cloudinit

At first install go.

echo "[Step] Install go" cd /usr/src/ # Install go git clone https://go.googlesource.com/go cd go git checkout go1.4.1 cd src ./make.bash echo "export PATH=$PATH:/usr/src/go/bin/" >> ~/.bashrc echo "export GOPATH = / usr / src / spouse" >> ~ / .bashrc export PATH=$PATH:/usr/src/go/bin/ export GOPATH = / usr / src / spouse mkdir / usr / src / spouse

Then install docker (on fedora it is a bit tricky). If you’re using ubuntu then download docker.io from repo. Otherwise:

echo "[Step] Install Docker" # Install docker mkdir / usr / src / docker-install cd / usr / src / docker-install yum install -y rpm-build glibc-static rpmbuild --rebuild http://copr-be.cloud.fedoraproject.org/results/gipawu/kernel-aufs/fedora-21-x86_64/aufs-util-3.9-1.fc20/aufs-util-3.9-1.fc21.src.rpm rpm -i /usr/src/docker-install/rpmbuild/RPMS/armv7hl/aufs-util-3.9-1.fc21.armv7hl.rpm yum install -y bridge-utils device-mapper device-mapper-libs libsqlite3x docker-registry docker-storage-setup mkdir / usr / src / docker cd / usr / src / docker wget https://kojipkgs.fedoraproject.org//packages/docker-io/1.5.0/18.git92e632c.fc23/src/docker-io-1.5.0-18.git92e632c.fc23.src.rpm rpm2cpio docker-io-1.5.0-18.git92e632c.fc23.src.rpm | cpio -idmv tar -xzf docker-92e632c.tar.gz curl -L https://github.com/umiddelb/armhf/raw/master/bin/docker-1.5.0> docker ### INSTALL install -p -m 755 docker / usr / bin / docker # install bash completion install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/completion/bash/docker /usr/share/bash-completion/completions # install container logrotate cron script install -p -m 755 docker-logrotate.sh /etc/cron.daily/docker-logrotate # install vim syntax highlighting install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/doc/dockerfile.txt /usr/share/vim/vimfiles/doc install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/ftdetect/dockerfile.vim /usr/share/vim/vimfiles/ftdetect install -p -m 644 docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/syntax/vim/syntax/dockerfile.vim /usr/share/vim/vimfiles/syntax # install udev rules install -p docker-92e632c84e7b1abc1a2c5cb3a22e0725951a82af/contrib/udev/80-docker.rules /etc/udev/rules.d # Install systemd / init scripts install -p -m 644 docker.service /usr/lib/systemd/system # for additional args install -p -m 644 docker.sysconfig / etc / sysconfig / docker install -p -m 644-docker network.sysconfig / etc / sysconfig / network-docker install -p -m 644-docker storage.sysconfig / etc / sysconfig / docker-storage # install docker config directory sudo install -dp / etc / docker getent passwd dockerroot > /dev/null || sudo /usr/sbin/useradd -r -d /var/lib/docker -s /sbin/nologin -c "Docker User" dockerroot systemctl enable docker.service systemctl enable docker.socket

Install fleet:

echo "[Step] Install fleet" cd /usr/src/ go get golang.org/x/tools/cmd/cover git clone https://github.com/coreos/fleet.git cd fleet ./build ./test ln -s /usr/src/fleet/bin/* /usr/bin/ source ~ / .bashrc

Install etcd with patches:

# Install etcd echo "[Step] Install etcd" ETCDI_VERSION=2.0.4 curl -sSL -k https://github.com/coreos/etcd/archive/v$ETCDI_VERSION.tar.gz | tar --touch -v -C /usr/src -xz cd /usr/src/etcd-$ETCDI_VERSION curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/raft.go.patch > raft.go.patch curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/server.go.patch > server.go.patch curl https://raw.githubusercontent.com/mkaczanowski/docker-archlinux-arm/master/archlinux-etcd/patches/watcher_hub.go.patch > watcher_hub.go.patch patch etcdserver/raft.go < raft.go.patch patch etcdserver/server.go < server.go.patch patch store/watcher_hub.go < watcher_hub.go.patch ./build ln -s /usr/src/etcd-$ETCDI_VERSION/bin/* /usr/bin/ mkdir / var / lib / etcd systemctl enable cloudinit systemctl enable etcd systemctl enable fleetd

Install cloudinit and replace variables $2 = hostname, $3 = host ipaddr

echo "[Step] Cloud config" cd /usr/src/ git clone https://github.com/coreos/coreos-cloudinit.git cd coreos-cloudinit ./build ln -s /usr/src/coreos-cloudinit/bin/coreos-cloudinit /usr/bin/coreos-cloudinit sed -i "s/\$private_ipv4/$3/g" /usr/src/cluster/cloud-init/odroid.conf sed -i "s/\$private_ipv4/$3/g" /etc/fleet/fleet.conf sed -i "s/\$host/$2/g" /usr/src/cluster/cloud-init/odroid.conf

Build archlinux images

Building docker images is pretty simple. As ArchlinuxARM is very lightweight system I'm using it as base image for further services in docker. Systemd is enabled in all images.

How to build nginx container?

FROM mkaczanowski/archlinux-systemd MAINTAINER Mateusz Kaczanowski, [email protected] RUN pacman -Syu --noconfirm nginx; systemctl enable nginx RUN pacman -Scc --noconfirm; RUN mkdir -p /etc/nginx/conf.d RUN mkdir -p /etc/nginx/sites-enabled ADD ./config/nginx.service /etc/systemd/system/multi-user.target.wants/nginx.service ADD ./config/nginx.conf /etc/nginx/nginx.conf ADD ./config/00-default.conf /etc/nginx/sites-enabled/00-default.conf ONBUILD RUN rm /etc/nginx/sites-enabled/00-default.conf ONBUILD ADD ./sites/*.conf /etc/nginx/sites-enabled/ VOLUME ["/srv/http/"] EXPOSE 80 EXPOSE 443 CMD ["/usr/lib/systemd/systemd"]

Run container:

CONTAINER=$(sudo docker run --privileged -d -v /srv/http:/srv/http -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 -p 443:443 mkaczanowski/archlinux-nginx)

Run nginx container with fleet

Eventually we are able to run nginx container on all of machines at once using fleet!

Nginx service:

[Unit] Description=nginx After=docker.service Requires=docker.service [Service] TimeoutStartSec=0 ExecStartPre=-/bin/docker kill nginx ExecStartPre=-/bin/docker rm nginx ExecStartPre=/bin/docker pull mkaczanowski/archlinux-nginx ExecStart=/bin/docker run --name nginx --privileged -v /srv/http:/srv/http -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 -p 443:443 mkaczanowski/archlinux-nginx ExecStop=/bin/docker stop nginx

Now using fleet we may execute container:

fleetctl --endpoint=http://192.168.4.100:4001 submit nginx fleetctl --endpoint=http://192.168.4.100:4001 start nginx

It works! Now you have fully managed cluster!

Demo