Jun 12, 2016

Jun 16, 2016 update: explained reasons for using Docker with QEMU.

For one of my previous assignments I had to automate building ARM-based device firmware and over-the-air updates. While the recovery partition of a device was relatively small and simple and could be built using Buildroot, the root partition was based on Debian 8 “Jessie”.

Let’s make a firmware for a hypothetical ARM-based HAL 9000.

The goal is to describe the process overall and not to delve into device specifics. You still need to figure out the boot sequence, configure and build the kernel, set up partitioning and write custom setup scripts that work for your device.

TL;DR: scripts for the lazy are in the end.

Prerequisites

I assume you’re using some flavor of Linux, can figure out when to use elevated privileges and you have these tools at your disposal:

chroot

debootstrap

Docker

Statically-compiled QEMU User Emulation, aka qemu-user-static.

The reason why I used Docker and QEMU User Emulation instead of just scripting in chroot was that some software needed to be cross compiled and it had complex dependencies. So compiling it natively, thanks to QEMU dynamic binary translation, in the same environment (i.e. OS, kernel and packages) where it’s going to run was the easiest.

Making a minimal Debian ARM image using debootstrap and chroot

According to debootstraps’s website it is:

a tool which will install a Debian base system into a subdirectory of another, already installed system.

And that’s just what we’re going to do:

deboostrap Stage 1

1

2

3

4

rootfs_dir=rootfs



distro=jessie

debootstrap --arch=armhf --foreign $distro $rootfs_dir



deboostrap Stage 2

Enable QEMU user emulation by copying qemu-arm-static to run ARM binaries:

1

2

qas=$( which qemu-arm-static)

cp $qas $rootfs_dir $qas



Now start stage 2 of deboostrap:

1

chroot $rootfs_dir /debootstrap/debootstrap --second-stage



This will take a while but in the end you’ll have a directory with a minimal Debian system, sans your device’s kernel, modules and software. At the time of this writing, I ended up with 266MB rootfs.

Importing minimal Debian rootfs into Docker

Making rootfs archive:

1

2

3

4

5

pushd . > /dev/null

cd $rootfs_dir

tar -czf ../rootfs.tar.gz .

popd > /dev/null

rm -rf $rootfs_dir



Importing rootfs into Docker:

1

2

tag= 'georgesapkin/hal9000:base'

docker import rootfs.tar.gz $tag



Now you have a Docker Debian image that serves as a base for future firmwares. I set up a job in Jenkins to rebuild it once a week, so device firmwares are always based on the latest software.

Installing device-specific kernel, modules and software inside Docker

I use the base image from previous steps:

1

FROM georgesapkin/hal9000:base



Rootfs overlay

All static OS configuration files and scripts are in rootfs_overlay directory that is applied to the base image before any other scripts are run.

1

COPY rootfs_overlay /



A good candidate for the overlay is fstab file from /etc/fstab that list device-specific partitions. Another one is a dpkg filter to be placed in /etc/dpkg/dpkg.cfg.d/01_filter to prevent some unnecessary files from being installed. There’s a good article on Ubuntu Wiki about reducing disk footprint.

Kernel and modules

This is also a good place to install the pre-built device-specific kernel and modules. Keep in mind when configuring and building a kernel for your device that some OS features (e.g. iptables, webcam support, etc.) require specific modules to be enabled. These modules can take up significant amount of space.

1

2

3

COPY kernel/boot /

COPY kernel/lib/firmware /lib/

COPY kernel/lib/modules /lib/



Installing software

I put all the software setup into a few scripts files, e.g. setup_some_feature.sh . First, since we cannot reply to any questions when installing packages we need to set:

1

export DEBIAN_FRONTEND=noninteractive



Then we run apt-get install with -y --no-install-recommends -o Dpkg::Options::='--force-confold' arguments. force-confold is needed in case we want to prevent configurations files carried over from the overlay in previous steps from being overwritten. If you are building software inside Docker, you will need to install and setup the development environment first. Here’s a likely set of packages that you might need when building software:

1

2

3

4

5

6

7

8

apt-get install -y \

--no-install-recommends \

-o Dpkg::Options::= '--force-confold' \

build-essential \

ca-certificates \

curl \

git \

python



If your device has Wi-Fi or networking capabilities that you manage from scripts, you might need one or more of the following packages:

crda

isc-dhcp-client

iw

wireless-tools

wpasupplicant

In case you have software that needs elevated privileges that is configured to do so via a /etc/sudoers.d/* file, you will need to install the sudo package.

The relevant part of the Dockerfile to run the setup scripts:

1

2

COPY setup_some_feature.sh /scripts/

RUN /scripts/setup_some_feature.sh



Cleanup

I put pre-image-authoring cleanup into author.sh . First, let’s get rid of any packages that might be left over from the build environment and some other misc packages that are not needed in the final image, remove unused dependencies and clean Apt cache:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

apt-get remove -y \

binutils \

build-essential \

cpp \

cpp-4.9 \

dpkg-dev \

g++ \

g++-4.9 \

gcc \

gcc-4.9 \

git \

git-man \

libc6-dev \

libc-dev-bin \

libgcc-4.9-dev \

linux-libc-dev \

libstdc++-4.9-dev \

make \

manpages \

nano \

wget



apt-get autoremove -y

apt-get -q clean



We can go a step further and remove Apt utilities as well, since we’re not going to install packages on a running system.

Don’t forget to disable the root login:

1

usermod -L root



Now, let’s remove some static files, caches, histories, logs and unused locales. Again, Ubuntu Wiki has a good rundown of the process.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

rm -rf \

/usr/share/applications \

/usr/share/apps \

/usr/share/doc \

/usr/share/games \

/usr/share/groff \

/usr/share/icons \

/usr/share/info \

/usr/share/linda \

/usr/share/lintian \

/usr/share/man \

/usr/share/pixmaps \

/var/cache/* \

/var/lib/apt/lists/* \

/var/ log /* \

2> /dev/null || true



find /usr/share/locale -mindepth 1 -maxdepth 1 ! -name 'en*' \

! -name 'locale.alias' | xargs rm -rf 2> /dev/null || true



rm -rf ~/.bash_ history 2> /dev/null || true



The relevant part of the Dockerfile to run the author script:

1

2

COPY author.sh /scripts/

RUN /scripts/author.sh



The final step before making a firmware image is to remove temporary scripts and QEMU:

1

2

RUN rm /usr/bin/qemu-arm-static

RUN rm -rf /scripts



Now it’s time to build the firmware image using the Dockerfile:

1

2

image_tag=georgesapkin/hal9000:$(date -I)

docker build --tag= " $image_tag " -f Dockerfile .



Since Docker stores all individual layers, you can structure your Dockerfile so that most-frequently changed scripts and commands are at the bottom. That way you can dramatically speed up your builds.

Making a firmware image from a Docker image

First, let’s make a sparse image file, format it as ext4 and mount it. According to Wikipedia a sparse file is:

…a type of computer file that attempts to use file system space more efficiently when the file itself is mostly empty. This is achieved by writing brief information (metadata) representing the empty blocks to disk instead of the actual “empty” space which makes up the block, using less disk space.

1

2

3

4

5

6

7

8

size=500M

rootfs_image=hal9000-$(date -I).img

dd if =/dev/zero of= $rootfs_image bs=1 count=0 seek= $size status=none



mkfs.ext4 -b 4096 -F $rootfs_image



mkdir -p $rootfs_dir

mount -o loop $rootfs_image $rootfs_dir



It’s only possible to export from a Docker container, so we need to make a temporary one, export and then remote it:

1

2

3

4

container_name=hal9000-$(date -I)

docker run --name $container_name $tag true

docker export $container_name | tar -xf - -C $rootfs_dir /

docker rm $container_name



Now we can unmount and pack the rootfs image:

1

2

umount $rootfs_dir

gzip < $rootfs_image > $rootfs_image .gz



The firmware for our HAL 9000 is ready for flashing. You can flash it using:

1

gunzip -c rootfs.img.gz | dd of=/dev/your_device_root_partition bs=64K



You may take it a step further and create a signed image using cpio and openssl that you can flash using swupdate . Or make an incremental over-the-air update.

To make this fully-automated you can hook it up to your CI of choice and have a scheduled job (for the base image) or source repository triggers (for the firmware).

Feel free to ask questions in comments below or drop me an email if you need some advice.

build.sh - you’ll have to read the post for the comments :p

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41



set -euo pipefail



rootfs_dir=rootfs

distro=jessie

debootstrap --arch=armhf --foreign $distro $rootfs_dir



qas=$( which qemu-arm-static)

cp $qas $rootfs_dir $qas



chroot $rootfs_dir /debootstrap/debootstrap --second-stage



pushd . > /dev/null

cd $rootfs_dir

tar -czf ../rootfs.tar.gz .

popd > /dev/null

rm -rf $rootfs_dir



tag= 'georgesapkin/hal9000:base'

docker import rootfs.tar.gz $tag



date=$(date -I)

image_tag=georgesapkin/hal9000: $date

docker build --tag= " $image_tag " -f Dockerfile .



size=500M

rootfs_image=hal9000- $date .img

dd if =/dev/zero of= $rootfs_image bs=1 count=0 seek= $size status=none



mkfs.ext4 -b 4096 -F $rootfs_image



mkdir -p $rootfs_dir

mount -o loop $rootfs_image $rootfs_dir



container_name=hal9000- $date

docker run --name $container_name $tag true

docker export $container_name | tar -xf - -C $rootfs_dir /

docker rm $container_name



umount $rootfs_dir

gzip < $rootfs_image > $rootfs_image .gz



setup_some_feature.sh - you really have to write your own.

author.sh - this is a non-exhaustive sample. You will have different things to clean up in your firmware.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51



set -euo pipefail



apt-get remove -y \

binutils \

build-essential \

cpp \

cpp-4.9 \

dpkg-dev \

g++ \

g++-4.9 \

gcc \

gcc-4.9 \

git \

git-man \

libc6-dev \

libc-dev-bin \

libgcc-4.9-dev \

linux-libc-dev \

libstdc++-4.9-dev \

make \

manpages \

nano \

wget



apt-get autoremove -y

apt-get -q clean



usermod -L root



rm -rf \

/usr/share/applications \

/usr/share/apps \

/usr/share/doc \

/usr/share/games \

/usr/share/groff \

/usr/share/icons \

/usr/share/info \

/usr/share/linda \

/usr/share/lintian \

/usr/share/man \

/usr/share/pixmaps \

/var/cache/* \

/var/lib/apt/lists/* \

/var/ log /* \

2> /dev/null || true



find /usr/share/locale -mindepth 1 -maxdepth 1 ! -name 'en*' \

! -name 'locale.alias' | xargs rm -rf 2> /dev/null || true



rm -rf ~/.bash_ history 2> /dev/null || true



Dockerfile

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

FROM georgesapkin/hal9000:base



COPY rootfs_overlay /



COPY kernel/boot /

COPY kernel/lib/firmware /lib/

COPY kernel/lib/modules /lib/



COPY setup_some_feature_1.sh /scripts/

RUN /scripts/setup_some_feature_1.sh



COPY setup_some_feature_2.sh /scripts/

RUN /scripts/setup_some_feature_2.sh



COPY author.sh /scripts/

RUN /scripts/author.sh



RUN rm /usr/bin/qemu-arm-static

RUN rm -rf /scripts



/etc/dpkg/dpkg.cfg.d/01_filter - from overlay