Archie: Easy cross-compilation for busy developers

Architecture-agnostic builds on Docker with Azure Pipelines’ free tier

As frequent readers will know, I’ve long held an interest in supporting low-end devices (like the ever-popular Raspberry Pi) with more of today’s most popular development tools. In part, this is due to the prevalence of these devices in schools, and in part because it’s a lot easier to get a functional computer into someone’s hands if it can be bought for as little as $5.

Today I’d like to introduce a project I’ve been working on for some time.

Archie is a multi-architecture compilation tool with a goal of making it easier for developers to support alternative architectures for their applications.

Among the largest compiler stacks there are two broad ways to go about compiling code for a different architecture. The first, and in most cases the fastest, is to cross-compile from the host architecture (in 2019 this is most likely to be amd64) to the target architecture.

For simple programs, this is relatively straightforward. Taking GNU C as an example, we effectively just need to point the correct target compiler at our code, let it run, and a binary will pop out the other side for our target device. Easy peasy… sort of.

For scenarios more complicated than simple utilities, cross-compilation quickly becomes a nightmare of crossed-wires, edge cases and unintended consequences (the most obvious of which are explored at the end of this article for those interested).

Archie exists to make this a bit easier.

Target-agnosticism with Archie

The core idea behind Archie is that code which is platform-agnostic should be no more difficult to compile for foreign architectures than it is for it’s native architecture.

NOTE: While Archie may be useful to projects that have architecture-specific code, there’s no black magic here — it won’t make code that uses x86 specific calls “just work” on ARM, for example. Archie’s intention is to help with projects that don’t have architecture-specific logic in their code.

Archie approaches compilation in three different ways, referred to internally as strategies. The intention is that each of these strategies are interchangeable without changes to the application code or build scripts.

cross strategy

The cross strategy configures the environment to support multiple architectures and installs dependencies for the target architecture directly into the build host. The system is configured to use a host-native cross-compiler to run the build script. This is the fastest compilation method, but can fail if the dependencies require target-native code to be executed during setup.

emulate strategy

The emulate strategy uses the excellent QEMU tool to set up an isolated environment for building our application code which will transparently emulate target architecture code during the setup process, and then use a target-native compiler to build the code. This strategy uses root filesystems produced by prebootstrap, another project I’m maintaining over at GitHub. This is the slowest compilation method, as the entire compilation process is being emulated, but is also expected to be the most reliable for more complex projects. Be aware that for large codebases, this can take a long time to build.

hybrid strategy

The hybrid strategy is where Archie really shines, and is expected to be used by most projects. It is intended to be a best-of-both-worlds approach to target-agnostic builds (albeit with a few caveats). Dependencies are installed in an emulated environment of the target architecture, allowing target-native code in dependencies to execute during installation, but afterwards the application code is then compiled with a host-native compiler mapped to that environment. This allows dependency scripts to execute in order to prepare the environment, but also offers the speed benefits of executing a native compiler on the host.

Managing dependencies, and why Archie exists

When our code grows beyond simple utilities, otherwise simple cross-compilation breaks down rather quickly for a few different reasons. We can’t simply link (for example) arm binaries to amd64 libraries (or rather we could, but the output would make no sense — and they wouldn’t run anywhere without some level of emulation).

If we want to target a variant of ARM (or POWER, or even RISC-V for that matter), we need to be able to serve dependencies for those architectures to link our binaries to — and ensure that the compiler is able to link to those resources correctly.

Luckily, most distributions and their package managers have support for multiple architectures, allowing us to obtain the chain of dependencies we need to build our software for a given target.

Once we have these dependencies in place, we need to ensure that we correctly instruct the compiler to use the dependencies and headers from our target architecture rather than the host. Undoubtedly more complicated, but not yet a nightmare scenario.

Exec format error

So we’ve been able to set up our compiler. We’ve got our list of dependencies for the target, and we’ve tried to install them ahead of compilation. Now, however, our logs are showing this new type of error all over the place: “Exec format error”, and half of our dependencies seem to be either missing or broken even though we’ve just installed them. What gives?

As it turns out, we’ve tried to install dependencies for our target architecture (e.g. armhf) — but those dependencies also have dependencies (which in turn have their own dependencies) that expect the host architecture to match the package architecture (i.e. the package installation script is trying to execute armhf code on our amd64 host, and is failing).

QEMU to the rescue!

The way around this problem is through use of the excellent QEMU tool, which allows for executing foreign instruction sets on your local architecture. The most straightforward way to get up-and-running with it, and the method you’ll see in a great many forums all over the web, is to install the qemu-arm-static and binfmt_misc packages. This allows our host system to execute target architecture code as if it were compiled for the local architecture transparently, which is fantastic for usability.

It’s not without it’s own gremlins, though.

Figure A — Example of transparent ARM emulation on x86

A couple of the simpler problems with using a transparent emulation layer are shown by Figure A.

The mix of different dependencies means that it’s not directly clear which architecture the linker is using to satisfying the dependencies for the application (or whether they’ll be available on the actual end hardware).

The other issue this presents is that it’s not immediately clear if we’re actually compiling for the target architecture at all (the target architecture in this scenario would be ARM, but the dotted line shows the potential problem of compiling natively for x86–64 without realising.)

Now, both of these problems could be resolved by analysing the final binaries to ensure that they’re compiled for the correct architecture, and this is fairly easy to do. The real problem is that the nascent market for alternate-architecture systems is so small that it’s usually deemed to not be worth the extra effort of getting builds up-and-running and supporting it.

Archie tries to help with this by taking responsibility for mapping compiler search paths based on the specified strategy, meaning that once the build script has been edited to use a few environment variables provided by Archie, it should be completely agnostic of the underlying architecture.

What works currently?

Archie currently supports mapping environments and compiler globals for GCC and some of the downstream stacks that really on it. This currently includes NodeJS, so Electron apps with native dependencies are in effect supported “out-of-the-box”.

I’m working to add more — in principle it’s really straightforward to add the necessary flags for Ruby, Python and Rust (from what I’ve looked at) — but as I don’t have a lot of personal experience in these realms I’d love to get a pull request from the community!

Why Azure Pipelines for cloud builds?

Archie is a docker image available on Docker Hub like most others. It can be pulled from there, and by passing arguments into docker run, be used in projects on private build servers with relative ease. Not so in the cloud, as things aren’t nearly that simple — the reason being as much to do with features of the Linux kernel as anything else.

Archie relies on binfmt_misc for transparent emulation, a feature that needs explicit support enabled in the kernel. As binfmt_misc also requires elevated privileges to enable, the host needs to allow the permissions necessary to set that up. In the Docker world, this means being able to run the container with apparmor:unconfined and the CAP_SYS_ADMIN flag set. Azure is one of the few providers that allow for this.

In addition — the free tier on Azure Pipelines is generous enough that most projects should be able to compile for multiple targets on that infrastructure quickly and with relative ease (10 parallel jobs with unlimited minutes). If anyone wants to adapt the scripts for running on another providers infrastructure it shouldn’t be too difficult, and I’m open to pull requests.

I plan to write a follow up post explaining exactly how I’m using Archie to build Visual Studio Code for armhf and arm64, but for now more information about Archie and how it manages these jobs is available in the README on the website, which can be found here. Please let me know if you end up using it on one of your own projects (and please star the repositories on GitHub if you want to help me out!)