As part of some new Rabbit features, we’ve been investigating the use of boot files to start multiple applications without requiring us to build our own application loader. After some initial experimentation, we built a fairly simple utility that would generate a .rel file, then request the Erlang toolchain to compile this and generate .script and .boot files. In doing this, we learnt a great deal about the internals of OTP startup sequences.

As a result of the experimentation, we found a number of interesting properties. Firstly, we discovered that if you go down the path of using boot files you should use Supervisors on all of your top-level applications. Whilst this is generally considered good practice in all Erlang applications, it is even more important when these applications are listed in the boot file, since if an application listed in the boot file crashes at any point the entire Erlang VM will be halted! This behaviour certainly piqued a number of people’s interests – after some consideration we agreed that it is a sensible enough behaviour, since if you instructed the VM to start with a given set of applications, then any of them crashing out probably constitutes a fairly major failure.

Through some investigation and reading of the Erlang documentation, we learnt this wasn’t actually a boot file specific behaviour. Instead, it was because applications listed in the boot file are actually started as permanent (instead of the default of temporary). The effect of marking an application as permanent is that if it crashes, the Erlang kernel will shutdown as a result. Those wishing to try this at home can simulate it with the use of

application:start(app_name, permanent)

and some rather adventurous use of the Erlang toolbar.

After the initial implementation, a number of other issues were discovered. Currently, Rabbit performs a number of “VM-configuration” tasks during the rabbit:start method. Unfortunately, using the boot file bypassed (or in the very least de-prioritised)Â these operations. So investigation began into finding a way to configure the VM running Rabbit before applications were started.

Somewhat expectedly, the -eval flag is not evaluated until after the boot sequence is completed. Applications run via “-s” flags are similarly delayed until after the boot sequence has finished. So we began to dig a little deeper.

Anyone who has experimented with .boot files will have noticed the boot file isn’t plain text. However, the boot file is actually just a binary form of the associated .script file. Inside the .script file, following the various path-setup commands, are a series of MFA formatted commands, looking something like:

{apply,{application,start_boot,[kernel,permanent]}}, {apply,{application,start_boot,[stdlib,permanent]}}, {apply,{application,start_boot,[mnesia,permanent]}}, {apply,{application,start_boot,[sasl,permanent]}}, {apply,{application,start_boot,[os_mon,permanent]}}, {apply,{application,start_boot,[rabbit,permanent]}}

This opened up a whole new set of options. We tried a few different techniques to get the behaviour we were after. Currently, we’ve settled on inserting an apply line like {apply,rabbit,prepare,[]}} after the stdlib application startup. This allows us to do custom VM initialisation before the non-kernel applications are started. (We decided it would be wise not to do any application specific changes to the VM before the kernel or stdlib are started).

Until starting down this path, I certainly didn’t know there were so many possibilities for customising the Erlang boot sequence. Hopefully others might be able to find interesting things to do too!