How Debian managed the systemd transition

This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

Debian's decision to move to systemd as the default init system was a famously contentious (and rather public) debate. Once all the chaos regarding the decision itself had died down, however, it was left to project members to implement the change. At DebConf 2015 in Heidelberg, Martin Pitt and Michael Biebl gave a down-to-earth talk about how that implementation work had gone and what was still ahead.

Pitt and Biebl are the current maintainers of the systemd package in Debian, with Pitt also maintaining the corresponding Ubuntu package. The pair began with a brief recap of the init-replacement story, albeit one that steered mercifully clear of the quarrels and stuck to the technical side. Initial discussions for replacing the System V init system began as far back as 2007, but pressure grew in recent years, included considerable demand from system administrators and upstream projects (typically wanting specific features like support for logind or journald). Once the Technical Committee had made its decision to adopt systemd as the default, Pitt said, "the real work" began.

The jessie release

Only a few months remained between the decision (in February of 2014) and the first freeze for "jessie" that November. Nevertheless, the migration was completed and jessie shipped with systemd 215. In the end, Biebl said, getting systemd into shape turned out not to be as big of a deal as had been feared—the systemd package in "wheezy" was well-tested. On the other hand, the requirement that administrators would be able swap in one of the other init-system packages added quite a bit of complexity. First, the old init system had long been marked as "essential," which meant that removing it required the user to fight quite a few warnings and protests from the package manager and other software. And, just as importantly, the idea of easily swapping init systems in and out was a new one for Debian.

Eventually, the systemd team split the existing sysvinit into several packages, which made it easier to cope with dependent packages that assumed some part of sysvinit would be available. The team also created a new init meta-package, which allowed them to ensure that users would not accidentally remove all of the init-system packages from their system.

Supporting init-system swapping is not merely a package-management problem, however: users are likely to expect some system state to be preserved between any two init systems. For example, if a service is started at boot time by one init system, the user likely expects that service to be started in a comparable fashion under the other init system, too. To ensure this preservation of state, the team re-implemented a subset of systemd's systemctl interfaces in a new package called init-system-helpers.

Along the way, team members also created a package called dh-systemd, which lets Debian package maintainers ensure that their package will be properly configured regardless of which init system the user employs. Finally, they created the systemd-shim package, which supports certain downstream projects like GNOME that have hard-coded dependencies on individual systemd components.

Although the team worked to develop these interface and compatibility packages, it took a different approach when considering what to do with sysvinit's collection of configuration files—such as those in /etc/default or /etc/rcS.d . "The major reason was that we would have to carry a patch for that for all eternity," Biebl said. So they performed a one-time migration of several such settings files.

The integration work consisted of a lot of small changes, Biebl said, "but I think we succeeded. I think that people don't realize that they are using systemd." He noted that people have actually approached him asking how they can start using systemd and been surprised when he told them that they already are.

Along those same lines, he noted that users who upgrade an older Debian system configured with sysvinit to the jessie release are automatically migrated to systemd, and few have noticed. The goal in undertaking this "scary" change, he said, was to have jessie systems be the same regardless of whether they are new installations or updated ones. For upgraded systems, though, there is a fallback sysvinit init binary installed, complete with a bootloader option to boot the system with it if the user encounters a problem with systemd.

There are bugs in every software package, of course. Biebl said that the team braced for a flood of bug reports when jessie was released, but that flood never came. Most of the bugs that have been reported have stemmed from the fact that systemd is stricter in situations where sysvinit tries to hide errors. For example, Pitt said, Debian had a longstanding bug in its ecryptfs-setup-swap package; systems ended up getting configured with unencrypted swap or no swap at all, and thanks to sysvinit, no one noticed for several years. But systemd complained, and now the bug has been fixed.

Looking forward to stretch

Pitt then turned the discussion to the changes that are in store for the next Debian release, "stretch." The first change is to udev, which will begin assigning predictable, stable names for network interfaces (in place of names using the ambiguous "eth0" form). This change has already landed, he said, and should fix problems that many users have encountered in the past. But there are some wrinkles involved: interface names cannot be migrated to the new format automatically when upgrading to stretch, because doing so could break (for example) firewall rules written with the old names in mind.

Another upcoming change is support for networkd, a lean interface for bringing up and taking down network interfaces. Pitt noted that there have been a lot of questions about whether networkd is meant to replace NetworkManager. It is not, he explained; rather, it is akin to ifupdown in its scope. But Debian users will benefit from it because it better handles virtual interfaces and hot-plugging.

Biebl explained another change that is still in the works: the ongoing process of removing sysvinit's rcS.d scripts. There is a wiki page detailing the current status and the roadmap of the process. Because these scripts are executed early in the boot process, they can be difficult to remove, and the removals can trigger dependency cycles. He noted that Felipe Sateler has started working on this task, and in doing so has located quite a few packages that need attention from their maintainers.

Other systemd-related changes that may or may not land in time for stretch include the addition of kdbus support. The Debian systemd team regards kdbus as a beneficial addition, since it would be available at the earliest stages of the boot process, but whether Debian includes it will depend on its inclusion in the kernel. There is, of course, an out-of-tree kdbus module, and Debian users interested in testing it out can do so using the systemd tools packaged for Debian unstable.

There are plenty of opportunities for new volunteers to help out, Pitt and Biebl said. But, just as importantly, Debian developers and package maintainers could start taking advantage of systemd's "shiny new features," including timers, socket activation, or security confinement. Where things go from here, in other words, depends at least as much on how Debian as a whole chooses to use its init system as it does on the team that maintains the init-system packages.

[The author would like to thank the Debian project for travel assistance to attend DebConf 2015.]

