Unexpected fallout from /usr merge in Debian

LWN.net needs you! Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing

Back in 2011, Harald Hoyer and Kay Sievers came up with a proposal for Fedora to merge much of the operating system into /usr ; former top-level directories, /bin , /lib , and /sbin , would then become symbolic links pointing into the corresponding subdirectories of /usr . Left out of the merge would be things like configuration files in /etc , data in /var , and user home directories. This change was aimed at features like atomic upgrades and easy snapshots. The switch to a merged /usr was successful for Fedora 17; many other distributions (Arch, OpenSUSE, Mageia, just to name a few) have followed suit. More recently, Debian has been working toward a merged /usr , but it ran into some surprising problems that are unique to the distribution.

Debian and its derivatives are definitely late to the /usr merge party. Systems running Debian testing that were initially installed before June 2018 still have /bin , /sbin , and /lib as normal directories, not as symbolic links. The same applies to Ubuntu 18.10. But both Debian and Ubuntu want to make the switch to a merged /usr . Debian tried, but it hit something completely unexpected.

The Debian /usr merge history started in 2016, when Marco d'Itri got the usrmerge package into Debian unstable. This package contains a Perl script that converts an existing system into the state with a merged /usr . Also, a change was made to the debootstrap program (which installs a Debian system into a chroot), so that it could create the needed symbolic links by itself before installing any packages. The end result is the same in both cases.

The plan was to default to a merged /usr when it is ready (and the initial testing revealed only three broken packages), but it was hoped that both merged and non-merged setups would be supported. In other words, the expectation was that there would be no flag day when everyone must switch.

The decision on whether /usr merge would be done by default has changed multiple times during the debootstrap development timeline. The initial support was coded in September 2016, and released in debootstrap version 1.0.83, in a disabled-by-default state. In October 2016 there was an attempt to enable it by default, but this was reverted in November, because the dpkg-shlibdeps program (which is used during package builds to automatically generate dependencies on packages that provide the needed shared libraries) broke. Therefore, Debian 9.0 (with the code name "Stretch") was released in June 2017 without this feature.

A second attempt to re-enable /usr merge by default happened in June 2018, with debootstrap version 1.0.102. Since June 25, 2018, new default installations of Debian testing from the "daily" builds of the install CD have a merged /usr , and that has not been reverted so far.

Happy ending? Not so fast ...

Problems with R

On November 10, a new version of debootstrap was uploaded to the "backports" repository for the Debian stable distribution. This repository is not enabled on end-user systems by default, but it is enabled on the build daemon (buildd) machines for the purpose of getting a few updated packages including debootstrap . Therefore, shortly thereafter, Debian build daemons started using a merged /usr for their chroots. One of them attempted to automatically rebuild the r-base package and produced an R binary that did not work on non-merged systems. As the bug report says:

/usr/bin/R: line 193: /usr/bin/sed: No such file or directory

Traditionally, sed was always placed in /bin . The Debian package sed also has /bin/sed , not /usr/bin/sed . In the bug report, the problem is treated like a one-off issue, to be solved by a rebuild. However, on the debian-devel mailing list, Ian Jackson quickly pointed out that the problem is, in fact, due to /usr merge on the build daemons. He suggested that the change should be reverted. Dirk Eddelbuettel seconded that suggestion, and noted that he expects "much more breakage to follow". Indeed, similar problems were triggered in sympow, pari, and monitoring-plugins. Other bugs of this nature can be found by searching the Debian bug tracking system for a special tag (but this search also finds other kinds of issues).

Jackson provided a good explanation of the mechanics of what has happened, quoted below.

R's autoconfery is autodetecting the location of (say) sed at build time by searching the PATH. R then bakes the discovered path into the built binaries. With usrmerge, /bin is a symlink to /usr/bin [...]. Consequently the R autoconfery always detects (say) sed in the first place out of /usr/bin and /bin it looks, and bakes /usr/bin/sed into its binaries. On a system without usrmerge, /usr/bin/sed does not exist, because sed is in /bin .

Matthias Klumpp suggested that a sensible build system would automatically detect the correct paths even on a /usr -merged system; he also wondered why the issue with incorrect paths has not been encountered by other distributions. The crucial difference was found to be that all other distributions had a flag day, before which merged /usr was unsupported, and after which it was mandatory. That is, on other distributions there was no requirement that a package built on a /usr -merged system should work correctly on a system without a merged /usr .

D'Itri said that there is only a handful of packages affected and provided an example of what a fix would look like. Namely, the proper locations of programs should be provided to the configure script or its equivalent, or the software has to be modified so that it always looks for programs using $PATH at runtime. In fact, Debian maintainer scripts (that is, scripts written by Debian maintainers that run before or after the package installation, upgrade, or removal) are already required, by Debian policy, to use $PATH instead of hard-coding the paths to the tools they use. There is, however, no such requirement in the policy for programs other than maintainer scripts.

The argument that there is only a handful of affected packages was not compelling enough, because the number of packages affected is, in fact, unknown without someone trying to rebuild them all. So Simon McVittie added a new check to the reproducible builds infrastructure, specifically for differences caused by /usr merge. This check caught problems in perl, quilt, and systemd.

On November 20, the change was reverted — build daemon chroots were recreated without a merged /usr .

Plan B?

Due to the perceived damage to the distribution caused by /usr merge, Adam Borowski started a new discussion thread on the debian-devel list, with the subject "usrmerge — plan B?". He proposed to scrap the usrmerge package, and, instead, move binaries and libraries to /usr one by one. "If it takes 10 years, so what?" McVittie pointed out that moving binaries on a package-by-package basis would not solve anything. Indeed, any move of a binary would break all packages that hard-coded its expected location, which is exactly the same problem that was triggered by /usr merge on the build daemons.

The arguments usually presented by /usr -merge proponents have also been questioned, for example on the basis that atomic upgrade is worthless if it leaves the system with mismatching /etc or /var . McVittie wrote a long email in response to the complaint about a lack of detailed justification of the change. The criticism regarding /var and /etc mismatch was answered by showing how existing systems with immutable /usr (like OSTree and systemd-based stateless systems) are dealing with it.

McVittie explained the /usr merge failure mode again and mentioned the one-way compatibility property: packages built on a system without a merged /usr would work fine on a merged- /usr system, but not the other way around. Therefore, he concluded that the transition simply happened, by accident, in the wrong order: the build daemons should have transitioned last.

The rest of the email presents the benefits of /usr merge. In particular, "it's a significant simplification for reliable special-purpose systems" such as consumer appliances. It brings benefits for end-user systems, too, because of increased portability of scripts from other systems. This is important because some upstream authors work on merged- /usr systems and don't know (and don't need to know) the "proper" location of binaries. In addition, it was noted that the change would simplify the security-relevant code of sandboxing systems like bubblewrap .

But Borowski's objection was not to /usr merge itself, but of the need to support systems both with and without a merged /usr . In other words, according to him, in order to be supportable, /usr merge must either be mandatory or not happen at all.

The discussion also touched upon the topic of the transition freeze that will happen in January 2019 and limits the time available for planning and testing any changes. So far, there is no consensus, but the most popular opinion seems to be that a mandatory /usr merge has to be postponed until after the release. As Russ Allbery put it: "That's going to be a disappointing delay for a lot of people, I'm sure, but it's still better for them than never doing this at all."

Ubuntu also plans to go through /usr merge. Dimitri John Ledkov described the situation this way: in the upcoming Ubuntu 19.04 release, new installations will use merged /usr , but existing ones will not have their /usr merged on upgrade. Of course, the official Ubuntu build daemons will not have merged /usr .

The status quo in Debian is that debootstrap (and, thus, the Debian installer) installs a system with /usr merged, and the build daemons explicitly pass a command-line switch to avoid this. This is still dangerous, because initial uploads of new packages in Debian (but not in Ubuntu) require a binary package built by the developer on his system. As long as a single uploader has a system with merged /usr , there is a potential for the bug to resurface in his Debian package. On the other hand, as d'Itri noted, there are already too many possibilities for a Debian maintainer to misbuild a package by not using a chroot, so there is nothing new in this danger. Anyway, support for marking packages as "tainted" by the build environment has been implemented in dpkg in response to this concern, and dpkg taints the package if it detects the symbolic links related to /usr merge.

The discussion is still in progress, though; no consensus has been reached. A bug was filed against debootstrap by Jackson to revert the change to merge by default for the next release of Debian. Due to the disagreement of the debootstrap maintainer to the proposed change, Jackson reassigned the bug to the Debian Technical Committee, which is the ultimate authority for resolving otherwise unresolvable technical disputes within Debian. There is also a request from the Debian backports FTP master that the default should be the same in Debian stable backports and in Debian testing. Emilio Pozuelo Monfort, a member of the release team, also spoke in favor of reverting to non-merged /usr in new installations.

It is impossible to predict now how the Technical Committee will rule. In the worst case for /usr -merge proponents, proper introduction of a merged /usr into Debian may be delayed by a few more years. But, if it votes for keeping the status quo, new end-user systems in the next stable release of Debian will have merged /usr , old but upgraded ones won't, and the build daemons will reliably build packages suitable for both cases, just like what's planned for Ubuntu 19.04. No flag day is needed in this scenario, so it would follow the best Debian traditions of not forcing transitions onto users.