A while ago I compiled a wiki page that documents where are we with support of snapd on various popular distributions.

Unfortunately the table there may look a bit bleak. Apart from Ubuntu and Debian most of the other distributions don’t ship snapd today. Since a few people were asking me about this (after all I was supposed to get this to work). I think some explanations are in order.

Preface

First of all, snapd development moves fast. There’s a lot of things landing all the time, many of those non-trivial and introducing big changes. A few of those include me in the loop which significantly cuts the amount of time that I have left to work on other things. Ultimately the whole problem is a problem of having insufficient time to work on everything.

But let’s not get ahead of ourselves. Let’s rewind time to summer 2016. During the snappy sprint in Heidelberg I started working on distribution packages for Fedora, OpenSUSE, Arch, Gentoo and others. After a few quick experiments (just clone the code and build a few things) I was positive that it is doable and probably not that hard. Boy, was I wrong!

Security

At that time I took the strong security out of the picture as Fedora was (and still is) based on a different LSM (SELinux) and OpenSUSE did not ship the bleeding edge Apparmor patch-set from Ubuntu. Porting the security policy to a different LSM is a separate problem. The problem of bleeding edge Apparmor will eventually fix itself once all the patches will be in the upstream kernel and released downstream.

If you are on a distribution that doesn’t enable an LSM (Linux Security Module) then, well, you probably don’t expect the added security anyway. For distributions like that snapd has a concept of devmode where confinement system is disabled or severely limited and where there is no strong security guarantee.

With that out of the way I started looking at the actual package. Packaging snapd is both straightforward and hard. As a Go project snapd builds to a small set of executables and a few supporting systemd units. Snapd has a small brother, snap-confine, that is written in C and builds anywhere with autotools. So far so good, right? Well, no.

Fedora 25 requires any service started by systemd to be confined by SELinux. Since snapd itself is not confined (even on Ubuntu) this meant that we could build a nice package but you’d have to disable SELinux (or switch it into permissive mode) to actually use it. This is not a great experience for our users so the package never got released.

Neal Gompa worked on creating a selinux policy that would apply to snapd and let it do everything it needs to. Since writing a SELinux policy is not an easy thing to do and we had no expertise with it, the project just got stuck for weeks and months without progress.

Recently there’s been a bit of light at the end of the tunnel though because we managed to switch the policy into permissive mode for snapd alone. This means that on a system with enforcing SELinux, the policy for snapd itself would be permissive. Eventually all the missing bits can be added and it can be fully confined but at least we are not as stuck as we were before.

Distribution Policy

Each Linux distribution has its own quirks and nuances. Software skew aside, the policy is what really differs Debian from Ubuntu, Fedora, OpenSUSE or Gentoo. In one distribution doing something is frowned upon while in another it is perfectly fine or even encouraged. Navigating this maze is required to successfully deliver software to the broadest possible audience that is not running Windows or MacOS.

Vendoring

There are a few things that feel most divergent across those distributions. One such problem is the act of bundling or vendoring a dependency into a project. For Go that is most esoteric concept as Go “packages” are really just source files spliced into weird places. To build a project that depends on a .go file from another package you need to recursively package all of your dependencies. A rough count for snapd indicates that it depends on twenty-two other go projects. That implies that to package snapd one needs to additionally package 22 additional projects.

Each package, even a trivial one that literally contains one .go file, needs to go through review (by each distributor), following different process and schedule.

Normally I don’t disagree with distribution policy but at some point you need to acknowledge the busy-work. Fedora has built a special tool gofed to manage such packages. The work is almost entirely automated but you still need to hand-hold it during the process (edit descriptions, copyright data and what not). Obviously this just Fedora. OpenSUSE, while also RPM based used (or used to use) entirely different process that has nothing in common. Those 22 packages need to be made again, following different names, different packaging rules and, well, pretty much everything.

So you roll up your sleeves and start packaging, send your package for review and iterate with someone that can eventually say “yes” or “no”. If the package already exists but is out of date you need to reach out to the maintainer and ask for an update or work together on the update.

I must say that this work has some nice property to it. It almost feels pleasant, like sorting LEGOs by color or arranging shoes that kids hastily left in the hall. You get the feeling of accomplishment and regularity. At the same time it consumes a lot of time and you start to want to stab or strangle yourself when days pass and nothing happens. Package waits in Debian’s NEW queue for a blessing by ftpmaster? Waiting for a reply in bugzilla? Contacting a maintainer on holidays? During that time snapd can change and may depend on half a dozen new things and no longer need the one package you got ready…

In Ubuntu and Debian vendoring is not allowed (although in Ubuntu we switched to vendoring because it was just insanely costly to backport everything to Ubuntu 14.04 and it really makes no sense to do so). Fedora, to the best of my knowledge, allows it to a limited extent as long as they are properly described in the resulting package. I guess that’s the price of having chromium in the repository. OpenSUSE does not allow it, though this may have changed now, as golang in OpenSUSE feels very volatile. Follow that link for some FOSS drama.

The alternative to vendoring golang packages is to go through the required busywork. A quality that a successful packager must endure. In the end all those packages contain a highly blessed export of the git tree, shipped somewhere in /usr . It takes exuberant amount of time to produce and land all those packages and it requires the soul of an accountant, but it can be done.

The FHnS

On a Linux distribution we get to enjoy the File-system Hierarchy non-Standard. I mean this only half jokingly. The standard is not an industry practice if top two OS software distributors on the planet, namely Microsoft and Apple don’t care about it. In the zoo of Linux distributions the state of the matter is equally bleak.

For Snapd the first offending problem is libexecdir aka “the place for executables that are not on PATH but are needed internally”. Debian and Ubuntu differ widely from Fedora. On Fedora you have to use /usr/libexec/ , on Debian you must use /usr/lib . I actually like Fedora’s libexec directory if it wasn’t such a hassle (and if it actually mattered in the end). Snapd is written in Go and it thankfully did not inherit the monstrosity known as autoconf. There’s a small extra bit of code needed to handle the location of this directory. It’s just detected at runtime depending on the distribution you are running on. Perfect? No, there’s always one more distribution out there that the detection logic will get wrong. That’s the best we can do though. Humans seem to agree on disagreeing being fun.

The second problem was that of /snap . I asked the FESCO (Fedora Engineering Steering Committee) for an exception to use /snap for snapd. After some discussion my request was denied. Disclaimer: Fedora has full rights to deny this and I don’t dispute that.

This decision put a bit of a problem on my hands. After some considerations and discussion I decided to move /snap to /var/lib/snapd/snap . The code for that hasn’t landed in snapd (at least for testing the change was extensive) but there’s enough abstraction in the non-test path of the code for this to work with a minimal patch that can be brewed in Fedora and upstreamed once the initial package is working. The test part is harder as snapd has extensive test suite and sure enough there’s no variable (in both shell and go) that says where snaps are mounted. I think this can be done over time, it’s just tedious.

You can read the conversation in the ticket to see why I argued for single, uniform /snap in all the distributions. You can agree (or not) but it does simplify a lot.

The last problem is that of merged-usr. Fedora lead the way towards unifying /usr and / . For snapd that means some paths are different (e.g. where is removable storage mounted?). As things moved to /run and time passes this will standardize but it is just more divergence one has to be aware of. For months now snap-confine supports this but I bet we will find edge cases that can only be identified once the package is ready and people actually try this in practice.

The +s bit

Snapd relies on a small C executable, snap-confine, that is setuid root. It needs to be root to perform file system name-space manipulation necessary for the construction of the snap execution environment.

As an upstream all I need to do to make snap-confine setuid root is to put install -m 4755 in my Makefile. As a upstream that packages the software I need to jump through a burning, electrified, moving hoop to do that.

Fedora strongly discourages setuid root executables. They are allowed but not recommended. Instead of coarse statement “I want to be root” you should use extended file system attributes to convey a set of Linux capabilities that the application really requires. Alone in itself this is a good idea. It limits potential impact in a security compromise.

At the time I just rolled up my sleeves and did the work for snap-confine to work correctly using capabilities. I upstreamed that patch to snapd. The work-in-progress package for Fedora used this but the patch bit-roted, untested as new demands apparently broke it. At the time of this writing I didn’t investigate why the kernel denies one specific mount operation while allowing all others.

Debian is the exact opposite. You just cannot use capabilities and must run as root. I bet there’s a technical reason for this. It is just different.

OpenSUSE, to the best of my knowledge, requires you to first get a security review and then get your package / exact filename added the special package called permissions . Building your package before this is done results in a failure (and is quite annoying I may add).

Needless to say, this issue is still not resolved. For OpenSUSE the +s bit can, apparently, be used, side-stepping the permissions package but it relies on a documented hack that just fools the packaging review tools.

Digression. Here I must shout out to the fantastic OpenSUSE security team that did a review of snap-confine a while ago! End of digression.

Process

The process of contributing a package to a distribution obviously differs depending on each distribution but one thing is invariably constant. Having a friend helps and helps a lot. It is extremely useful if said friend is also familiar with various quirks and unwritten rules. Lastly it is incredibly useful if said friend has super powers that can review and land packages into the repository. If you read this post while looking for an advice on how to package software that is the best advice I can think of. It remains universal and true, regardless of the type of distribution: highly bureaucratic and rigid or extremely liberal and permissive.

Bottom line

Snapd is easy to build from source and run in almost any environment. With systemd and recent enough kernel (which includes ancient 3.10) you should be able to get devmode (without the security) copy of snapd running in a few minutes. Packaging that for wider use is a rat’s hole of issues and it is not done. Snappy is a free software project, if you care about snaps do help me out. Reach out to either myself or to any other snapd developer. We’re almost always on #snappy IRC channel on the freenode network as well as on the #snappy channel on rocket chat.

One day I will write a book about packaging software for Linux distributions. I will start the book with the phrase Ihis book is already out of date…