Improving Ubuntu's application upload process

This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

There has been a surplus of articles recently on how Linux "lost" the desktop. Fingers have been pointed in all directions, from the Windows monopoly to competing desktop projects to Linus Torvalds's management style. Over in the Ubuntu camp, though, there does not appear to be any sense that the desktop has been lost; they are still working hard to win it. Ubuntu's recently-proposed new application upload process highlights its vision of the desktop and what they think needs to be done to make things happen there.

The problem

Serious Linux users tend not to think of availability of software as a problem; distribution repositories typically carry tens of thousands of packages, after all, and any of those packages can be installed with a single command. The problem with distribution repositories, from Ubuntu's point of view, is that they can be stale and inaccessible to application developers. The packages in the repository tend to date from before a given distribution release's freeze date; by the time an actual distribution gets onto a user's machine, the applications found there may be well behind the curve. In some cases, applications may have lost their relevance entirely; as Steve Langasek put it:

Why put an upstream through a process optimized for long-term integration of the distribution when all they care about is getting an app out to users that gives them information about this month's beer festival?

Beyond that, getting a package into a distribution's repository is not something just anybody can do; developers must either become a maintainer for a specific distribution or rely on somebody else to create and add a package for their application. And, in most distributions, there is no place in the repository at all for proprietary applications.

Ubuntu's owner Canonical sees these problems as significant shortcomings that are holding back the creation of applications for the Linux desktop; that, in turn, impedes the development and adoption of Linux as a whole. So, a few years back, Canonical set out to remedy these problems through the creation of the Ubuntu Software Centre (USC), a repository by which developers could get applications to their users quickly. The USC is not tied to the distribution release cycle; applications added there become available to users immediately. There is a mechanism for the handling of payments, allowing proprietary applications to be sold to users. A glance through the USC shows a long list of applications (some of which are non-free) and other resources like fonts and electronic books. Guides to nearby beer festivals are, alas, still in short supply.

Naturally, Canonical does not want to provide an unsupervised means by which arbitrary software can be installed on its users' systems. Experience shows that it would not take long for malware authors, spammers, and others to make their presence felt. So the process for putting an application into the USC involves a review step. For paid applications, for which Canonical takes a 20% share of the price, there appears to be a fully-funded mechanism that can review and place applications quickly. For free applications, instead, review is done by a voluntary board and that group, it seems, has been having a hard time keeping up with the workload. The result is long delays in getting applications into the USC, discouraged developers, and frustration all around.

Automatic review

The new upload process proposal aims to improve the situation for free applications; Canonical does not seem to intend to change the process for paid applications. There are a number of changes intended to make life easier for everybody involved, but the key would appear to be this:

We should not rely on manual reviews of software before inclusion. Manual reviews have been found to cause a significant bottleneck in the MyApps queue and they won’t scale effectively as we grow and open up Ubuntu to thousands of apps.

In other words, they want to make the process as automatic as possible, but not so automatic that Bad Things make it into the USC.

The first step requires developers to register with the USC, then request access to upload one or more specific packages. Getting that access will require convincing Canonical that they hold the copyrights to the code or are otherwise authorized to do the upload; it will apparently not be possible for third parties to upload software without explicit permission, even if the software is licensed in a way that would allow that to happen. A review board will look at the uploader's application and approve it if that seems warranted.

Once a developer has approval, there are a few more steps involved in putting an application into the USC. The first is to package it appropriately with the Quickly tool and submit it for an upload. That is mostly basic packaging work. Uploads through this mechanism will be done in source form; binaries will, it seems, be built within the USC itself.

But, before the application can be made available, it must be accompanied by a security policy. The mechanism is superficially similar to the privilege scheme used by Android, but the USC bases its security on the AppArmor mandatory access control mechanism instead. The creation of a full AppArmor profile can be an involved process; Canonical has tried to make things simpler by automating most of the work. The uploader need only declare the specific access privileges needed by the application. These include access to the X server, access to the network, the ability to print, and use of the camera. Interestingly, access to spelling checkers requires an explicit privilege.

All (free) USC applications will run within their own sandbox with limited access to the rest of the system. Only files and directories found in a whitelist will be accessible, for example. Applications will be prevented from listening to (or interfering with) any other application's X server or D-Bus communications. There will be a "helper" mechanism by which applications can request access to non-whitelisted files; the process will, inevitably, involve putting up a dialog and requiring the user to allow the access to proceed. That, naturally, will put some constraints on what these applications can usefully do; it is hard to imagine a new compiler working well in this environment, for example. The payoff is that, with these restrictions in place, it should not be possible for any given application to damage the system or expose information that the user does not want disclosed.

And, with all that structure in place, Canonical feels that it is safe to allow applications into the USC without the need for a manual review. That should enable applications to get to users more quickly while taking much of the load off the people who are currently reviewing uploads.

Naming issues

From the discussion on the mailing lists, it would seem that the biggest concern has to do with the naming of packages and files. If an application in the USC uses a name (of either a package or a file) that is later used by a package in the Ubuntu or Debian repositories, a conflict will result that is unlikely to make the user happy. Addressing this problem could turn out to be one of the bigger challenges that Ubuntu has to face.

Current USC practice requires all files to be installed under /opt ; this rule complies with the filesystem hierarchy standard and prevents file conflicts with the rest of the distribution. The problem, according to David Planella (one of the authors of the proposal), is that a lot of things just don't work when installed under /opt :

We are assuming that build systems and libraries are flexible enough to cater for an alternative installation prefix, and that it will all just work at runtime. Unfortunately, this has proven not to be the case. And I think the amount of coordination and work that'd be required to provide solid /opt support in Ubuntu would be best put in other parts of the spec, such as its central one: sandboxing.

In other words, the /opt restriction was seen as making life difficult for developers and Ubuntu lacks the resources and will to fix the problems; the restriction has thus been removed in the proposal. With Ubuntu, Debian, and USC packages all installing files into the same directory hierarchy, an eventual conflict seems certain. There has been talk of forcing each USC package to use its own subdirectory under /usr , a solution that, evidently, is easier than /opt , but nothing has been settled as of this writing.

Presumably some solution will be found and something resembling this proposal will eventually be put into place. The result should be a leaner, faster USC that makes it possible to get applications to users quickly. Whether that will lead to the fabled Year of the Linux Desktop remains to be seen. The "app store" model has certainly helped to make other platforms more attractive; if its absence has been one of the big problems for Linux, we should find out fairly soon.

