Among its many activities, the Software Freedom Conservancy (SFC) is one of the few organizations that does any work on enforcing the GPL when other compliance efforts have failed. A suggestion by SFC executive director Karen Sandler to have a Q&A session about compliance and enforcement at this year's Kernel Summit led to a prolonged discussion, but not to such a session being added to the agenda. However, the co-located Linux Plumbers Conference set up a "birds of a feather" (BoF) session so that interested developers could hear more about the SFC's efforts, get their questions answered, and provide feedback. Sandler and SFC director of strategic initiatives Brett Smith hosted the discussion, which was quite well-attended—roughly 70 people were there at a 6pm BoF on November 3.

Sandler started by noting that she is passionate about free software in part because she has a heart defibrillator implanted in her body, but has no access to the software running there. Smith recently joined SFC after stints with the Free Software Foundation as a license compliance engineer and as a system administrator at the World Wide Web Consortium (W3C). The SFC currently has 40 projects under its umbrella, for which it handles various administrative tasks, but the organization also runs the GPL compliance project for Linux kernel developers, which was the topic for the session.

Sandler laid out the ground rules for the discussion: no recordings and to ask people before quoting them (which we have followed here). As is generally the case, those who spoke up are not representing the company they work for, but were speaking for themselves. And, of course, Sandler is a lawyer, but was not giving any legal advice ("iaal, but tinla, ianyl" from her slides).

The only current legal action that the SFC is involved with is Christoph Hellwig's GPL enforcement suit against VMware, which was filed in 2015 and dismissed back in August—though Hellwig is appealing. The SFC is funding the suit as part of its GPL compliance project. Hellwig spoke up to say that he was particularly interested in hearing the opinions of those in the room who are not part of the SFC GPL compliance project.

Sandler noted that the SFC had just released copies of the template agreements for developers who join the compliance project, either anonymously [PDF] or under their own name [PDF]. That is an effort to be more transparent about the project, which is, as with all GPL enforcement efforts undertaken by the SFC and others, done under "The Principles of Community-Oriented GPL Enforcement". That document makes it clear that the goal is for GPL compliance and that legal action is a last resort. That document is "codifying our gentle approach" to enforcement, she said.

McHardy suits

She asked the room who had heard of Patrick McHardy and the GPL-enforcement lawsuits in Germany. Those suits appear to be well outside community expectations of how enforcement should be done (as well as running afoul of the Principles). Most in the room seemed to have heard of the suits, so Sandler did not give an introduction to all of that.

An audience member asked why it is that people have such a negative view of enforcement. Sandler said there is lots of confusion out there about lawsuits; they also tend to make people skittish. One key thing to remember is that today's violators are tomorrow's contributors, which is something that it is rare for companies to acknowledge—Samsung has been comfortable talking about that transition, but few others are. There is also a lot of "fear, uncertainty, and doubt" (FUD) about lawsuits, which muddies the waters. The Principles are meant to help with that by bringing clarity so that companies and others know what to expect with respect to GPL enforcement.

Lawsuits have a cost, in terms of time, money, opportunity, and other things, Smith said. It is natural for some in the community to question whether a particular suit was worth it, which can also lead some to have a negative view of enforcement actions.

But the McHardy suits are entirely different; James Bottomley called them a "generic shakedown". It means that there is a "copyright troll" in the community who is extracting a "couple of hundred thousand euros" in settlements from companies, he said. The existence of that troll makes companies have the mindset that the GPL is bad, he said. It makes for a "nasty environment" where companies will tar and feather anyone launching lawsuits because they associate them with the McHardy shakedown.

Grant Likely noted that there is more to the GPL than simply a license; it is a value statement as well. Using the GPL says that a project values collaboration and openness; it is both a social document and a legal document. Likely said that stressing that to companies helps when talking to them about the GPL.

Hellwig said that he preferred to refer to his suit as a copyright action, rather than a GPL-enforcement action. In the end, that is what it is and copyright lawsuits happen all the time in industry. But if the Linux community uses that same tool, suddenly it becomes a big problem.

The Linux community has been so lax in its enforcement of the GPL that it has made it much easier for McHardy to find targets, David Woodhouse said. If the community had pushed harder for compliance and done more enforcement, those companies would have been in compliance, which would have left McHardy with fewer (or no) companies to go after. Though Tim Bird pointed out that more enforcement may have led to fewer kernel contributors; that was one point that Linus Torvalds was making in the long ksummit-discuss thread, Bird said.

BusyBox is often brought up as an example of a community that was "killed" by the GPL enforcement done on its behalf, Laura Abbott said. It is a common complaint, she said, but is it true? Sandler said that it was an exaggeration, but that the BusyBox community "took a hit" for its enforcement actions. The suits bolstered other GPL projects, but companies switched away from using BusyBox. On the other hand, in-house counsel at some of the affected companies thanked those who brought the suits for making it easier to get the internal attention needed to address compliance issues.

Bird asked if anyone had evaluated removing McHardy's code from the kernel. Bottomley said that the Linux Foundation had done that analysis and found that there was simply too much of it to remove and replace. But Likely objected to the idea of removing the code at all, noting that the principles behind the GPL are important. Removing code that is covered by the GPL in order to avoid lawsuits is not the right way forward; it sends the wrong message to companies and others about what the GPL means.

Matthew Garrett said that the message about why compliance is a good thing needs to get out; the benefits of compliance need to be clear, since compliance for its own sake "doesn't gain us anything". The number of times where a company doesn't want to comply is small, but they are often unable to do so because they don't have the code or even know what was shipped. Compliance is not just about getting the code upstream, it is proof that the company even can be responsible to update their devices for security flaws.

Sandler said that manufacturers are starting to require complete source code from their vendors because of the need to be able to do security updates, rather than only to be able to comply with the GPL and other licenses. The television industry is definitely in that camp, Smith said. In a world where TVs listen to everything we say near them, access to the source code becomes even more important.

But it is not just about making sure the license is respected for existing code, Ted Ts'o said, it is also about making sure that new projects choose copyleft licenses. There have been some who are claiming that the GPL and other copyleft licenses are in decline. He also wondered about using the term "enforcement" as it is a word that can be scary. Sandler agreed and said that the SFC tries to use "compliance" in preference to "enforcement".

Transparency

Ts'o said that he knew that the SFC can't talk about all that it does, but that he would like to see some kind of report that at least gives an indication of how much compliance work is going on. He suggested something like the transparency reports that other organizations provide. It could show how many reports of violations were received and what compliance steps had been taken for those in general terms.

Sandler said that the SFC is working on figuring out how to get numbers out there. It is publishing what it can, but there are a number of privacy and other considerations that need to be taken into account. Ts'o suggested that reports of that nature could help with fundraising as some might be more inclined to "open their wallet if they could see progress" without resorting to lawsuits.

Pursuing GPL compliance is a tortuous process, Sandler said. Once the SFC is in contact with a violator, it often takes many rounds going back and forth with deliberate delays between each. On average, the SFC gets twenty different source candidates from a violator before getting a version that is complete and correct. In the VMware case, that process finally stalled out after three years or so before the SFC realized that litigation would be required.

Smith said that he was at the FSF when it sued Linksys. Before that happened, there was an "infinite loop" of finding a product with a license problem, starting to work to resolve that problem, then finding that a new product with the same license problems had been released. The Linksys compliance process simply wasn't working so, after four years of these discussions, it was time to escalate to a lawsuit.

There are a lot of small and medium-sized companies out there that are rattled and scared by the McHardy suits, Thomas Gleixner said. There is a need to get information to them so they know what to do and how to comply. Sandler agreed and asked if there were good ways to get that information to them. Gleixner suggested that various trade organizations and the like that had these companies as members might be one way to do so. It is important for those companies to get the information from the "good people" first, before they are contacted by McHardy.

One audience member wondered about compliance tests or certifications that vendors could use to demonstrate that they are following the licenses. That certification would allow buyers to choose suppliers who comply and would encourage vendors to comply instead of forcing them to through enforcement efforts. Smith noted that the FSF Respects Your Freedom certification goes further than just license compliance, but it does provide a carrot for suppliers.

Hellwig noted that most companies that work with software will already have some kind of compliance officer (or should). Most software licenses in the industry are complicated, but the GPL is quite simple.

With that, time was running out. Sandler said that the SFC only performs compliance activities as a service to and under the direction of the developers that are part of the coalition. The organization is trying to be as transparent as it can about those efforts and the BoF was part of that. She encouraged anyone with feedback to bring it to the SFC. Woodhouse ended things by encouraging developers to sign up with the compliance project so they could have a say in what direction those compliance efforts take.

[ Thanks to LWN subscribers for supporting my travel to Santa Fe for LPC. ]

Comments (3 posted)

Ninja is a build tool, similar in spirit to make , that was born from the Chrome browser project in 2011. It is unique among build systems in that it was explicitly designed to be used as an "assembly language" by higher-level build tools. Beyond that, Ninja is fast. It is used by several large open-source projects and it has changed the way I approach build systems.

Ninja lacks many of the features that make users are familiar with: no conditionals, no string-manipulation functions, and no patterns or wildcards. Instead, you are supposed to put this logic into a separate program that generates Ninja build files; you aren't supposed to write Ninja build files by hand, though they are still quite readable. Some people write this generator program from scratch in a high-level language like Python, and they usually call it configure . Others use a build system such as CMake, GYP, or Meson, all of which can generate Ninja files based on rules written in their own custom language.

Despite its minimalist philosophy, Ninja does have a few features that make lacks: a progress report on its standard output, support for filenames with spaces, a "pools" mechanism to limit the parallelism of certain rules, and the ability to "include" other Ninja files without polluting the namespace of the including file. Ninja also has built-in support for some features that you could implement in a Makefile by using various techniques, but it's tedious to do so. I'll cover some of these in the article.

Rules and targets

A Ninja file to compile and link a C program from two source files might look like what's below. I have highlighted the targets (that is, the files that are created by the build process) in bold:

rule cc command = gcc -c -o $out $in description = CC $out rule link command = gcc -o $out $in description = LINK $out build source1.o: cc source1.c build source2.o: cc source2.c build myprogram: link source1.o source2.o

The last three lines are build statements. For example myprogram is a target, source1.o and source2.o are its inputs or dependencies, and link is the rule (defined a few lines earlier) that will build the target from the inputs. In the rule's command line, $out is replaced by the target and $in is replaced by the list of inputs, with the appropriate shell quoting. Ninja also supports implicit dependencies, implicit outputs, and order-only dependencies. Providing a description for a rule turns on automake-style silent rule output.

Auto-generated dependencies

In Makefiles, the standard technique for detecting changes to implicit dependencies (such as C header files) is to generate a small Makefile snippet that specifies the dependencies and to include this generated file into your main Makefile.

The equivalent technique in a Ninja file looks like this:

rule cc command = gcc -c -o $out $in -MMD -MF $out.d depfile = $out.d deps = gcc

Here, -MMD tells GCC to output the list of included files and -MF says where to write it. Normal compilation happens too; the dependency file is generated as a side-effect of the compilation process. The depfile statement tells Ninja where to read those additional dependencies from. The first time you run Ninja, it will build the source1.o file because it is out of date compared to its explicit source1.c dependency. The next time Ninja is run, it will check to see if source1.c has changed or if any of the header files listed in depfile have changed. To support this technique, Ninja understands just the subset of Makefile syntax that is actually generated by C pre-processors. Ninja also supports a similar feature of the Microsoft Visual C++ compiler, which prints specially-formatted lines to stderr when given the /showIncludes flag.

Keeping track of build command lines

Ninja remembers the command line that was used to build each target. If the command line changes (probably because the build.ninja file itself changed), then Ninja will rebuild the target, even if the dependencies didn't change. Ninja tracks this information across invocations by storing a hash of the build command for each target in a .ninja_log file in the top-level build directory.

You can do this with Makefiles too — the technique is described here and it's used by the Makefiles for both Git and Linux. But it's tedious, error-prone, and slow.

Generating (and re-generating) the build file

Ninja doesn't support loops, wildcards, or patterns; you're supposed to generate the Ninja build file from another program. A simple configure script written in Python might look like this:

f = open("build.ninja", "w") sources = ["source1.c", "source2.c"] for source in sources: f.write("build {outputs}: {rule} {inputs}

".format( outputs=source.replace(".c", ".o"), rule="cc", inputs=source)) # etc

If you want to support environment variables like $CFLAGS , it is best practice to read these variables in the configure script, and bake the values into the Ninja file. This makes it easier to maintain multiple build folders, such as a debug and a production build. The autotools behave this way.

Now if you edit your configure script to add another source file, source3.c , you'll want to ensure that build.ninja is re-generated. You can achieve this with another Ninja rule:

rule configure command = ./configure generator = 1 build build.ninja: configure

Thus, if build.ninja is out of date (older than configure ), Ninja will run configure to re-create build.ninja , before it does anything else. The generator statement is necessary to exclude the target (the build.ninja file) from being removed by Ninja's auto-generated clean command. In practice, you would also want to remember any parameters originally given to configure (such as $CFLAGS ), and bake them into the rule that re-runs configure .

If you're using a generator program like CMake, the principle is the same. The build.ninja file generated by CMake will arrange for itself to be re-generated if you edit CMake's build description file CMakeLists.txt .

Performance

Ninja's original motivation was speed. A no-op build of Chrome (where all the targets are already up to date) reportedly took 10 seconds with make , but less than a second with Ninja.

According to my own benchmarks, Ninja's speed difference is only really significant for very large projects (on Linux, at least). However I didn't try to implement, in make , my own version of Ninja's "rebuild if command-line changes", which would presumably slow the performance of make further.

Ninja generators and users

CMake is the most widely-used build system with Ninja support. CMake has always been a "meta build system" in that it generates build files for other build systems: various varieties of Makefiles, XCode project files for Mac, or Visual Studio project files for Windows. Since v2.8.8 in 2012, it can generate Ninja files as well.

GYP ("Generate Your Projects") is the build system used by Chromium and related projects such as the V8 JavaScript engine. As far as I know, it doesn't have much adoption elsewhere, and I don't know much about it.

Meson is a fairly recent build system that seems to be gaining traction. Unlike CMake, which seems to have come from the Windows world, Meson wants to provide first-class support for the Linux open-source ecosystem such as pkg-config, Gnome/GLib, etc., and the maintainers are happy to merge patches to support these types of projects. Maybe one day this will be able to replace autotools. (For many projects at least.) The GStreamer project recently merged "experimental" support for building with Meson — see this talk [video] at last month's GStreamer conference.

A few other Ninja generators are listed on the Ninja wiki, but it's hard to tell which of those are toy projects and which are suitable for large or complex projects.

Large projects that use Ninja include Chromium, of course; LLVM since 2012, via CMake; the Android Open Source Project, since late 2015, by parsing and translating GNU Makefiles; and GStreamer's experimental Meson-based build system mentioned above. Ninja is available in major Linux distributions — after all, it's needed to build Chromium. It's usually packaged as "ninja-build".

The Ninja community

Ninja was originally written by Evan Martin, who was working on the Chrome browser, in 2011. Martin handed over maintainership in April 2014 to Nico Weber (also on the Chrome team) because "I actually haven't myself used Ninja in something like two years", he said, having left the Chrome team. Even so, Martin is still active on the mailing list and in the Git logs.

In the last two years, the release cadence has slowed down to one release every six to ten months. The feature set is pretty stable; these days the releases contain mostly bug fixes, though some useful new features do occasionally make their way in.

The latest major release (1.7.1 in April 2016) had 160 commits by 28 different contributors. 57% of the commits were from Weber and Martin, 14% from other Google employees, 5% from Kitware (the company behind CMake), 7% from a handful of other companies (SAP, Bloomberg, SciTools), and the remaining 17% of commits from 16 contributors whose company affiliation isn't obvious from the Git logs. The mailing list is fairly quiet, with a handful of threads per month, but it does include a good amount of feature discussion, not just support queries.

Some fairly obvious bugs are still open after four years: for example, Ninja doesn't support sub-second timestamp resolution. Other convenient features never get implemented (such as deleting output files if the build failed like GNU Make's .DELETE_ON_ERROR ) partly because it's easy to implement workarounds in your Ninja-file generator. Keeping the Ninja codebase small and focused seems to be the driving philosophy. All in all the project seems healthy and mature. Ninja is written in C++ (12,000 lines of it, of which 40% are tests). It is released under the Apache 2.0 license.

Ninja's big idea

For me, Ninja's biggest contribution is to popularize the concept of generating build files from a real programming language. Many projects will find CMake or Meson to be a good fit, but when the needs are more complex, it can be surprisingly simple and elegant to use a real programming language like Python atop a dumb build system like Ninja or even a subset of make .

At $DAY_JOB , we build what is essentially a custom Linux distribution for appliances, with services packaged in Docker images. The build system was getting hard to debug and we were losing confidence in the correctness of incremental builds. We decided to try Ninja. Step one was to get rid of patterns and conditionals in the Makefiles, and write a Python script to generate the Makefiles. Step two was to output Ninja instead of make format. Before even getting to step two, however, we had already gained significant improvements in understandability, traceability, and debuggability.

Of course generating Makefiles is not a new idea — the autotools and CMake have been doing it for decades. But Ninja has taught me just how easy and flexible this approach is. For more information, Ninja's manual is a short and pleasant read. The free book "The Performance of Open Source Applications" has a chapter on Ninja that covers the original motivations for Ninja and some implementation details.

Comments (32 posted)

The "Internet of things" has become a buzzword for a whole raft of new products, many of questionable value, that the industry would like to sell to us. It has also come to symbolize many of the problems afflicting the software development and delivery process. In a brief, fast-paced talk during the 2016 Linux Plumbers Conference's Android microconference, Kees Cook described how the Brillo project is trying to fix some of those problems, especially with regard to kernel maintenance.

Brillo, he said, is a software stack for the Internet of things based on the Android system. These deployments bring a number of challenges, starting with the need to support a different sort of hardware than Android normally runs on; target devices may have no display or input devices, but might well have "fun buses" to drive interesting peripherals. The mix of vendors interested in this area is different; handset vendors are present, but many more traditional embedded vendors can also be found there. Brillo is still in an early state of development.

There are a number of longstanding problems endemic to this area. Each device has its own special, static kernel version mixing changes from multiple trees, including the mainline, the common Android tree, and any vendor-specific trees. Fixes and new features must be backported to this kernel, and out-of-tree drivers must be carried forward for future products. As the number of products increases, the number of combinations of kernels, patch sets, and hardware configurations grows exponentially, leading to maintenance problems. This growth is relatively manageable when the problems are small, but one of Brillo's requirements is device support for at least five years after the last unit is sold. On that sort of time scale, exponential growth in maintenance issues is simply not sustainable.

The solution to this problem, according to Cook, is a simple matter of making two changes:

Maintain a single kernel for all systems, reducing patch combinations and backporting work.

Keep everything in the mainline kernel, reducing forward-porting work when a new kernel comes along.

Cook allowed as to how those principles might scare some vendors but, he said, if this approach seems too scary, "you're not testing enough."

Brillo is thus built in a single kernel tree containing the Android patches and all necessary vendor patches. This adds an interesting constraint, as it requires the vendors to all play well together with their own patches. These vendor patches should preferably be upstream anyway but, in any case, they must have been sent upstream for consideration. The kernel itself is the latest long-term support kernel from Greg Kroah-Hartman, and it follows the -stable updates as they are released. When a new long-term support kernel comes out, everything moves forward to that release.

Part of making this idea work is reducing the delta between the Brillo kernel and mainline. There are about 600 patches in the Android common kernel currently, Cook said; that has been reduced to less than 150 in the Brillo kernel. That was done by consolidating small patches, tweaking the Android user-space code to not need the patches in the first place, and upstreaming the patches that are easy to get merged.

The upgrade process has been tested once, in the move from the 4.1 to the 4.4 kernel. It went relatively easily and, happily, the list of add-on patches got quite a bit shorter, thanks to the upstreaming of a fair amount of vendor code. It was also possible to drop a whole bunch of backported patches, thanks to the newer kernel. This test may have only been run once so far but, Cook said, it demonstrates that the idea is "not entirely crazy."

For vendors who are afraid of regressions from kernel upgrades, Cook had some advice: get your code upstream. Then, create a better set of automatic tests to verify that everything is working. All vendors should be thinking about just what they fear might break and write tests to detect that when it happens. It is hard work, but it has to be done anyway to verify that things work in the first place; it also only has to be done once. Then perform regular testing on linux-next to catch problems before they end up in the next long-term support kernel.

Will this approach work? He certainly hopes so, he said. Something has to be done to get out of the "backport treadmill" that vendors are on now. Most vendors, he said, have already agreed to this approach, and they are becoming more proactive about upstreaming their code. Some vendors fear the five-year support rule but, for many in the embedded world, five years looks relatively short and doesn't bother them at all. "Handset vendors panic" at the idea, he said, but, in the end, they are going to have to decide between paying the up-front costs of upstreaming their code or the long-term costs of supporting old code for far longer than they have been accustomed to.

[Thanks to LWN subscribers for supporting our travel to the event.]

Comments (3 posted)

November 24 is the traditional Thanksgiving holiday in the US. As is our own tradition, we'll be taking the week off on the theory that we will have eaten too much food to get any useful work done anyway. The front-page stream will be occasionally updated as usual, and we'll be back to normal operations for the December 1 edition. Happy Thanksgiving to all who are celebrating it — and those who aren't as well.

Comments (none posted)