Robert M. "r0ml" Lefkowitz was the fourth and final keynote speaker at linux.conf.au 2017 in Hobart, Tasmania. He immediately served notice that his talk was going to differ from those of his predecessors; they offered "deep insightful questions", while he had answers. Also, the first three were nice people. What followed was an energetic event that left many in the audience wondering what they had just witnessed. The future of free and open-source software is grim, he said, but maybe we can make something better to replace it.

Your editor will endeavor to recount the main points of the talk, but there are limits to what can be done with the written word; watching the video is recommended for those who are interested.

Pia Waugh, in her keynote, said that we don't have to carry the systems of the past into the future, that we should challenge our assumptions; that was good advice, r0ml said. Dan Callahan correctly said that free licenses are not enough. Nadia Eghbal told us that we need to think about uncomfortable issues. To that last he posed the question: what is the future of free and open-source software? The answer was: it has no future.

We had set out on a goal to make the world better by liberalizing access to technology. Free and open-source software were a great tactic to advance that goal. Free software had a great run from 1984 to 1998; open-source software had another great run from 1998 to around 2018. But that run is coming to an end, what do we do next? He had three proposals for what the next thing should be: lithification, app freedom, and liberal software.

Lithification

The run of free and open-source software is coming to an end because it is no longer well matched to the substrate of where technology is. Imagine we were still using punched cards; how would one do a merge or a pull request in that environment? The technology would not support such strategies. Our strategies have to match the available technology, and, when the technology changes, we have to come up with new strategies to match.

At this point, he said, the crowd must think he's a bozo. Open source has won, after all. But others, including Tim O'Reilly and Doc Searls, have been saying the same thing. Searls said that we won the battle for Linux, but are losing the battle for freedom — a sad observation. Putting up a Google Trends plot, he showed that searches for "open source" have been in decline for years. The same is true for "free software" — and that includes people looking for free-as-in-beer software. Even plain "software" is in decline, as are searches for "Linux". People may call him a bozo and respond that software is eating the world, but the truth is that the world is eating software.

Searches for "CVS" follow a similar downward path. "Subversion" had a bump, but then went into a steady decline. "Git", instead, has been climbing for years and is still on the way up. Another line that is going up is "app". Yes, apps are just software. But if people think of them as two different things, then that is what they are. "Is Linux an app?", he asked. If it's not, but it is software, then perhaps the two are indeed different.

Small Pieces Loosely Joined is the name of a 2003 book about the web by David Weinberger. But that idea predates the web, it's part of what makes Linux great — pipes allow small pieces to be joined on the command line so that functionality can be composed. But that term also describes lithification, the process by which small grains, initially loosely packed, are compressed into sedimentary rock. That, he said, is what is happening to software.

Putting up a picture of the linux.conf.au web page on a handset, he noted that, if he wants to send it via email, he just hits the appropriate button and it happens. The page can be piped to another app. Zawinski's law says that all programs expand until they can send mail [yes, he said "send" even though the original is "read"]. But now all programs have to be able to do that, as well as send tweets and more. We can build this into our programs, or just make them all talk to each other — small pieces loosely joined.

While preparing the talk, he wanted to add a reference to a book. He thought about creating a barcode-reading app that would allow him to quickly obtain an image of a book's cover, but found that barcode-reading functionality is already built into the iOS and Android systems, along with other features like face recognition and optical character recognition. It's all built into the operating system now. The small pieces have all become a part of the substrate.

But that's Android, not Linux. Perhaps we can say that Android is "Linuxish". How about Windows? Now that it runs Bash and such, it's Linuxish too. Now we have GNU/Linux, Android/Linux, and Windows/Linux. This is a Linux conference, he said, so we can say that, once Linux is a subcomponent of all our other systems, we have won. But, going back to the search trends, it's clear that the world sees Android as being "some other thing that's better than Linux". That hurts.

So what do we do? The reason that commercial Linux is succeeding while free Linux is not is the latter's focus on copyright. Google has a concept called "vendoring"; rather than have dependencies, Google developers simply copy the code they need into their own repository. If that code is BSD licensed, the process is done. You'll have a big repository that you can package and distribute with no dependencies. That is lithification.

Why don't we in the Linux community do that? It feels wrong somehow. But there's a lot of evidence that shows it's a superior solution to the problem. Back in 2000, Butler Lampson, who arguably invented personal computing, argued that a project should only accept dependencies that are at least 5 million lines of code; 20 million lines is better. For anything smaller, just figure out which parts you need, copy them into your repository, and avoid having a dependency to deal with.

We're moving into a different world where the substrate is huge, while apps are getting smaller. Deployment environments are doing the same thing. The whole leftpad() fiasco was caused by developers who couldn't figure out how to implement that functionality themselves, and couldn't figure out how to copy the existing function into their own repositories. Or they could, but there is a social anxiety that prevents them from doing so. But this anxiety is fictitious. Copying the two functions we need from a big library to avoid dependencies is a skill we are all going to need to do lithification after open source no longer works.

App freedom

Bret Victor once said that advanced programming environments conflate the runtime environment with the development-time environment. Open-source methodologies are explicitly about preventing this conflation. But the two should not be separate. This failure to understand can be seen in the four freedoms that define free software. Explaining why this so requires looking at some history, or everybody will just think he's a bozo.

In the 1980s, the world was trying to figure out the best way to build software in a world where programs are large and people have to collaborate to get the job done. One school of thought said that we should just make everybody give their source to everybody else, and that will make it easier to develop software. "But everybody knew that would never work".

The other school of thought was object-oriented programming, where you don't need to see the source for the libraries you're using because the abstraction allows behavior to be clearly described, inherited, and overridden. There would be no issues around conflation of development time and run time or copyright, and we would be able to lithify our software. That didn't work out, and it's now just another legacy we're stuck with, he said. But that was the state of the world at the time.

Also at that time, ordering a GNU CD from the Free Software Foundation was relatively expensive. That expense was arguably unnecessary, but the process of building and delivering the software appeared to have value. That is partly because the recipient was expected to have a computer in their basement to run that software on. But we don't live in that world anymore.

Now, when you use a computer, it's nowhere near you. Almost everything you do is running on a distant machine that you can't download software to or build stuff on. But that's what you want to do: make changes to your code and run it on somebody else's computer. That is the hard part: replicating Google's vast number of servers in your basement. Without that, the second freedom, the ability to modify the software, is lost to us.

Imagine if you wanted to change how GitHub works. If you had the source, all you would need to do would be to reproduce GitHub's entire infrastructure in your basement, make your changes, and convince all of your friends to use the result. Or you could somehow get GitHub to allow you to make the change. The latter is generally easier, and points out another failing in the four freedoms. Freedom 3 is the right to distribute modified versions to others, but, in this world, it's not necessarily access to the source that enables this freedom. Instead, it's access to the runtime environment.

GPLv3 section 6 actively disallows this sort of freedom; it reads:

Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network.

Anybody who thinks that a network provider couldn't find some reason to proclaim any given change harmful is more optimistic than he is. GPLv3 gives license to that sort of discrimination because it doesn't care what happens at run time; it is only concerned with development time. The authors of the GPL think that development and run time are two different things.

Imagine a world where source-code management was incorporated into a web site. Users could push their own modifications on branches and specify which version they wanted to use. Even if the source is available today, we don't know how to give people the freedom to modify our web sites at run time. We give them the source, but no way to deploy it. That is something we need to change.

When he raises this kind of idea, people immediately start asking questions. How would security issues be handled, for example? Wouldn't it be great, he asked, if there were a community of people out there who understood computers, who were committed to liberalizing access to software, and who might be able to work out technical answers to some of these questions? They might be able to figure something out. He would call the result "liberal software".

Here he took a moment to say that, if you think he's a bozo, you should join the Software Freedom Conservancy to help them defend the free-software world that we have now. If, instead, you agree with him, you should realize that we still need free software until this future is built. So you should still join the Software Freedom Conservancy.

End-user development

Years ago, Guido van Rossum started the "Computer Programming for Everybody" project to pursue a utopian vision in which everybody could program. That didn't work out. Now there seems to be a renewed interest in learning how to program, though it's mostly focused on jobs. But one should consider: what is the most successful programming environment ever created? Excel. More people have solved more problems using spreadsheets than any other way. Spreadsheets don't have separate source and libraries; everything, including the data, is all mixed in together.

Perhaps someday one could request an app by asking Google or Siri to make it. This is a scenario a lot like Excel, with some similar issues, including just where the source code is. An increasing amount of the "source" is personal data stored in databases; nobody would want that to be released. In a world where data and code are intermingled, releasing the result looks increasingly like an invasion of privacy.

So copyright starts to not apply anymore. Already "nobody thinks it's a good idea" as demonstrated by how few GitHub repositories have a license associated with them. Before 1974 software couldn't be copyrighted; it was seen as a mathematical activity. An alternative view said that it was a literary activity instead. The government's view was that it was a useful activity. It was important to the promoters of free software to convince people that software is not useful, because otherwise it would be patentable. We had to invest a lot of energy into copyright and into convincing people that software was not useful, it was literary. We have succeeded with open source.

One can see how the world is changing today. There are already sites that will ask about a user's requirements, then construct a library to suit. As the tooling gets better, constructing such things will become easier. People's freedom will increase as they can construct their own programs and deploy them on infrastructure they don't have to manage themselves.

Some of us, he concluded, are going to have to build that.

[Your editor would like to thank linux.conf.au and the Linux Foundation for assisting with his travel to the event.]

Comments (35 posted)

Benno Rice, a member of the FreeBSD core team, might be expected to feel out of place at linux.conf.au, but it was not his first time there. While at the 2016 event in Geelong, he saw a presentation on Rust community automation [YouTube] by Emily Dunham and wondered: why can't FreeBSD have such nice things? The 2017 conference, held in Hobart, chose "the future of open source" as its theme but, Rice said, he was there to speak about the past; by looking at how FreeBSD ran into trouble, perhaps we can all learn something useful about how we run our projects.

He got involved with open source in the 1990s; he actually started with Linux, but somebody told him that "Linux is rubbish" and he should use FreeBSD instead. So he bought an iMac computer and got FreeBSD running on it; the project then punished him by giving him commit access. It is a great project, but it does have some problems relating to three factors in particular: FreeBSD is big, it's old, and its leadership can be slow to act.

How big is FreeBSD? The project's Subversion repository is currently about 3.1GB in size; a checked-out tree takes about 600MB. It consists of 71,100 files, about 32 million lines of code. It takes 20-30 minutes to build the whole thing, which is a big improvement from the old days, when it could take several hours.

The project is old, having gotten its start in November 1993; it was based on the 1992 386BSD release which, in turn, was based on much older code. So there is a lot of history. FreeBSD developers have only recently discovered automated testing, he said, but, for such testing to work, the software has to be structured properly, and FreeBSD isn't. Testing the kernel in an automated manner is especially hard. They do have some tests, though, many of which were borrowed from NetBSD, but they are focused mainly on user-space code.

With regard to slowness to act: like many community groups, FreeBSD is consensus-driven. The leadership is a nine-person elected team, but this team does not want to be seen as picking sides in any particular debate. The core team also does not want to be in control of the project's architectural direction. Originally, the core team was made up of the four founders of the project. The project as a whole followed the usual BSD model, giving commit access to good contributors, then appointing the best of those to the core team. Around 2000, though, the project switched to the current elected model. It works, but the consensus-driven model makes decisions slow.

Tools

For an example, consider the problems around tools and processes. A development project's source code is its core asset, and it must be well looked after. So the project needs a source-code management system. Initially, FreeBSD used CVS, which was a reasonable choice at the time, but the project stayed with CVS until 2008. Why didn't FreeBSD move off of CVS faster? It worked well enough for basic release management, but the real problem was that there was no consensus around what should replace it. BitKeeper was considered, but that was a proprietary system. Git and Mercurial came around later. The project considered Git, but didn't think it could handle a repository of that size, so it moved on.

The discussion went on for a long time. All communities have their perennial arguments that come back over and over; for FreeBSD, this was one of them. Developers did trial conversions to other systems, but consensus was elusive. When the decision was finally made, the project chose Subversion. It was not an optimal solution, but it did at least allow for changesets involving more than one file at a time, which was the biggest pain point with CVS at the time. In 2008, after eight years of arguments, Peter Wemm simply imposed a switch to Subversion; there has been relatively little whining since.

In comparison, Python started on CVS, moved to Subversion in 2004, then to Mercurial in 2009, and is now moving over to GitHub. The Python community's PEP process was helpful enabling the changes to happen. If, in the Python community, you want to make a change, there is a process to follow that forces you to think about it and make good arguments. Once the PEP is approved, you are clear to proceed. FreeBSD has no such mechanism.

In the Django community, developers realized that they were already using GitHub for everything except Django itself, so they simply pushed the change through and were done with it.

The source-code management debate was an example of the core team not wanting to take sides on an issue. The usual response in this kind of situation is to tell the developer to go and do a proof of concept implementation; that usually shuts them up, he said. But the only way to make a decision like this more quickly is to take a side. Doing so might have caused more "project angst", but perhaps that wasn't avoidable. Meanwhile, the project is talking about moving to GitHub. It has a repository there now, but nobody is paying attention to it, so if a developer submits a pull request, nobody will see it. With GitHub, he said, you have to either be there for real or not at all.

Rock-star developers

Another example has to do with community interactions, and the "not fun" task of dealing with bad behavior. A small community can usually talk things out fairly easily, but FreeBSD has hundreds of committers. FreeBSD leaders thus have to do a certain amount of human resources work. Developers tend to look down on the human resources department in their company, but it does have its value; it's how you don't end up in court. FreeBSD's disagreements are unlikely to end up in court, but you still need somebody to step in and straighten things out occasionally.

FreeBSD does maintain a committer's guide, which has two sections on interactions with other developers. It has advice like "do not impugn the intentions of somebody you disagree with", "accept correction", "ask for help", and more. There are also rules, including "respect other committers", "discuss significant changes before committing them", and "do not fight in public with other committers". There are some paragraphs on actions the core team can take to deal with violations; these include suspending commit access or removing a violator from the project's mailing lists. It could, he said, be the beginning of a code of conduct for the project.

"One of the more spectacular incidents" in this area involves the famous Matthew Dillon, who joined the project in 1994. He quickly showed himself to be a talented developer, with a special knack for dealing with esoteric virtual-memory bugs. But he was not a team player, preferring to be in charge. So, when he started working in any particular area, he would tend to take charge at the expense of anybody else working on that code.

Like most early multiprocessor-capable kernels, FreeBSD had a big kernel lock; Dillon wanted to get rid of it for the FreeBSD 5 release. John Baldwin had been already looking at breaking up the big kernel lock; Dillon joined the effort and things went well for a while. Then the two developers ran into a disagreement over how critical sections should be handled. Baldwin argued for simply disabling interrupts, but Dillon said that was too inefficient; he committed a different change involving a flag in the low-level interrupt handler. It was faster on the x86 architecture, but x86 is not the only architecture supported by FreeBSD.

This change led to a number of big arguments, with Dillon eventually backing out the change. A month-long acrimonious debate followed; eventually Dillon's change went in, but there were a lot of other, similar incidents. It was a classic case of a "rock-star ninja 10x developer" not getting along with the rest of the project.

The core team responded with a set of new rules about when commits could be made and the penalties for engaging in "commit wars". Eventually the problem came to a head when Dillon committed a change to what was supposed to be the FreeBSD 5 stable branch. After more argument, he backed it out with a commit that read: "Bow to the whining masses..."; it also threw in some unrelated cleanups "just for the hell of it". After that, the core team took his commit access away, and he eventually left the project.

Leadership is hard, Rice said. If the project had had an established code of conduct at that time, it might have been in a better position to deal with this rock-star developer problem.

Burning trash piles

After Dillon left, things calmed down for a while — until Gamergate ("a burning trash pile made of other burning trash piles") came along. One of the figures in Gamergate was Randi Harper, a former FreeBSD developer. She did a lot of much-appreciated work on the installer, then got a job at Amazon and, as often happens with Amazon employees, she stopped doing community work, though she still hung around on the mailing lists and such.

Harper, a combative personality, jumped into the Gamergate fray with gusto after being attacked. One day she decided to block the Gamergate "trash humans" out with a Perl script; it worked, causing them to lose their minds, Rice said, at which point they started attacking FreeBSD. The core team started getting emails asking it to deal with the problem, saying that Harper was dragging FreeBSD into the whole mess. Initially, they responded by saying it wasn't related to the project and there wasn't much they could do about it.

Things got worse when a FreeBSD developer joined the Gamergate side and started attacking Harper on Twitter; she attacked back. Things moved to a private FreeBSD IRC channel, which witnessed some heated discussions; the attacker then leaked a log of the conversation to Breitbart. That was a violation of the project's policies, so, at that point, the core team had no choice but to get involved.

Harper wanted three things from the core team: an official code of conduct for the project, a statement that the team was investigating this behavior and would not tolerate harassment, and for that person to be quietly removed from the project. The core team responded that the code of conduct was already in progress and that sanctioning the other developer was hard due to the fact that the bulk of the activity happened outside of the project's communication channels. Policing your own area is relatively easy; it's harder when things happen on sites like Twitter or Reddit. In any case, that became a moot point when the developer resigned.

The FreeBSD project does now have a code of conduct; it is, Rice said, a "patched-up rush job". Project members still honestly thought that "meritocracy" meant "inclusiveness" initially. They have since learned otherwise, and the code is being reworked. Meanwhile, Harper wrote a blog post about how badly the whole thing was handled; there were some valid criticisms in there, Rice said, with some things to learn from.

The real trouble with FreeBSD

So how can the problems with FreeBSD be fixed? Some of those problems are a function of the project's size, which is something that can't really be fixed. There are things that could be done, though. FreeBSD is still shipping sendmail, for example, which may no longer be necessary; taking it out would make FreeBSD a little smaller. FreeBSD is still a big, monolithic distribution, but there is some talk of splitting it up into packages. This is a place where interested people could help. There are also opportunities in automation, testing, and elsewhere.

Other problems are tied to the project's age, but that is hard to fix too. FreeBSD will not be getting any younger. It can be hard to change baked-in attitudes. For example, there is resistance to moving to GitHub because people in the core team and beyond are attached to running their own infrastructure. Such things can be worked around, but an attitude shift will be needed.

With regard to leadership, that could be fixable but it, too, would require an attitude shift in the core team. The team does not need to be an architectural leader, it needs to be a "scaffolding leader". It just needs to ensure that "the things the project needs to do a good job" are available. The core team should focus on how FreeBSD is made, rather than what it should be. Its job should be reducing friction for developers.

But that points to the real trouble with FreeBSD: it is made and led by volunteers. There is nobody whose job is purely to make FreeBSD more awesome. There are plenty of people tasked with making it more awesome for their employers, but that work doesn't always help in the long term. Those employers get a faster build system, but automated testing languishes. Somehow, the project needs more resources to be able to focus on making FreeBSD "a fun and awesome place" because that is how they can ensure that it will be around for a long time to come.

The full video of this talk is available.

[Your editor would like to thank linux.conf.au and the Linux Foundation for assisting with his travel to the event.]

Comments (64 posted)