Why Debian returned to FFmpeg

This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

Slightly less than one year ago, the Debian community had an extended discussion on whether the FFmpeg multimedia library should return to the distribution. Debian had followed the contentious libav fork when it happened in 2011, but some community members were starting to have second thoughts about that move. At the time, the discussion died out without any changes being made, but the seeds had evidently been planted; on July 8, the project's multimedia developers announced that not only was FFmpeg returning to Debian, but it would be replacing libav.

Chances are that many Debian (and Ubuntu) users are not more than peripherally aware of which multimedia library is running on their system. Libav has been in use for some years and has generally filled the bill, so it is natural to wonder what drove the project to make a change that will require a lot of work and which seems certain to prove disruptive while it is underway. Getting to the answers is an interesting study in how distributions try to ensure that the code they ship comes from healthy upstream projects.

Security and more

The Debian project is not normally known for being afraid to ship multiple packages providing the same function, so one might wonder why it can't just ship both FFmpeg and libav, letting users decide which one they want to use. The big sticking point in 2014 was security support. Both projects have had more than their share of security issues, and the Debian security team didn't think it could keep up with patching both of them. At that time, FFmpeg seemed to have a better record of responding to vulnerabilities than libav, but that still was not enough to convince the security team to support both libraries.

One year later, security issues remained at the top of the list, but it would appear that FFmpeg has pulled well ahead of libav in this regard. Debian security team member Moritz Muehlenhoff made it clear that he saw FFmpeg as being more responsive to security reports. A rather stronger argument came from Mateusz “j00ru” Jurczyk who, in his security-oriented role at Google, has been doing extensive fuzz testing of both projects and reporting the problems that come up:

We have been quite successful working on the above effort with FFmpeg for the last ~3.5 years: every single issue we have found (even the least severe ones) has been fixed in a timely manner. As a result, after tens of fuzzing iterations, there are currently no bugs in FFmpeg that we are able to find using our current input corpus and mutation algorithms. The situation is entirely different with Libav, which is still affected by hundreds of such bugs, even though we have provided the developers with reproducing testcases a number of times in the past.

The notion of hundreds of open security issues is generally unappealing. FFmpeg does not appear to lead on just security updates, though; by all accounts, it now supports a far wider variety of codecs and containers than libav does. There is an increasing range of formats that FFmpeg can play, but libav cannot. As the feature gap grows, the project's desire to stay with libav wanes.

The libav maintainer's perspective

When Alessio Treglia restarted the discussion at the end of April, the above points were quickly expressed. Even so, the conversation did not appear to be heading toward any sort of consensus. Arguably, the turning point was when Debian libav maintainer Reinhard Tartler entered the discussion. Reinhard argued forcefully for the advantages he saw in libav, but, in the end, could not bring himself to say that he was sure libav was the better choice.

With regard to security issues, Reinhard attributed the difference in fix rates to a difference in how the two projects approach development ("Michael" is Michael Niedermayer, the lead developer of FFmpeg):

Michael seems to have much more capacity and time, and thus is usually faster with pushing patches for such crashers. Libav takes the time to investigate, reproduce and understand those patches. Unfortunately, in the majority of cases, this is not trivial at all, often because of terse (or even wrong) commit messages, or the fact that there are better places to fix a particular issue in the code. "Better" usually means that more than a single instance of the issue is fixed.

Reinhard initially asserted that, even so, libav had parity with FFmpeg when it came to fixing security-related bugs, but he later backed down on that.

In Reinhard's view, the two projects are managed differently, with different goals; that difference makes libav appealing in a number of ways. Libav, he said, is trying to improve the state of the code and come up with something better than the "horrible" APIs it inherited from FFmpeg. He summarized the differences between the project this way:

I think the debate on the development methodology is the biggest disagreement between the two projects: FFmpeg insists on keeping its development velocity and allowing developers to "get work done", compromising on maintainability and common understanding of the code base in favor of more features. Libav on the other hand rather focuses on clean implementation and let's say better designed APIs. This requires more work, which in effect significantly lowers the development velocity. For a system integration job on the scale of Debian, less velocity seems more appealing to me.

Even through he seems to like the libav approach more, Reinhard, in the end, was not able to argue against the change; his position came down to: "I still have some concerns with this move, but I can't claim Libav to be superior to FFmpeg at this point". With the project's libav maintainer taking that position (and also, importantly, saying that he no longer has the time to maintain libav at the same level as he has in the past), the decision seemed to settle out fairly quickly.

Other concerns

A desire that was expressed more than once in this discussion was that the two projects would stop fighting and join back into a single, well-supported effort. There is, however, no real indication that any such reconciliation is in the cards. There is another way that the community could go back to having a single project, though: if one of them were to simply fail. Dmitry Smirnov suggested that a switch to FFmpeg by Debian could maybe bring that about:

I have a feeling that Debian already became life support for libav. Ever since Debian chosen libav, FFmpeg remained alive and apparently doing well without our help. I'm not too sure if libav would be able to stay alive without Debian.

Opinions vary on how much "life support" Debian actually provides to libav, but the loss of Debian and Ubuntu seems certain not to do the project any good. There aren't a lot of distributions out there that carry libav anymore; without Debian, that list will be short indeed. It might just be that libav is not sustainable without Debian.

That said, there are some concerns about the sustainability of FFmpeg as well. By all accounts, Michael is a highly productive developer; he accounts for, by far, the largest share of the patches going into FFmpeg. Reinhard asked whether FFmpeg is a one-developer project that would find itself in trouble should Michael stop working on it. "To me, this constitutes a serious bus-factor: Without Michael, (probably) nobody is able to replace him." He went on to suggest, though, that Michael's departure could do a lot to bring an end to the fork.

As an argument against the "one-man show" concern, Andreas Cadhalpun posted some commit statistics for both projects, covering the period since September 2014:

libav FFmpeg Commits Developer Commits Developer 294 Vittorio Giovara 1831 Michael Niedermayer 253 Martin Storsjö 294 Vittorio Giovara 206 Anton Khirnov 252 Martin Storsjö 131 Luca Barbato 197 Anton Khirnov 72 Diego Biurrun 179 Clément Bœsch 46 Michael Niedermayer 155 James Almer 32 Rémi Denis-Courmont 150 Carl Eugen Hoyos 21 Andreas Cadhalpun 114 Andreas Cadhalpun 17 Hendrik Leppkes 113 Luca Barbato 16 Gabriel Dume 98 Lukasz Marek 16 Himangi Saraogi 93 Paul B Mahol 16 wm4 85 Ronald S. Bultje 14 Federico Tomassetti 83 wm4 12 Peter Meerwald 66 Christophe Gisquet 11 Janne Grunau 48 Benoit Fouet

At a first glance, the table shows that (1) FFmpeg appears to have a much higher commit traffic than libav, and (2) Michael, while being the largest contributor, is certainly not the only contributor. But, as Reinhard pointed out, there is a bit more to this story. Changes to libav are routinely merged into FFmpeg, but the flow of patches in the other direction is quite low. If the libav changes are subtracted out of the FFmpeg numbers, the result is that Michael very much stands alone; no other developer is even close.

The Debian multimedia developers decided to make the switch to FFmpeg even though nobody really had an answer to Reinhard's concern. For now, FFmpeg appears to be going strong, but there is a single-developer risk there that could come to the fore in the future. Given that nearly the entire distribution ecosystem now depends on FFmpeg, chances are that a way would be found to keep the project going if Michael were to decide he had better things to do. But the process of getting there might prove to be a little rough.

The Debian project was faced with a difficult choice: given that it was not possible to support both libraries in the distribution, which one offers the most to Debian's users while presenting the least long-term sustainability and security risk? The developers involved chose to move away from a project that many of them see as lacking the resources needed to be truly healthy. That choice will result in a lot of work, but, assuming the choice was the correct one, Debian users should benefit in the long term.

