Debian Buster will only be 54% reproducible (while we could be at >90%)

To: debian-devel@lists.debian.org, General discussions about reproducible builds <rb-general@lists.reproducible-builds.org>

Subject: Debian Buster will only be 54% reproducible (while we could be at >90%)

From: Holger Levsen <holger@layer-acht.org>

Date: Tue, 5 Mar 2019 13:33:30 +0000

Message-id: <[🔎] 20190305133330.i3jcqnku6qhd355y@layer-acht.org>

hi, disclaimer: this has not yet been verified by anyone other than myself, so I could very well be wrong. Reproducible builds are about enabling anyone to independently verify that... ;p == Reproducibility in theory == According to https://tests.reproducible-builds.org/debian/buster/index_suite_amd64_stats.html we have 26476 source packages (92.8%) which can be built reproducibly in buster/amd64, out of 28523 source packages in total. (These 28523 source packages build 57448 binary packages.) But these tests are done without looking at the actual .deb files distributed from ftp.debian.org (and we always knew that and pointed it out: "93% reproducible _in our current test framework_".) == Looking at binary packages Debian actually distributes == So, Vagrant came up with an idea [1] to check buildinfo.debian.net for .deb files for which 2 or more .buildinfo exist (where "exist" means that the .deb files sha1sum is listed in the .buildinfo file) and I turned that into a jenkins job doing this check for all 57448 binary packages in amd64/buster/main (incl downloading all those .deb files from ftp.d.o). The current main results (from this job [2]) are: reproducible packages in buster/amd64: 30885: (53.7600%) unreproducible packages in buster/amd64: 26543: (46.2000%) and reproducible binNMUs in buster/amd64: 0: (0%) unreproducible binNMU in buster/amd64: 7423: (12.9200%) == why are binNMUs unreproducible? == Because of their design, binNMUs are unreproducible, see #894441 [3] for the details (in short: binNMUs are not what they are ment to be: the source is changed and thrown away) and our proposed solution: 'binNMUs should be replaced by easy "no-change-except-debian/changelog-uploads'. So that accounts for 12%, but 12% are not enough to explain the difference between 54% and 93%... == packages which have not been rebuilt since December 2016 == And today I remember a thread I started last year in May, titled "packages which have not been rebuilt since December 2016" [4] (because these packages were build with an old dpkg not producing .buildinfo files, which Chris turned into #900837 [5] "release.debian.org: Mass-rebuild of packages for reproducible builds" and so today I ran Chris' script [6] again on coccia.d.o, and today it showed that 'only' 6804 source packages need a rebuild (compared to 9192 eight months ago). 6804 of of 28523 is 23.9%. And 54%+12%+24% equals 90%. Bingo. Bummer. (While #900837 was only filed in 2018 we knew about this issue since 2015 or so... probably earlier. Sigh.) == After the release is before the release. == So, as we first need to fix #894441 before we can sensibly fix #900837 and because Buster is practically frozen, I think we can just conclude that Buster is quite reproducible in theory (similar but better than Stretch...) and that we need to make sure to address #894441 ASAP, which means for Bullseye, the release after Buster. Fur future reference, a summary of the current status of Debian's reproducibiliy is available at https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues [7] Happy hacking and many many thanks to everyone who has contributed so far! [1] https://lists.reproducible-builds.org/pipermail/rb-general/2018-October/001239.html [2] https://jenkins.debian.net/job/reproducible_compare_Debian_sha1sums/103/console [3] https://bugs.debian.org/894441 [4] https://lists.debian.org/debian-devel/2018/05/msg00499.html [5] https://bugs.debian.org/900837 [6] https://lists.debian.org/debian-devel/2018/06/msg00007.html [7] https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues -- tschau, Holger ------------------------------------------------------------------------------- holger@(debian|reproducible-builds|layer-acht).org PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C