Debian Bug report logs - #942898

debcargo: reduce the impact of debcargo-built crates on the package index, and facilitate debian packaging of crates

Reported by: Daniel Kahn Gillmor <dkg@fifthhorseman.net> Date: Wed, 23 Oct 2019 00:39:01 UTC Severity: normal Tags: bullseye, moreinfo, sid, unreproducible, wontfix Merged with 945542 Found in versions rust-debcargo/2.4.0-1, rust-debcargo/2.2.10-1

Reply or subscribe to this bug.

Toggle useless messages

Report forwarded to debian-bugs-dist@lists.debian.org, Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> :

Bug#942898 ; Package debcargo . (Wed, 23 Oct 2019 00:39:04 GMT) (full text, mbox, link).

Acknowledgement sent to Daniel Kahn Gillmor <dkg@fifthhorseman.net> :

New Bug report received and forwarded. Copy sent to Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> . (Wed, 23 Oct 2019 00:39:04 GMT) (full text, mbox, link).

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net> To: submit@bugs.debian.org Subject: debcargo: reduce the impact of debcargo-built crates on the package index, and facilitate debian packaging of crates Date: Tue, 22 Oct 2019 20:34:29 -0400

Package: debcargo Version: 2.4.0-1 Control: affects -1 ftp.debian.org Hi! I'm trying to summarize in this report the state of conversation i had today between members of the FTP team (and others on #debian-ftp) and members the debian Rust packaging team. We seem to be in a bit of an impasse, and i'm hoping that we can use this report to find a concrete way out of the impasse. This report explains my understanding of the problem and proposes one way forward. I'd be very happy for constructive engagement with this report. If you think i'm wrong, help explain what would be better, or why my proposed changes aren't acceptable. Background ---------- debcargo creates .deb packages corresponding to Rust crates. I'm going to focus here on the librust-* .deb binary packages, and ignore the packages that ship tools that have, for example, a command-line interface. Each source package corresponds to a Rust crate, but it might produce multiple librust-* binary packages -- one for each crate "flavor". the librust-*.deb files contain mainly a distribution of .rs source code in /usr/share/cargo/registry/… Most of the "+flavor" .debs basically have an additional source dependency on some other rust crate, and also ship a symlink back to the folder of .rs source code from the standard (non "+flavor") .deb. For each binary package shipped, it also defines a list of Provides: that include version numbers in the name. Problem Statement(s) -------------------- From talking with the FTP team, the concern appears to be that the large number of packages, and the large number of Provides produced for some of the packages is seen as a potential problem for the apt Packages index. In particular, a larger Packages index: * increases the cost of data transfer for every debian system that does updates * means that apt has to do more complicated dependency resolution I have not yet gotten measurements of what kinds of costs we're talking about with respect to this shared resource. If someone could provide some numbers and a methodology for getting them, that would be useful in figuring out whether any proposed change contributes a substantial solution to the problem. Kinds of measurements that might make the risks a bit clearer for leaning on this shared resource too heavily: - size of Packages file - RAM needed by aptitude to ingest the Packages file - CPU time taken by aptitude to do dependency resolution Additionally, from the Rust team's perspective, when a crate upstream proposes a feature, this creates a new binary .deb, which triggers inclusion in the NEW queue. The delay in the NEW queue causes friction which makes packaging Rust-related projects harder to do. I've experienced this myself, where i spend about an hour or two on rust packaging, and then find myself waiting for weeks before i can do the next hour of work -- this isn't conducive to effective maintenance. Proposed Solutions ------------------ AIUI, the FTP team thinks that debcargo could reduce the impact on the shared resource of the Packages index by adopting one or both of the tactics described below: 0) reduce the number of "+feature" .debs produced by each crate, perhaps by creating two base .debs for each package: one with no "features" and one that bundles together all of the features that are not mutually-exclusive. Any features that are mutually exclusive would still get their own separate "+feature" .deb. 1) drop version numbering from the Provides: entries for standard packages -- this should reduce the number of Provides: by a substantial fraction. Given that crates are expected to hew to semantic versioning, a generated version number range should be sufficient to declare an API-compatible version dependency. Concerns -------- A concern I've heard about tactic (0) is that this approach might result in dependency loops, as in: https://github.com/sfackler/cargo-tree/issues/34#issuecomment-394266843 It's conceivable that apt could figure out how to deal with those loops, and given that these librust-*-dev .debs are basically just sourcecode, these loops will only affect Rust developers (who might be expected to have beefier machines than regular end users). Such a change would also reduce the number of packages A concern i've heard about tactic (1) is that it makes it harder to ship a two versions of a crate in a given debian distro. For example, we might want to ship librust-foo-dev version 1.x alongside librust-foo-dev version 2.x. If they have the same package name (librust-foo-dev) then they can't both be in the archive at once. We currently do ship librust-foo-dev (meaning "the latest version"), and we create a new package librust-foo-1-dev for when we intend to maintain a stable API for an older release. I don't think anyone wants that to change (many C library packages use the same pattern based on their SONAMEs). The use of Provides: makes it so that a package can directly Depend: on the package with the API in the name, rather than depending on a versioned range. But perhaps the fix is to make the dependencies marginally more complex, rather than stuffing lots of Provides in each binary package. For example, a package that depends on the foo crate at 1.2 or higher could use: Depend: librust-foo-dev (>= 1.2) (<< 2) || librust-foo-1-dev (>= 1.2) Do the above approaches seem feasible? Would they satisfy both the FTP team and the Rust packaging team? Are they implementable? Sorry that i don't have the capacity to actually implement them myself, i just want to identify a way forward that will make both Debian and the Rust ecosystem more compatible. Regards, --dkg

Added indication that 942898 affects ftp.debian.org Request was from Daniel Kahn Gillmor <dkg@fifthhorseman.net> to submit@bugs.debian.org . (Wed, 23 Oct 2019 00:39:04 GMT) (full text, mbox, link).

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> :

Bug#942898 ; Package debcargo . (Wed, 23 Oct 2019 13:39:05 GMT) (full text, mbox, link).

Acknowledgement sent to Ximin Luo <infinity0@debian.org> :

Extra info received and forwarded to list. Copy sent to Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> . (Wed, 23 Oct 2019 13:39:05 GMT) (full text, mbox, link).

Message #12 received at 942898@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org> To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>, 942898@bugs.debian.org Subject: Re: [Pkg-rust-maintainers] Bug#942898: debcargo: reduce the impact of debcargo-built crates on the package index, and facilitate debian packaging of crates Date: Wed, 23 Oct 2019 13:27:00 +0000

Control: tags -1 moreinfo unreproducible wontfix Daniel Kahn Gillmor: > [..] > > Problem Statement(s) > -------------------- > > From talking with the FTP team, the concern appears to be that the large > number of packages, and the large number of Provides produced for some > of the packages is seen as a potential problem for the apt Packages > index. > > In particular, a larger Packages index: > > * increases the cost of data transfer for every debian system that does > updates > > * means that apt has to do more complicated dependency resolution > > I have not yet gotten measurements of what kinds of costs we're talking > about with respect to this shared resource. If someone could provide > some numbers and a methodology for getting them, that would be useful in > figuring out whether any proposed change contributes a substantial > solution to the problem. Kinds of measurements that might make the > risks a bit clearer for leaning on this shared resource too heavily: > > - size of Packages file > - RAM needed by aptitude to ingest the Packages file > - CPU time taken by aptitude to do dependency resolution > As a user, I haven't noticed Debian slowing down because of rust packages. As a volunteer, I have better things to do with my free time than "fix" a non-problem because someone else thinks something else is too big. If you like to see other people fix this bug, please provide more detailed information that clearly indicates that the current situation in concretely problematic, to convince them to fix it. If you would like to fix this bug yourself, please suggest a concrete solution and we can comment on whether it's feasible/sustainable/maintainable. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git

Added tag(s) moreinfo, wontfix, and unreproducible. Request was from Ximin Luo <infinity0@debian.org> to 942898-submit@bugs.debian.org . (Wed, 23 Oct 2019 13:39:06 GMT) (full text, mbox, link).

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> :

Bug#942898 ; Package debcargo . (Wed, 23 Oct 2019 13:48:02 GMT) (full text, mbox, link).

Acknowledgement sent to Ximin Luo <infinity0@debian.org> :

Extra info received and forwarded to list. Copy sent to Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> . (Wed, 23 Oct 2019 13:48:02 GMT) (full text, mbox, link).

Message #19 received at 942898@bugs.debian.org (full text, mbox, reply):

From: Ximin Luo <infinity0@debian.org> To: Daniel Kahn Gillmor <dkg@fifthhorseman.net>, 942898@bugs.debian.org Subject: Re: [Pkg-rust-maintainers] Bug#942898: debcargo: reduce the impact of debcargo-built crates on the package index, and facilitate debian packaging of crates Date: Wed, 23 Oct 2019 13:45:00 +0000

Daniel Kahn Gillmor: > > Proposed Solutions > ------------------ > > AIUI, the FTP team thinks that debcargo could reduce the impact on the > shared resource of the Packages index by adopting one or both of the > tactics described below: > > 0) reduce the number of "+feature" .debs produced by each crate, > perhaps by creating two base .debs for each package: one with no > "features" and one that bundles together all of the features that > are not mutually-exclusive. Any features that are mutually > exclusive would still get their own separate "+feature" .deb. > Detecting "mutually-exclusive" isn't straightforward as you'd have to detect the cycles we mentioned below. I can't be bothered writing this code, the cost-benefit tradeoff is not worth it. > 1) drop version numbering from the Provides: entries for standard > packages -- this should reduce the number of Provides: by a > substantial fraction. Given that crates are expected to hew to > semantic versioning, a generated version number range should be > sufficient to declare an API-compatible version dependency. > This isn't possible due to #901827. The solution that we're already doing (I'll refer to this as "solution A" for future reference) for exceptionally large crates like web-sys is to just patch out the unused features. This is by far the easiest option that cuts away most of the size, whilst retaining the other benefits of the current automation. So I don't see anything additional to do on this topic. X -- GPG: ed25519/56034877E1F87C35 GPG: rsa4096/1318EFAC5FBBDBCE https://github.com/infinity0/pubkeys.git

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> :

Bug#942898 ; Package debcargo . (Thu, 24 Oct 2019 16:18:06 GMT) (full text, mbox, link).

Acknowledgement sent to Daniel Kahn Gillmor <dkg@fifthhorseman.net> :

Extra info received and forwarded to list. Copy sent to Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> . (Thu, 24 Oct 2019 16:18:06 GMT) (full text, mbox, link).

Message #24 received at 942898@bugs.debian.org (full text, mbox, reply):

From: Daniel Kahn Gillmor <dkg@fifthhorseman.net> To: Ximin Luo <infinity0@debian.org>, 942898@bugs.debian.org Subject: Re: [Pkg-rust-maintainers] Bug#942898: debcargo: reduce the impact of debcargo-built crates on the package index, and facilitate debian packaging of crates Date: Thu, 24 Oct 2019 12:10:27 -0400

On Wed 2019-10-23 13:27:00 +0000, Ximin Luo wrote: > [ dkg wrote: ] >> I have not yet gotten measurements of what kinds of costs we're talking >> about with respect to this shared resource. If someone could provide >> some numbers and a methodology for getting them, that would be useful in >> figuring out whether any proposed change contributes a substantial >> solution to the problem. Kinds of measurements that might make the >> risks a bit clearer for leaning on this shared resource too heavily: >> >> - size of Packages file >> - RAM needed by aptitude to ingest the Packages file >> - CPU time taken by aptitude to do dependency resolution > > As a user, I haven't noticed Debian slowing down because of rust packages. I've certainly noticed that apt takes longer reading the index than it used to take back when i started using debian. And we have much more powerful machines today. Clearly, we can't attribute that to the rust ecosystem specifically, but it's also evident that this is a shared resource and if everyone grabs more of it just because we can, we'll all be worse off. That said, it would be good to have someone take the time to measure these costs specifically. Over on #942893 i've done baseline measurements for the size of the Packages file, and proposed a way to cut that down by 13%. Perhaps someone else who is better with profiling could do some comparable baselines for RAM and CPU time if they want to demonstrate a concrete problem? --dkg

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> :

Bug#942898 ; Package debcargo . (Sun, 09 Aug 2020 21:24:05 GMT) (full text, mbox, link).

Acknowledgement sent to Geert Stappers <stappers@debian.org> :

Extra info received and forwarded to list. Copy sent to Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> . (Sun, 09 Aug 2020 21:24:05 GMT) (full text, mbox, link).

Message #29 received at 942898@bugs.debian.org (full text, mbox, reply):

From: Geert Stappers <stappers@debian.org> To: 945542@bugs.debian.org, 942898@bugs.debian.org Subject: merging two debcargo bug reports Date: Sun, 9 Aug 2020 23:21:45 +0200

Control: merge 945542 942898 In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945542#94 is this text | it's not clear to me how this bug report is supposed to be distinct | from #942898, which already captures some of these details. if the | reporter or maintainer wants to merge them, i think that'd be reasonable That merge is now done. No, I'm not the maintainer of debcargo, I'm a DD that wants Rust packages in Debian. Infact I want a smooth way from Rusts dependency resolve and builder 'cargo' to the Debian archive. I'm fully aware it will take effort both from Rust team and FTP team Links for clicking to the bug reports https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945542 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942898 Regards Geert Stappers -- Silence is hard to parse

Information forwarded to debian-bugs-dist@lists.debian.org, Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> :

Bug#942898 ; Package debcargo . (Sun, 09 Aug 2020 21:42:02 GMT) (full text, mbox, link).

Acknowledgement sent to Geert Stappers <stappers@debian.org> :

Extra info received and forwarded to list. Copy sent to Debian Rust Maintainers <pkg-rust-maintainers@alioth-lists.debian.net> . (Sun, 09 Aug 2020 21:42:02 GMT) (full text, mbox, link).

Message #34 received at 942898@bugs.debian.org (full text, mbox, reply):

From: Geert Stappers <stappers@debian.org> To: 945542@bugs.debian.org, 942898@bugs.debian.org Subject: merging two debcargo bug reports that effect ftp.debian.org Date: Sun, 9 Aug 2020 23:39:01 +0200

Control: affects 945542 ftp.debian.org Control: merge 945542 942898 On Sun, Aug 09, 2020 at 09:24:05PM +0000, Debian Bug Tracking System wrote: > Processing control commands: > > > merge 945542 942898 > Bug #945542 [debcargo] Rust packages add and remove binary packages > Unable to merge bugs because: > affects of #942898 is 'ftp.debian.org' not '' > Failed to merge 945542: Did not alter merged bugs.

Marked as found in versions rust-debcargo/2.2.10-1. Request was from Geert Stappers <stappers@debian.org> to 942898-submit@bugs.debian.org . (Sun, 09 Aug 2020 21:42:03 GMT) (full text, mbox, link).

Added tag(s) sid and bullseye. Request was from Geert Stappers <stappers@debian.org> to 942898-submit@bugs.debian.org . (Sun, 09 Aug 2020 21:42:04 GMT) (full text, mbox, link).

Merged 942898 945542 Request was from Geert Stappers <stappers@debian.org> to 942898-submit@bugs.debian.org . (Sun, 09 Aug 2020 21:42:05 GMT) (full text, mbox, link).

Merged 942898 945542 Request was from Geert Stappers <stappers@debian.org> to 945542-submit@bugs.debian.org . (Sun, 09 Aug 2020 21:42:08 GMT) (full text, mbox, link).

Send a report that this bug log contains spam.