Preamble

I was thinking about plugins and serde and it occurred to me that there are three types of common plugins, with different interactions with stability and development workflow:

Lints. These are not necessary for compilation; but are a tool to help the developer. It would be very hard to stabilize these.

Internal syntax extensions: These are syntax extensions used internally to make code nice. For example, the syntax extensions defined in html5ever are for usage within the library. These could be replaced by things like Mako templates if necessary. They are not a core feature of the library; technically the generated code is all that is necessary.

Exported syntax extensions: These are exported by libraries to make life easier for other libraries. These are a core feature of the library. phf is an example of this. However, libraries which use phf technically only need the generated code, in most cases. For some libraries like [string-cache][1], the macros are used multiple libraries downstream too, though in the case of string-cache there are non-macro alternatives to them.

This proposal makes it possible to use libraries with internal syntax extensions (and libraries which transitively use external syntax extensions) on stable Rust builds. In other words, if the dependency graph is like:

plugin P <- library A <- library B <- binary C

And only A uses the plugin, it will be possible to compile B and C with stable Rust. but not P or A . At the same time, those wishing to use nightlies can do so without any change in experience.

Currently, we have [syntex][2] by the brilliant @erickt which offers a solution by doing expansion manually with a libsyntax clone. However, it’s a bit hard to set stuff up so that a library may compile with or without syntax extensions depending on the user’s choice, and additionally it sort of cheats semver – plugin libraries will continue to be flaky and unstable with syntex.

This (pre) RfC takes inspiration from the idea of syntex and makes it into a baked in, easy to use system which doesn’t cause semver problems.

RfC

Overview

At its core, we basically allow users to upload two versions of the same crate. One is a “stabilized” crate which has all of the syntax extensions expanded and is checked for usage of any unstable features. When someone else decides to use the library and wishes to use stable Rust, this expanded copy of the library will be downloaded instead, and the downloaded Cargo.toml will be stripped of any plugin deps.

This only applies to libraries on Crates. While the infra here could be extended to support git deps (using branches), I’m limiting the scope of this RfC to Crates for now.

Detailed design

####Changes to rustc

Rustc nightly should get a -Z as-stable option (IIRC this is already sort of possible with the right cfg flags) which makes a nightly compiler behave as if it was on the stable channel. This makes feature gates hard errors.

For full flexibility, it would be nice if rustc could have a more nuanced expansion mode where only syntax extensions not defined by rustc itself are expanded. This means that macros still work, and macro stability still works. (otherwise macros with unstable innards will fail) Complex nesting might still be broken though.

An additional wishlist item would be for that expansion mode to expand file-by-file to maintain the file structure as much as possible.

Also if it would be possible to do readable gensym-ing when necessary.

Changes to Crates.io

Crates can now have a “stabilized” version in addition to the regular one. It is encouraged to upload “stabilized” versions even if your library does not use syntax extensions, so that we can add some visual indication “compiles on stable Rust!” later.

“stabilized” crates should only be different from regular ones as far as their build process goes (one uses syntax extensions, one doesn’t, but the post-expansion code is the same). I think that further code fragmentation for providing “stable” and “nightly” versions of a library should be done via feature flags.

Changes to cargo

cargo build and cargo publish have an optional flag --stabilize . This will recompile from scratch, and do the following things:

Except for libraries with plugin = true , compile everything with -Z as-stable . Fail if a dependency needs unstable Rust.

, compile everything with . Fail if a dependency needs unstable Rust. For all non-plugin libraries from Crates, fetch and build “stabilized” versions if possible. If a library doesn’t have a stabilized version and fails to build with -Z as-stable , bail. It’s up to the library owner to publish stabilized crates. Existing stable libraries won’t be affected by this since they will build fine with -Z as-stable

, bail. It’s up to the library owner to publish stabilized crates. Existing stable libraries won’t be affected by this since they will build fine with For all plugin=true crates which are direct dependencies of the crate being built or any of the path deps (i.e., deps which are part of the repo and will be in the same bundle), compile with nightly.

crates which are direct dependencies of the crate being built or any of the path deps (i.e., deps which are part of the repo and will be in the same bundle), compile with nightly. Expand the repo and all path deps with --pretty=expanded (or something better as described in the above section) using a nightly.

(or something better as described in the above section) using a nightly. Compile these crates with -Z as-stable .

. If we are publishing, upload the expanded version of the crates.

--stabilize is mainly an option for the publishers of crates which use syntax extensions (eg the authors of library A in the dependency chain above). It’s only part of cargo build so that things can be tested locally without causing a premature publish.

If cargo build / cargo publish are called with a stable compiler or with, they should build by fetching stabilized versions whenever possible, and upload the package as a “stabilized” one. We can of course have a --no-stabilized option to both to opt out of this, and a --with-stabilized option for cargo build that opts in to this when the compiler is nightly. This way nightly users can compile the libraries with regular expansion info, instead of having to debug generated code when something goes wrong.

These flags are intended for users of libraries which use syntax extensions; eg the authors of library B and binary C in the chain above.

It’s also worth considering if cargo publish by default should do both a nightly and stable publish when a nightly compiler is available, unless it’s configured or told otherwise.

What this means for users

Plugin writers

Carry on. In case of hygeine issues, macro wrappers may be necessary.

For plugins like atom!() which are intended to be used by all child libraries, provide a non-plugin interface.

Library writers

Use --stabilized whenever possible. If your crate uses syntax extensions, try to ensure that they’re hygenic and publish both nightly and stabilized versions

Library/binary users

If you like nightly, use Cargo with the appropriate flags to fetch libraries full of syntax extension goodness. If you like stable, use cargo with the appropriate flags to fetch stabilized versions.

(One of these two will be the default; I don’t know which yet.)

Unresolved questions

What should the defaults for Cargo be?

Should/can we do this in a way that macro stability isn’t affected? Regular --pretty=expanded will produce code which won’t compile when there’s internal macro stability involved. If we can selectively expand things (and don’t expand external syntax extensions who have macros fed to them as input), we can cover a broad range of edge cases I think.

will produce code which won’t compile when there’s internal macro stability involved. If we can selectively expand things (and don’t expand external syntax extensions who have macros fed to them as input), we can cover a broad range of edge cases I think. How do we handle hygeine?

Alternatives

Continue using syntex. It’s not perfect, but it does accomplish the job pretty nicely.

Avoid syntax extensions till they get stabilized (which could be a while)

cc @erickt @kmcallister @eddyb @alexcrichton @sfackler @nrc

[1]: https://github.com/servo/string-cache/ [2]: https://github.com/erickt/rust-syntex