Good morning Rustaceans! Hot off the heels of the Rust All Hands in Berlin last week we’ve had numerous discussions about the stabilization of Macros 2.0 and what we can have ready for the Rust 2018 release. I’d like to detail what we discussed here as well as solicit for your help in stabilizing a “Macros 1.2” in the upcoming 1.27 release possibly.

All-hands recap

In Berlin we had a session specifically about Macros 2.0 and what, if any, we could stabilize for the edition release. While “Macros 1.1” is stable today it doesn’t include features like:

Macros through attributes (aka #[my_proc_macro] )

) Custom bang-style macros (aka my_proc_macro! )

) Span information, everything is lost today with to_string()

Rich diagnositics beyond panic!

Hygiene information to avoid name collisions programmatically instead of “hopefully” via __foo

These features have gone through a number of RFCs to date and have seen quite extensive testing throughout the community. In our discussion, however, we specifically ruled out two large features for the Rust 2018 release:

Declarative macros (aka macro macros) Hygiene

Both of these features were deemed as too high risk to stabilize at this point, but fear not as Macros 2.0 is large and has many other features internally! We came up with a design (which I like to call “Macros 1.2”) which is aimed at stabilizing a subset of Macros 2.0 functionality which is sort of the 80% of what crates use today.

In summary, we concluded that the following subset is possible to stabilize for the Rust 2018 release:

Importing macros through the module system

Attribute macros, only attached to items but not modules.

Custom bang-style macros, but only invoked in module contexts, not as expressions or inside functions.

No hygiene information through explicitly requesting “call site” hygiene, aka copy/pasting code.

A large chunk of the proc_macro API to enable preserving span information on tokens.

The subset above we concluded was imminently stabilizable with no major open questions (unlike hygiene and declarative macros). And with that, let’s dive into these pieces!

Macros and the module system

The first item on this list is the ability to import procedural and other macros through the module system (aka use ) rather than #[macro_use] . This dovetails very nicely with the edition’s attempt to remove the need for extern crate and is in general much nicer to use than #[macro_use] as well!

The tracking issue for this issue has now entered FCP for stabilization and the main takeaways are:

You’ll be able to import attributes via use as well as procedural bang-style macros

as well as procedural bang-style macros You’ll also be able to import Macros 1.0-style macros (aka macro_rules! )

) Importing 1.0-style macros is unhygienic, so if the macro invokes another macro you’ll have to import that as well

You won’t use use to import macros within your own crate but will rather instead rely on today’s scoping rules.

While not a perfect stabilization the warts are specifically related to 1.0-style macros which are long-term going to be phased out with macro macros. In the meantime the experience of using 1.0-style macros changes very little and we’re mostly just rationalizing use with the backwards-compatibility of today’s macro system.

Macros and items

We discussed this at the work week and felt that the only part of invoking a macros 2.0 macro was in the item position like a function attribute or an invocation inside of a module scope. This appeared to cover most use cases of procedural macros in the majority.

Concerns related to hygiene were brought up related to expression macros (or those that could expand to expressions). I’m not personally too privvy on these details but @nrc may be able to fill in more if others have questions about this! I think the general gist though is that “copy paste hygiene” is relatively straightforward for items where there’s already not a huge amount of hygiene in Macros 1.0, but expression copy/paste hygiene is tricker.

Lack of hygiene information

We universally agreed that hygiene was a good thing we’d like to stabilize one day. We basically all agreed as well though that it’s realy hard to stabilize hygiene and the compiler is definitely not ready to do so. As a result we wanted to stabilize a forward-compatible solution with hygiene but still not actually stabilize anything hygienic today.

In rustc hygiene information today runs through Span , where each Span indicates what syntactical context it came from as well as the literal bytes in the original source it corresponds to. This is expressed as well in the proc_macro API with Span having constructors like call_site (copy/paste hygiene) and def_site (not accessible by expanded code hygiene).

Our conclusion here was that we would only stabilize the Span::call_site() function and no others. This way if you ever manufactured a span you’d be explicitly opting-in to copy/paste call-site hygiene. Eventually once we stabilize hygiene it’ll either be through spans or a separate field, and this way we should have avenues for inserting hygiene information, if necessary.

This is also highly related to…

proc_macro API changes

And finally the last piece of stabilization that would be required here would be the proc_macro API itself. @dtolnay, @nrc, and I took a long hard look at the current API with an eye towards stabilization and came up with a reorganized API which is now available in today’s nightly! The crate should provide everything it did before, only in a reorganized and more forward-compatible and ergonomic fashion. The main changes are:

The TokenNode enum has been removed and the public fields of TokenTree have also been removed. Instead the TokenTree type is now a public enum (what TokenNode was) and each variant is an opaque struct which internally contains Span information. This makes the various tokens a bit more consistent, require fewer wrappers, and otherwise provides good future-compatibility as opaque structs are easy to modify later on.

Literal integer constructors have been expanded to be unambiguous as to what they’re doing and also allow for more future flexibility. Previously constructors like Literal::float and Literal::integer were used to create unsuffixed literals and the concrete methods like Literal::i32 would create a suffixed token. This wasn’t immediately clear to all users (the suffixed/unsuffixed aspect) and having one constructor for unsuffixed literals required us to pick a largest type which may not always be true. To fix these issues all constructors are now of the form Literal::i32_unsuffixed or Literal::i32_suffixed (for all integral types). This should allow future compatibility as well as being immediately clear what’s suffixed and what isn’t.

Each variant of TokenTree internally contains a Span which can also be configured via set_span . For example Literal and Term now both internally contain a Span rather than having it stored in an auxiliary location.

Constructors of all tokens are called new now (aka Term::intern is gone) and most do not take spans. Manufactured tokens typically don’t have a fresh span to go with them and the span is purely used for error-reporting except the span for Term , which currently affects hygiene. The default spans for all these constructed tokens is Span::call_site() for now. The Term type’s constructor explicitly requires passing in a Span to provide future-proofing against possible hygiene changes. It’s intended that a first pass of stabilization will likely only stabilize Span::call_site() which is an explicit opt-in for “I would like no hygiene here please”. The intention here is to make this explicit in procedural macros to be forwards-compatible with a hygiene-specifying solution.

These changes are all reflecting in proc-macro2 , syn , and nightly itself as of today.

Final push for stabilization

My hope is that this is the final cut of the proc_maco API before stabilization. If all goes well I’d like to FCP this subset of macros 2.0 towards the end of the current nightly cycle (1.27) and then stabilize for the 1.27 release. If discussions show that we’re not quite ready for stabilization then we will definitely switch to a different schedule.

I think that this subset is going to enable quite a few use cases seen in the wild today to work on stable Rust while also providing a solid base for us to extend and grow with hygiene/diagnostic/etc information in the future. In the meantime though we need your help in testing out the new APIs. If you’ve got a procedural macro crate or custom derive consider updating to syn 0.13 or proc-macro2 0.3 and give us you’re feedback! We’re particularly interested in bugs and ergonomic issues with the current API.

Here’s to hopefully stabilizing Macros 1.2 in Rust 1.27.0!