Grant Proposal: RakuAST

# Project title RakuAST # Project synopsis I recently presented a proposal for RakuAST ([slides](http://jnthn.net/papers/2020-gpw-realizing-raku-macros.pdf), [video](http://jnthn.net/papers/2020-gpw-realizing-raku-macros.mp4)), an abstract syntax tree for the Raku language that will become part of the language specification. I have also released my work on it so far, which is in a [branch](https://github.com/rakudo/rakudo/tree/rakuast) in the Rakudo repository. So far, I've worked on it during spare moments. The purpose of this grant is to accelerate progress by enabling me to spend more of my working time on it. While I already am funded through an ongoing Performance and Reliability grant - which I shall continue to work on - the RakuAST work does not naturally fit within its scope, thus this grant application. # Your name Jonathan Worthington # A discussion on the benefits of the project to the Perl Community An AST can be thought of as a document object model for a programming language. The goal of RakuAST is to provide an AST that is part of the Raku language specification, and thus can be relied upon by the language user. Such an AST is a prerequisite for a useful implementation of macros that actually solve practical problems, but also offers further powerful opportunities for the module developer. For example: * Modules that use Raku as a translation target (for example, `ECMA262Regex`, a dependency of `JSON::Schema`) can produce a tree representation to `EVAL` rather than a string. This is more efficient, more secure, and more robust. (In the standard library, this could also be used to realize a more efficient `sprintf` implementation.) * A web framework such as `Cro` could obtain program elements involved in validation, and translate a typical subset of them into JavaScript (or patterns for the HTML5 `pattern` attribute) to provide client side validation automatically. RakuAST will also become the initial internal representation of Raku programs used by Rakudo itself. That in turn gives an opportunity to improve the compiler. The frontend compiler architecture of Rakudo has changed little in the last 10 years. Naturally, those working on it have learned a few things in that time, and implementing RakuAST provides a chance to fold those learnings into the compiler. Better static optimization, use of parallel processing in the compiler, and improvements to memory and time efficiency are all quite reasonable expectations. However, before many of those benefits can be realized, the work of designing and implementing RakuAST, such that the object model covers the entire semantic and declarational space of the language, must take place. This grant focuses on that work. # A list of deliverables 1. Class and role implementations defining an document object model for the Raku language, including both the regex and string sub-languages. These should all be constructable and introspectable from within the Raku language. 2. Generation of QAST, the backend-independent intermediate representation, from RakuAST nodes, such that one can execute an AST. This functionality is exposed as an overload of `EVAL`, since it's ultimately the same operation; the RakuAST form of `EVAL` can skip the parsing stage entirely, since it already has the AST. 3. Tests that cover `EVAL` of RakuAST. Every node should be test covered. 4. Integration of RakuAST into the compilation process. This functionality can be behind an environment variable rather being the default mode of compilation; this is to avoid a situation where there is pressure to enable it by default in order to consider the grant complete, when it would serve Raku users better to make the switch when it is really ready for that. However, it should be good enough to pass at least 90% of the test suite on the MoarVM backend. # Project details and a proposed schedule In *May* I would continue work on the design and implementation of the AST to cover more of the Raku language. I'll continue to work test-first and implement compilation of nodes in order they can be `EVAL`'d and produce the required meta-objects. The goal is to cover a broad range of the main language. In *June* I would start work on a means to get the compiler frontend to use RakuAST, most probably enabled by an environment variable. This need not deal with such concerns as `BEGIN`, module usage, or precompilation, but it should be possible to run various simple scripts in this mode successfully. In *July* I would take on modeling the regex and string sub-languages in RakuAST, fill out most remaining gaps in the main language, and integrate these into the compiler frontend as I go. In *August*, I would focus on some of the more advanced integration with the compiler frontend, including module usage, `BEGIN`, `constant`, and making sure any problems with the interaction of RakuAST and precompilation are taken care of. This should open the way to passing a significant portion of the specification test suite in RakuAST mode. # Your biography I am the founder and architect of MoarVM, the virtual machine used by the majority of Raku users, and architect of the Rakudo compiler. I led the previous significant overhaul of the Rakudo compiler, which took place around a decade ago. I work at Edument, where my roles include product lead for the Comma IDE for Raku. My experience designing the program representation there is highly relevant for the work on RakuAST. I hold a degree in computer science from the University of Cambridge, and aside from my work on Raku have participated in a range of commercial projects in the compiler and developer tooling space, including in an architectural capacity. # The requested amount for the Project Grant, preferably in US dollars. $8,500 USD

Comments (6)

Yes please, this sounds great!

yes please! sounds like a very sensible approach to the problem

I'm highly in favor of funding this grant. Jonathan has demonstrated countless times that he can deliver high-quality software in the Rakudo compiler and underlying VM, and that his design ideas tend to work out very well. I also think that a higher-level AST as we have now both helps in terms of performance (because it offers more data for optimization passes to work with) and with advancing macros.

The arguments for ReakuAST both in the grant application and in the video/slides provided are very convincing. Who else if not Jonathan will be able to provide a well designed and implemented solution. The requested funds sound like a bargain.

This feels like a natural next step and a great way of smoothing out the waterbed of complexity between the various layers and internal representations within the Raku 'stack'. jnthn++ has contributed at all levels of the Raku architecture and he is uniquely qualified to balance all the design dimensions needed to successfully deliver this. I'm excited about macro support. I'm also keen to see how the changes can help Raku's speed and memory performance. This work will lay the foundation for these improvements. +1

Aleks-Daniel Jakimenko-Aleksejev wanted to comment, but was unable to post directly: his comment appears below. -- Everything about this grant proposal sounds extremely uplifting. I can already see potential uses for RakuAST from language user's perspective (after all, nowadays I find myself spending more time using Raku than working on Raku itself). If RakuAST gets us closer to working macros then this work is truly invaluable. For some context, Rakudo Star announcements until 2019.03 used to say that “advanced macros” are not quite there yet and will appear in upcoming releases. This was indeed one of the big promises a lot of people were looking forward to. However, the work on macros was so stagnant that in the most recent announcements this mention was removed altogether. I'm happy to know that we will likely be able to get back on track by using RakuAST as one of the building blocks. As for the potentially improved and silghtly revamped Rakudo architecture, that would be great, although I'm a bit worried that this work will add another layer. That said, nowadays we have a lot of opportunities for noticing performance/memory regressions, so I'm sure it'll be possible to hold this work to high standards. And above all I'm extremely happy that it's jnthn out of all who is willing to make this all happen, as I have no doubt that they will be able to get RakuAST project to completion and in the best way possible. I hope this grant proposal is accepted.