Grant Proposal: Future::AsyncAwait

The Grants Committee has received the following grant proposal for the Sep/Oct round. Before the Committee members vote, we would like to solicit feedback from the Perl community on the proposal. **Update: You have until October 17th!** Review the proposal below and please comment here by October 10th, 2018. The Committee members will start the voting process following that. # Future::AsyncAwait - Name: Paul Evans - Amount Requested: GBP 4,800 (GBP 200/day for 24 days) At time of writing, equivalent to USD 6,256 or EUR 5,388. ## Synopsis Fix and polish the implementation of the [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) CPAN module by resolving the bugs that currently prevent it from being useful. ## Benefits to the Perl Community This CPAN module implements the `async/await` syntax, which has emerged as a common standard among various other languages. At the time of writing this proposal, C#, Python 3, ECMAScript 6 (JavaScript), Dart, and Rust all implement this syntax. It centres around the use of "futures" or "promises" as containers of values to be provided by some possibly-asynchronous or background work, and greatly improves on the readability of programs written using them. Logical flow reads very similarly to standard synchronous call-style. The examples in the CPAN module should help to illustrate the readability advantages, as do other examples written in the other languages mentioned above. There can be little doubt that `async/await` is gaining traction as the standard syntax for writing control-flow around future-based concurrency across many diverse programming languages. By using this particular syntax, Perl 5 gains the same semantics with the same syntax as used by these other languages. This helps readers to use the same concepts as they may already be familiar with from using those languages. ## Deliverables The primary deliverable would be a newer version of [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) whose internal implementation is robust and fully-functional in a variety of test cases and conditions. A number of known bugs already exist and are collected on RT (at [https://rt.cpan.org/Dist/Display.html?Queue=Future-AsyncAwait](https://rt.cpan.org/Dist/Display.html?Queue=Future-AsyncAwait)). Not all of the tickets in this queue are necessary for completion of this project, as some are "wish-list" items or design discussions about possible further extensions or additions. In addition to this CPAN release, a secondary deliverable will be a set of blog posts and presentations to educate potential users - bringing the new syntax to their attention, and demonstrating its use. Some blog posts may draw contrasts and similarities to other languages, helping to further emphasize the close relationship between the syntax across these languages. In particular, parallels to Python 3 and ECMAScript 6 may be useful, as both of those languages are ones that potential users may also be familiar with. ## Project Details In its current state, the CPAN module implements a perl parser plugin that parses the newly-defined keywords, and provides implementation semantics that work in many simple test cases. The distribution contains a selection of unit test files that demonstrate these cases. However, the implementation of the suspend/resume functionality behind the keywords is as yet insufficient for many larger, real-world applications to use in existing code. This is due to the incomplete understanding of some of the internal details of the `perl` interpreter, which causes some situations to end up in a mismatched state, making it crash. Central to this project's success is performing further research and understanding on these internals. Once the operation of the interpreter is better understood, the shortcomings of the current implementation of [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) can be identified and improved upon. This may yield further details that need to be understood which have yet to come to light, but with any luck the process will eventually terminate yielding a new syntax extension that is robust and useful. ## Inch-stones Initial progress will be made from a better understanding of certain details of the `perl` interpreter, primarily on the subject of the interaction between `JMPENV_PUSH` and `PL_top_env`. A discussion of the topic can be seen here: [https://www.nntp.perl.org/group/perl.perl5.porters/2018/08/msg251864.html](https://www.nntp.perl.org/group/perl.perl5.porters/2018/08/msg251864.html). It is plausible there may be other subjects that arise once this one is fixed, so the project may continue to switch between research and implementation phases, as bugfixes uncover new areas that don't yet work. There are particular bugs on the RT queue that need fixing; these are: - [https://rt.cpan.org/Ticket/Display.html?id=126037](https://rt.cpan.org/Ticket/Display.html?id=126037) - [https://rt.cpan.org/Ticket/Display.html?id=126036](https://rt.cpan.org/Ticket/Display.html?id=126036) - [https://rt.cpan.org/Ticket/Display.html?id=125613](https://rt.cpan.org/Ticket/Display.html?id=125613) - [https://rt.cpan.org/Ticket/Display.html?id=123062](https://rt.cpan.org/Ticket/Display.html?id=123062) Additionally to these, to support the secondary goals of developer education, a number of blog posts can be written: - A general introduction to the `async/await` syntax as provided by [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait). Suitable for all Perl audiences. - A comparison between [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) and similar syntax provided by Python 3. This will serve as a useful comparison between the two languages, and may help to draw some existing Python developers. Where possible it should include some "Perlish rewrites" of some standard Python documentation, to drive home that similarity. - A comparison between [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) and ECMAScript 6, in similar style to above, for similar reasons. Thirdly, a presentation for a Perl conference can be prepared to demonstrate the new language feature - perhaps as a follow-up to the talk I did for TPCiA in 2017 ([https://www.youtube.com/watch?v=Xf7rStpNaT0](https://www.youtube.com/watch?v=Xf7rStpNaT0)), except this time to report on some real-world success stories of the feature being used in production cases. Finally, the details of the perl interpreter discovered along the way can be better documented, so that even if ultimately it proves impossible to fix the implementation of [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait), at least others can benefit from the new understanding discovered in that research, which may be helpful to other similar projects. ## Project Schedule There are two main phases to this project: research into the existing operation of the `perl` interpreter, and fixing the existing implementation of suspend/resume semantics as a result of the discoveries made. The first phase - research - should be achievable with around 10 days of effort through a combination of code review and instrumented builds, resulting in a textual description of the relevant "moving parts" of the interpreter. This includes time to write down documentation of the discoveries. Based on this understanding, the second phase - fixing the implementation - can take place. This is likely to take a further 10 days including building more unit test files to cover the new test cases. In addition, the three blog posts and conference presentation will take around 4 days to prepare. This brings the total to 24 days. I'm currently a self-employed contractor with approximately three weeks each month taken by existing clients. Working at a rate of 5 days per month, I believe the project will come to a conclusion after around 5 months. I intend to work at a rate of 5 days per month so regular checkpoints can be established. ## Completeness Criteria At time of writing there are no known CPAN modules able to use [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait) because it is too unstable at present for even the smallest of unit-tests in other code to successfully pass. A useful moment to consider as a completion target may be when the module is sufficiently robust that it can be used as a dependency by at least a few other CPAN modules that currently use [Future](https://metacpan.org/pod/Future). For example, any of my [Device::Chip](https://metacpan.org/pod/Device::Chip) driver modules, that are currently heavily [Future](https://metacpan.org/pod/Future)-based at present, would greatly benefit using this new syntax. CPAN shows a great number of other modules using [Future](https://metacpan.org/pod/Future) by many authors, and any of these could also be used as test-cases for the robustness of [Future::AsyncAwait](https://metacpan.org/pod/Future::AsyncAwait): [https://metacpan.org/requires/distribution/Future?sort=\[\[2,1\]\]](https://metacpan.org/requires/distribution/Future?sort=[[2,1]]) ## Bio I am Paul Evans, PEVANS on CPAN ([https://metacpan.org/author/PEVANS](https://metacpan.org/author/PEVANS)). I have been a CPAN maintainer for over 10 years, and currently have 155 distributions under my name. Among this set of modules are a number of dual-life core modules - [List::Util](https://metacpan.org/pod/List::Util) and [IO::Socket::IP](https://metacpan.org/pod/IO::Socket::IP) being two that may be among the most heavily-depended upon on CPAN. I have a number of XS-based modules, including some such as [Syntax::Keyword::Try](https://metacpan.org/pod/Syntax::Keyword::Try) that provide keyword plugins to extend the Perl syntax. I am familiar with many parts of the core perl interpreter, and am well-known to many of the perl5-porters group. I maintain a blog on a variety of programming topics, often posting on Perl-related matters. [http://leonerds-code.blogspot.com/search/label/perl](http://leonerds-code.blogspot.com/search/label/perl). I have spoken at most London Perl Workshops in the past few years, and attend (and sometimes talk at) the European occurance of what was formerly called YAPC::EU, most recently called TPCiG. I maintain a YouTube playlist of recordings of talks I have given. [https://www.youtube.com/playlist?list=PL9-uV\_AVx5FOzWJIvpuebmyiIuNd4L7GJ](https://www.youtube.com/playlist?list=PL9-uV_AVx5FOzWJIvpuebmyiIuNd4L7GJ)

Comments (11)

I'd first like to speak to the value of this project, as I see it. As someone that loves Perl but has worked in many polyglot companies, one of the things that I constantly hear is 'how crappy Perl's support for asynchronous tasks' is. This is of course not true, there's several great systems for this on CPAN. However in a way the proliferation is confusing to newcomers to the language and makes people think asynchronous support for Perl is something bolted on, perhaps haphazardly, rather than well integrated into the language. I believe this work will go a long way towards solving that perception problem, as well as being of tremendous value in and of itself. I hope it will become a great point of integration between the various existing systems and move us closer together as a community. I am sure it will make it easier for people working in other languages to come to Perl, since they will already be familiar with the syntax and it will make it easier for those already working in Perl to take advantage of writing better asynchronous code, which is rapidly becoming a 'must have' in the programming world. In short, this is not only something I think Perl must have on CPAN, but also something we should consider for core integration as soon as possible.



Secondly, I want to speak to the ability of Paul Evans to complete this work. I've never worked with Paul on a job, but I've seen him over the years on IRC helping people and slowly but surely building up a fantastic portfolio of work on CPAN. He's been working in and with Futures and asynchronous code for many years. Additionally he has the trust and respect of some of Perl's most well known programers. I feel very confident in his estimations and ability.



In short, this is work that needs to be done and Paul Evans has my confidence! Thanks for taking the time to read my comments; feel free to follow up with me on any points raised should questions arise.



John Napiorkowski (JJNAPIORK)



+1, async/await is an extremely useful construct which is widely used in other languages. I look forward to production-ready version of Future::AsyncAwait and I certainly will use it in my code.



IMHO funding its development would be a great use of TPF's money.



This is awesome work that Paul has done so far - well worth the funding!



This would be a killer feature for Perl5, bringing much-needed parity to the existing async module support.



If it also shines a light on some of the murkier parts of the Perl internals, even better.



I'd also mention Paul's technical posts such as https://leonerds-code.blogspot.com/2018/09/develmat-investigation-into-c-part-3.html as evidence that he can manage a good publicly-available writeup as well as putting the code together in the first place.



Nearly every one of the CPAN distributions I'm personally developing would be simplified by the existence of this module. As such, I'm biased but hugely in favour.



(disclaimer: we are one of the existing clients that Paul mentions, and once this is stable we have every intention of using this module heavily in both our internal and public CPAN code)



+1 to this proposal! Having async/await in perl5 would be awesome!



I strongly support this grant request. Paul does great work and is highly responsive and communicative. This project will greatly benefit the community and Perl as a whole.



Folks, apologies; I think I accidentally deleted 2 real comments when removing the 700+ spam entries that were posted in the past week.



All queued comments have been posted as of this time; if your comment is missing, please re-add it.



Regards.



Async/Await would be a great addition to the language.

This should get funded.



After finally getting my head around promises in JavaScript it has been a pleasure to be able to use Promises in Perl via https://metacpan.org/pod/Mojo::Promise. Doing the same with async/await will also help me convince other developers at $work that Perl isn't as terrible as the 10+ year old non-strict-single-file-Perl code that they've been hacking on for the last 5 years.



First of all, I want to highly endorse this work and Paul's efforts to do it. This is an important project that Perl sorely needs. That said I do want to ask a few minor conditions on the project to support community funding of it.



This mostly revolves around Paul's use of his own Future stack to accomplish the behavior. This is understandable as with many authors we like to use our own stuff. The problem with Future is that it isn't entirely interoperable with other Promise implementations on CPAN. (Note that he and I have discussed this amicably on #mojo). There does exist a standard for Promise implementations across languages called the Promises/A+ spec (https://promisesaplus.com/). Future does not conform with that standard (fair, as it predates it) and due to some non-standard behavior as a result it isn't entirely easy for other implementations to conform to his. According to him, there are several methods that must be implemented in the Future way in order for the proposed AsyncAwait mechanism to work. I would ask that if the community were to support his work (and again, I hope we do) that he be asked to follow the Promises/A+ spec in doing so.



If that cannot be reasonably asked for in the time or support amount requested, then I could ask a lesser goal of having him expose his suspend/resume mechanism, the low level guts, in some higher level way so that other Async/Await mechanisms may be built on top of it. Preferably this would be at the perl level though if it must be in the XS level, I won't object too strongly as long as it is usable.



Once again, I want to reiterate that I want to see this grant supported, I just hope that the result is as broadly applicable as possible.









I would ask that if the community were to support his work (and again, I hope we do) that he be asked to follow the Promises/A+ spec in doing so.





We are, with some differences, following the API spec called "Promise/A" (and the clarification that is called "Promise/A+") which was created by the Node.JS community. This is, for the most part, the same API that is implemented in the latest jQuery and in the YUI Deferred plug-in (though some purists argue that they both go it wrong, google it if you care). We differ in some respects to this spec, mostly because Perl idioms and best practices are not the same as Javascript idioms and best practices. However, the one important difference that should be noted is that "Promise/A+" strongly suggests that the callbacks given to then should be run asynchronously (meaning in the next turn of the event loop). We do not do this by default, because doing so would bind us to a given event loop implementation, which we very much want to avoid.





Future::Q - a Perl implementation of http://documentup.com/kriskowal/q/ which is (claimed to be) Promises/A+-compliant



AnyEvent::Promise - the then method is only documented to support a single callback



AnyEvent::Promises - callbacks can't return AnyEvent::Promises instances directly, since the chaining behaviour means it'll wait for those to be resolved/rejected



Evo::Promise::* - mostly builds on other things, but having several different classes here makes it a bit hard to extract a common API



That seems like a reasonable request, but I don't think it'd be as effective as it might first sound.Firstly, I don't believe this is directly possible in Perl without a better definition of https://es5.github.io/#x10.3, which is required by https://promisesaplus.com/#point-34. The footnote brings in a requirement for an "event loop", which is an implementation detail - Future.pm itself does not require this, and a standard "event loop" or "event loop interface" seems a bit out of scope - look at how much time and effort has gone into getting "a MOP into core"!Given the existence of modules such as Mojo::Promise::Role::Futurify it seems more likely that Promises/A+-compatible implementations can be built on top of Future.pm rather than the other way around? There are several features of Future.pm - for example, support for a distinct "cancelled" state - that are a superset of the Promises/A+ spec. If it's not possible to build the compatibility layer on top of Future.pm, maybe it'd help to have a clear set of requirements that cannot be fulfilled by the current proposal?This section of the Promises.pm documentation also seems relevant:There are a few other Future/Promises implementations in Perl:Lastly, the Promises/A+ spec is popular in the JS/ES6 world, but C#'s Task/TaskCompletion is a different beast, as is the shocking mess of C++'s std::future. Python3's equivalent is a superset, but is at least compatible. Scala (and Java) also do their own thing: https://docs.scala-lang.org/overviews/core/futures.html (disclaimer: the majority of my CPAN modules have been based on Future.pm for years, so I know that API far better than the other options)