blog | oilshell.org

Review of Roadmap #3

Yesterday I glossed over a dozen or so blog posts, clearing some mental space for coding.

I want to write a Roadmap #4 to describe what I expect to implement in the next few months. But let's first review November's Roadmap #3. In that post, I stated two priorities:

Test the Oil Language Design by automatically translating real programs to it. Fill out the Shell Runtime. I talked about publishing the spec test results as HTML and accepting code contributions.

I stated a third priority in this status update:

Serialize the AST to a Binary Format for execution in C++.

I went into even more detail in Roadmap #3 Continued, listing tasks in these categories:

Ongoing Work Short-Term Plans Medium-Term Plans Long-Term Plans

How Did We Do?

Surprisingly, I've mostly followed the plan. The biggest divergence is that I thought I would translate OSH to Oil by simple printing the OSH AST used for execution to source code.

That didn't happen because it was naive — we need multiple representations. I developed ASDL to specify these representations, and changed the AST to an LST.

Here's what I did with respect to each of the top priorities:

I wrote code to automatically translate OSH to Oil. Although it's only about 25% done, I think the approach is sound. I showed translations of real programs in Translating Shell to Oil (part two). In the last few days, I wrote code to publish the spec tests as HTML. This will make it easier for others to help me finish the shell runtime in Python. I'll write about it shortly. I implemented and tested the OHeap file format and the associated C++ code generator. This allows us to write a tree-walking interpreter in C++.

Now let's review the detailed task list:

1. Ongoing Work

This work is straightforward but not urgent, so not much happened:

I didn't polish the error messages, but the Lossless Syntax Tree refined the data structures for tracking source code locations , which are required for good error messages.

, which are required for good error messages. I didn't implement many remaining shell features. As mentioned, I hope the spec tests will allow this work to be parallelized.

My writing has been directed at the blog and e-mail correpondence, rather than publishing Oil documentation.

2. Short Term Work

Mostly done:

3. Medium Term Work.

I sketched different strategies for porting various components to C++, e.g. using re2c to implement the lexer.

I haven't yet done that, but I did implement a C++ code generator for core/id_kind.py as part of the diversion on Bespoke Superoptimization.

And, as mentioned, the C++ API for OHeap is tested and ready to use.

4. Long Term Work

This part is unchanged. I still want to subsume Awk, susbume Make, and implement an interactive shell — but I want to write this code in Oil. Oil is a shell, but it will also be a programming language like Python or ML.

So these tasks are blocking on the Oil parser and interpreter.

5. Even Longer Term

I wrote about some even loftier goals in this blog post. I don't want to talk too much about work that's years in the future, but I think it's important to have a vision for the design. Oil is about respecting the past, but not calcifying it!

In particular, the shell is naturally related to distributed computing.