blog | oilshell.org

February Recap / Brain Dump

I outlined a dozen or so blog posts in last month's roadmap, and I ended up finishing seven of them.

The last one was discussed on Hacker News three weeks ago. The discussion covered many of the remaining blog topics, like which features I've cut from Oil, and opportunities for a better interactive hsell.

So rather than write five more posts, this post summarizes important comments in that thread.

I also explain some recent progress in translation to C++, and summarize two releases I've made in the last month.

Important Comments

"Fish Oil" Is A Good Idea (link)

I'd like to expose an API that will let others build a fish-like interactive shell on top of Oil.

Start a new thread on Zulip if you're interested in shaping the API. Here's one thread about the interactive shell.

I'm not sure exactly what this API looks like. It feels complicated because I learned that shells and libraries like GNU readline fight over SIGWINCH , the Unix signal that notifies a program when the terminal has been resized.

Context: I'm reducing the scope of the project to get it "done" in 2020, but I think Oil is substantial enough for bigger projects to grow around it.

Why Oil Won't Have Types in the Near Future (link)

This is the most important comment regarding language design.

It relates to experiences I had with Python 3 strings and the "third style" of HTML processing I'm now using to generate this blog (source code in lazylex/ and doctools/).

I tried to mint a slogan Bijections That Aren't Bijections, but I'm not sure it's a useful one.

I also updated the Structured Data in Oil wiki page with links to related projects involving CSV, JSON, recutils, and binary formats. I think these projects are more closely related to the kind of "types" Oil should have, in contrast to Python-like data types.

(Ironically, after writing this comment, I did refactor Oil to have something more like a proper integer type.)

Discussion on structured data is welcome. Prototypes with working code are even more welcome!

shopt -s nullglob and #real-problems (link)

Oil now has shopt -s nullglob on by default, which addresses the quoting problem described in A Surprisingly Common Mistake Involving Wildcards & The Find Command.

oil$ find . -name *.jpg find: missing argument to `-name' oil$ find . -name '*.jpg' foo.jpg

Details on Zulip.

I've also created the #real-problems tag, which contains this post and posts describing shopt -s strict_argv and strict_control_flow . Whenever I encounter a complaint about shell in the wild, I try to address it in Oil.

January Blog Recap

Recent Progress in Translation to C++

I said I would write Recent Progress in Oil, but let me focus on just the translation process, since I'll continue to mention it in the blog.

I wrote bin/osh_eval.py a few weeks ago, which includes parsers for all shell sublanguages and evaluators for some of them. Oil uses a clean dependency inversion style, so it's easy to create subsets of the interpreter.

This program translates to 16,648 lines of C++ code, compared with 9,735 lines in bin/osh_parse.py, which I got working in December with Oil 0.7.pre9.

The initial translation on February 3rd produced 296 compile errors. Last night there were 38 compile errors. Much of this work involved rewriting Python reflection as textual code generation of C++ and Python. Details on Zulip.

But compiling isn't the end of the translation process: it still has to run! I often write stubs with assert(0) bodies to get the translated code to compile.

In summary: I'm making progress on translation, but it's not clear how long it will take.

Unfinished Posts

I mentioned these posts in the January blog roadmap, but I won't finish most of them:

Use Cases For a Better Shell Language. Cut for lack of time.

Examples of mycpp Translation. Also cut, although I did describe the tool in e-mails to Michael Sullivan, Ethan Smith, and Guido van Rossum of the MyPy team. I could use those e-mails as a starting point.

A Rough Development Schedule / Engineering Tasks for Oil 1.0 An upcoming post will review progress on Oil 0.8.pre1, which will give me a clearer idea of what will fit in the next year.

Language Features Already Implemented / Language Features Not Yet Implemented I'm too busy with translation to write these. I'd like to talk with experienced shell users about the language, either on Github or Zulip.



In summary, I want to translate more of Oil to C++ before making any predictions. If you want the pithy answer, it's:

Translation is required to make Oil a production-quality shell, and whatever else fits in 2020 is what will get done.

Posts I Still Want to Write

On the other hand, these posts are worth the time:

What the Oil Project Has Accomplished A principled linguistic foundation, an interactive shell, strictness, the Oil language, Eggex, a way of speeding up statically-typed Python code, a modular architecture, and more.

How to Rewrite Oil in C++, D, Nim, or Rust If Oil doesn't reach its goals, I hope to spread the knowledge to another project.



Recent Releases

0.8.pre1 on February 17th

I'll describe this release in detail tomorrow, but the summary is:

I'm happy that it involved four other contributors. And I'm still looking for more help! Every contribution was something taken off my plate, which means that the shell will be more capable by the end of 2020.

other contributors. And I'm still looking for more help! Every contribution was something taken off my plate, which means that the shell will be more capable by the end of 2020. Most of my changes involved translating Oil to C++, which I described above. I enhanced the mycpp translator, refactored the code, and replaced Python reflection with textual code generation.

I implemented the -n "nameref" feature of assignment builtins, and tweaked some parts of the Oil language. I'm keeping the documentation up-to-date.

0.7.0 on January 20th: 6 months of Work

I didn't announce the 0.7.0 release on January 20th, but here's a summary of the 0.7.x series, which started in July 2019:

Even though I feel this project has taken too long, that looks like an impressive 6 months of progress to me! The Oil language didn't exist in July.

(For comparison, the 0.6.x series from 2018-2019 was about the #interactive-shell and the OSH language features to support it.)

What's Next

I had more #zulip-links in this post, but I moved them to tomorrow's post. I want to make the project friendly to contributors, and one way to do that is by explaining its components and their status.

I'll describe the Oil 0.8.pre1 release in more detail. I want to credit contributors and review the metrics. This release spanned about 30 days, so analyzing it will suggest what we can finish in 2020.