blog | oilshell.org

Blog Roadmap for January 2020

The last post explained that I'm reducing the scope of Oil in order get it "done" in 2020.

I also explained my "time-based" blogging strategy. So this post enumerates posts that I think I can finish in one day each. I explain the main point of each one in a sentence or two.

Right now it feels like too many posts to write, but I hope to publish most of them.

Recap and Review

This first group of posts will bring the blog up-to-date with the project. Last year, I published many updates to oilshell.zulipchat.com rather than the blog.

The Simplest Explanation of Oil

This tweet has a pithy summary of the frustrations of shell programming. I'll explain Oil's take on these problems.

Clarifying the Project's Goals

To help readers understand what's in and what's out of scope, I discuss three levels of "ambition" for the Oil project. Main point: Oil is still your upgrade path from bash, but it doesn't do many things that Python does.

Use Cases For a Better Shell Language

Even though it's less ambitious, the I want the Oil language to handle many problems: building packages and distros, automating data processing pipelines, customizing the interactive shell, and more.

Recent Progress on Oil

What's happened since the last release announcement? After translating Oil's parser to C++, I further optimized it. I also made good progress on the doc toolchain.

Examples of mycpp Translation

I could write many posts about this Python-to-C++ translator, but I'll just show pairs of code snippets to give you an idea of how it works. (For the impatient: compare the contents of the oil-native-$VERSION and oil-$VERSION tarballs.)

Blog Backlog

I've drafted two posts describing unique aspects of Oil's implementation:

The Interactive Shell Needs a Principled Parser. Unlike the parser for bash and other POSIX-compatible shells, Oil's parser can be used to produce accurate autocompletions.

Oil's Parser is 160-200x Faster than it was Two Years Ago. And it's faster than the bash parser, even though it does more work. Writing Oil in Python was a risk, but the results turned out better than I expected. Speed is no longer a problem.

What the Oil Project Has Accomplished

Let's review the project's major achievements going back to 2016. This includes running thousands of lines of existing shell scripts, comprehensive spec tests, and a new language design.

There are many shell users who haven't heard of Oil, and I want to convince them that the project is worth contributing to.

Plans for Oil

A Rough Development Schedule

I did 2-3 months of work on performance in 2019, so I have a good idea what the next year will look like. I expect around 6 months to be devoted to performance, leaving 6 months for features, bug fixes, and documentation.

Language Features Already Implemented

I've haven't written many blog posts about the #oil-language because I want to spend the time on the manual instead.

But it's worth describing the language informally. Oil has significant new features like Ruby-like blocks, but it won't have typed data like (nested) dictionaries and floats, and it won't reuse code from the CPython interpreter.

Language Features Not Yet Implemented

I'd like better debugging and tracing support, xargs-like functionality with each , namespaces with use , and a few other things. None of these features require typed data.

Engineering Tasks for Oil 1.0

A concrete plan to get Oil "done". All Python code will be translated to native code with tools like ASDL and mycpp. There will be no bytecode, which solves the "double interpretation" problem.

Two Technical Problems With Translation to C++

The C++ translation has worked better than expected, but I see potential problems in two areas:

Memory Management

Interfacing with native C code, e.g. GNU readline and CPython.

I outline solutions to these problems.

Backup Plans

Catch-22: Coding vs. Blogging

After publishing these posts, I want to go back to coding. But I still want to onboard new contributors, which requires explaining the project. I also want help maintaining Oil in 2021 and beyond.

Three Possible Futures for the Oil Project

I see a concrete way for Oil to become usable in the next year, but there's no guarantee it will happen. It could fail to reach my goals. I'll discuss a "minimum backup plan" for Oil. I think of it as libosh : a library for implementing a shell.

How to Rewrite Oil in C++, D, Nim, or Rust

I mentioned this post in July with the release of Oil 0.7.pre1. In essence, many parts of Oil like its lexer and ASDL schemas should give others a head start in replacing bash. I wouldn't mind seeing another effort now. I believe in the power of parallel open source development!

Writing a compatible shell is a lot of work, but it should help that Oil's codebase is 5-7x smaller than bash's.

Other Posts

These posts might not come in January, but they're on my mind.

Lessons Learned From Oil

Did the metaprogramming strategy make sense? Did going "wide" rather than going "deep" make sense?

Educational Posts

I have ideas for about 7 educational posts, based on work in these areas:

Translating statically-typed Python to C++.

Optimizing the resulting C++ code, e.g. with uftrace and other tools.

The doc toolchain. I wrote an HTML lexer and it's useful, and even novel!

Next

If this post makes sense to you, please see Help Wanted. Feel free to ask me questions!

Subscribe to /r/oilshell or follow @oilshellblog if you want read the posts I've just outlined. The next post will give you the simplest explanation of Oil.