blog | oilshell.org

Dev Log #8: Shell Protocol Designs

I didn't write much during the fall, so this is the second of four posts to bring readers up to date on the project's status:

Dev Log #7: Hollowing Out the Python Interpreter Dev Log #8: This post is about ideas I've had to improve the shell ecosystem. Dev Log #9: Oil Subprojects (OPy, OVM, etc.) Dev Log #10: Coding Experiments and Research.

I'm writing this for present and future Oil developers. If you have questions, please leave a comment!

Shell-Agnostic Autocompletion

After trying to run bash completion scripts with OSH, I realized that most alternative shells have the same problem: creating a new corpus of completion scripts.

It would be better if we could share this task rather than doing it N times — once for each new shell.

I started a discussion with several shell developers, including Qi Xiao of Elvish and zsh developer Oliver Kiddle. I was impressed that Qi started porting git-completion.bash to Elvish, which will teach us a lot.

This is work-in-progress, but here are some wiki pages that came out of it:

Please leave a comment if you've written a shell completion script in any language!

Running Bash Completion Scripts

The other approach I'm taking is to run existing completion scripts with OSH. Here's a wiki page for people who want to help:

Issues:

Coprocess Protocol

In the Shellac protocol proposal, I wrote that sending a request to a coprocess would result in lower latency compared to invoking a binary for every TAB completion.

In this usage, a coprocess as a single-threaded server that runs main() in a while (1) loop. (bash has had support for coprocesses since version 4, but they're limited and rarely used.)

A few weeks later, I had a productive conversation with Avery Pennarun, author of redo, about the problem of process startup time in builds.

This turned into a prototype of a coprocess protocol:

The idea is that is should be easy to turn existing command-line tools written in any language into coprocesses.

Please try the demo described in the README, and let me know if it makes sense. Can you think of a tool you use that starts slowly, and can you imagine retrofitting it with this protocol?

The Problem of VMs that Start Slowly

I'd like to write a post about small programs being burdened by large runtimes. If you write a 100-line Python or Java program, the runtime does a lot of computation and I/O before you even get to main() . This inhibits Unix-style reuse.

This problem seems to be getting worse, not better.

Here are some more rough documents on the potential solution:

There's still more to do, including:

Handling concurrent coprocesses.

Improving efficiency in some cases with descriptor passing over Unix sockets.

Proving that the protocol is viable by porting more command line tools to it. The tools will operate in coprocess mode only when an environment variable is set, so it will be a backward-compatible change.

Eventually build it into OSH and Oil.

Summary

I'm excited by the coprocess protocol! It's easy to implement and solves a concerete problem that's only getting worse.

I'm also excited by the possibility of a completion protocol, but I recognize that it's a long-term goal with many dependencies. On the bright side, every author of an alternative shell will hit this problem, and our interests are aligned in solving it.

These two protocols aim to improve the wider shell ecosystem. The next post will discuss projects that are part of Oil proper: OPy, OVM, and more.