blog | oilshell.org

Oil 0.8.pre3 - A Line Editor and a Continuous Build

This is the latest release of Oil, a Unix shell:

/release/0.8.pre3/: source tarballs, docs, metrics, and more.

Please try it on your shell scripts and report bugs! To build and run it, follow the instructions in INSTALL.txt. The wiki has more tips on How To Test OSH.

This release has two major themes:

Features and fixes toward running the Bash Line Editor (ble.sh). I learned of this fascinating program a few weeks ago. Behind-the-scenes work to continuously build and test Oil. We now have travis-ci.oilshell.org, and I'm excited about offloading more work onto it.

Tie-Ins

The two themes strongly relate to prior blog topics. Before diving into details, here is some higher-level context.

Project Scope

Shell is both an user interface and a programming language, and I've addressed this issue a few times.

There's no reason a shell can't do well in both respects — it's just a lot of work. For reference, bash itself is ~142K lines of code. Its companion GNU readline, which provides a minimal experienced compared to say fish, is ~34K lines of code.

Serendipity With ble.sh

A few weeks ago, Danilo Spinella pointed out the Bash Line Editor project, which I'll refer to as ble.sh from now on.

It gives you a fish-like interactive experience in bash. The astonishing thing is that it's written in bash. I'm no stranger to big shell scripts, both batch and interactive, but this surprised me!

I'll write more about ble.sh in the future, but here are some links:

The Biggest Shell Programs in the World: I believe it's the most substantial shell program in the world by a large margin. It's not just that it's large (20K-30K lines), but it also uses sophisticated data structures and algorithms. If you know of other big shell programs, edit the wiki page and let me know.

How Interactive Shells Work: Author Koichi Murase describes how ble.sh works. This information is useful for API design in Oil. If you have expertise on the internals of another shell, again, edit the page and let me know.



I'm not sure if Oil will be able to run ble.sh, but this release is major progress toward doing so.

If it eventually does, that will show an unexpected benefit of Oil's compatibility with bash, which has been a tremendous effort.

Shell: The Good Parts

The other major theme is the continuous build at travis-ci.oilshell.org.

Its source code lives in the new services/ directory, and I started calling it "Toil". Besides being a useful tool, its implementation demonstrates techniques that I want to write about under #shell-the-good-parts:

Toil is a set of shell scripts that invokes many tools: Python scripts I wrote, ssh with Bernstein chaining, scp , zip , etc. In other words, shell is always the main() ; I reuse as many Unix tools as possible; and write my own where there are gaps.

with Bernstein chaining, , , etc. Travis CI and its YAML config is just another tool. I try to decouple my code from the details of code I don't control. A service is just a process on another machine . Toil should eventually run on build services other than Travis, which is required for platforms other than Linux/OS X and x64 . When it does, this will be more convincing.

. Some parts of the build run in a Nix environment, but Nix is also just another tool (and it happens to be one with its own programming language). Sometimes its model is useful, and sometimes it isn't. Shell is the most general model. It's at the lowest level.

In short, I use the Unix philosophy of reusing programs written in different languages as shell "libraries".

More Shell Usage

These release announcements are now semi-automated with a shell script. The template it generates involves a git log in HTML, as well a list of closed issues with Github's API (Appendix B).

A script called html_head makes it convenient to generate HTML from shell, as again I did for both Toil and these release notes.

New in This Release

Now that we've reviewed how Oil's infrastructure uses the Unix philosophy, here are the detail of this release.

Code Changes

This section summarizes the full changelog, starting with external contributors:

Travis A. Everett fixed spec tests to make them more portable, e.g. to a Nix environment.

As of this release, we continuously run tests in Nix and publish results to travis-ci.oilshell.org. If you're interested in using Nix to work on Oil, let us know! Not all tests pass yet.

Koichi Murase, author of ble.sh, made many changes necessary to run it.

Implement declare/readonly/export -p to print variables in a form that eval understands. ble.sh uses this idiom to save and restore state.

to print variables in a form that understands. ble.sh uses this idiom to save and restore state. Implement several constructs in the printf language, including %()T for timestamps and %*s and %.*s for user-defined width and precision.

language, including for timestamps and and for user-defined width and precision. Implement most remaining file descriptor operations. 3>&- to close a descriptor and 3>&1- to move a descriptor. (These features are useful for compatibility, but Avoid Directly Manipulating File Descriptors in Shell in general.) {varname}>out to have the shell choose a file descriptor and store it in a variable. This needs a better syntax in Oil.

Fixed file descriptor leaks and added tests.

Fix the semantics of slicing the arguments array like ${@:i:j} . Surprisingly, ${@:0} gives you more items than ${@} has!

. Surprisingly, gives you more items than has! BASH_LINENO / FUNCNAME can be referenced as scalars or arrays, like BASH_SOURCE .

/ can be referenced as scalars or arrays, like . Fix a bug evaluating the ${!prefix@} construct.

construct. Add failing spec tests for read -d , mapfile / readarray , and dynamic RHS in arithmetic.

In addition to many other changes, I made the following changes related to ble.sh:

Implement read -d . (It's useful to have failing spec tests before implementing a feature. I merge changes that only include failing tests because they clarify what Oil should do.)

. (It's useful to have failing spec tests before implementing a feature. I merge changes that only include failing tests because they clarify what Oil should do.) Implemented test -v and test-k .

and . Enforced an upper limit on hard-coded file descriptors. The argument to F_DUPFD ensures that a shell's own descriptors (e.g. to implement source ) don't collide with the program's descriptors.

ensures that a shell's own descriptors (e.g. to implement ) don't collide with the program's descriptors. Refactored the file descriptor handling code. This code may be rewritten in C++ rather than translated because it's mostly I/O.

Added a special case for declare -A dict=() , which is now documented in Quirks.

, which is now documented in Quirks. Added location info to the error message for $(( a )) when a doesn't look like an integer.

when doesn't look like an integer. Fixed eval and trap to accept -- .

and to accept . Turned on set -o emacs in interactive mode like bash does.

in interactive mode like bash does. Hid a strict syntax check under shopt -s parse_ignored for bash compatibility.

Other:

Implement printf %b , which allows the argument to use backslash escapes like echo -e .

, which allows the argument to use backslash escapes like . Added failing spec tests for a word splitting bug found by GammaFn .

. Many changes to our Travis CI configuration and to "Toil".

Changes to run spec tests in an unknown release environment as opposed to a known dev environment. For example, we want to test the Oil release binary on Alpine Linux and OS X.

Documentation

The docs are still in their early stages, but I've updated a few of them:

Known Differences

Quirks for compatibility. OSH has generally maintained clean semantics while staying compatible, but sometimes they can't be clean.

I consider these two docs "missing" documentation from bash and other shells (still in draft state): Interpreter State. How does "shell memory" work? Process Model. When are processes started? Of course, the entire shell memory is copied on fork() .

Architecture Notes Parser Architecture was split into its own doc.



On the Wiki:

Language Design Principles. These principles guide decisions about the evolving Oil language and help me explain it to others. I also added a section on protocol design principles, which will be important in the near future.

Zulip Threads

Reminder: There are threads on oilshell.zulipchat.com about almost every topic in this post, and more:

Research on Interactive Shells . I'd like more people with experience to help us figure out what to do and how to implement it.

. I'd like more people with experience to help us figure out what to do and how to implement it. ble.sh

Language Design Principles

The Unix Philosophy

What's Next?

If you're a bash user who wants a fish-like experience, please try ble.sh and let me know what you think. It took me less than 20 seconds to install and try. The instructions are at akinomyoga/ble.sh.

Appendix A: Selected Release Metrics

Let's compare this release with the previous one, version 0.8.pre2, which I reported detailed metrics on earlier this month.

Tests

The features and bug fixes for ble.sh caused many more spec tests to pass. And we have more failing spec tests to guide future work.

A few more Oil spec tests fail because I made the test framework stricter. If there's a Python traceback on stderr , the test fails no matter what. This surfaced a few bugs which I quickly fixed, but others remain.

Lines of Code

I'm happy that all the new features fit in ~400 new significant lines of code:

cloc for 0.8.pre2: 15,272 lines of Python and C, 293 lines of ASDL

lines of Python and C, lines of ASDL cloc for 0.8.pre3: 15,633 lines of Python and C, 300 lines of ASDL

... and ~600 new physical lines of code:

Benchmarks

The translation to C++ still works, but it didn't change in this release. There was no meaningful change in parser performance:

Or the size of the binary:

ovm-build for 0.8.pre2: osh_eval is 663,792 bytes of native code (under GCC)

is bytes of native code (under GCC) ovm-build for 0.8.pre3: osh_eval is 667,888 bytes of native code (under GCC)

Appendix B: Closed Issues