blog | oilshell.org

Release of OSH 0.7.pre1

This is the latest version of OSH, a bash-compatible shell:

Please try it on your shell scripts and report bugs! To build and run it, follow the instructions in INSTALL.txt. Please report bugs on Github.

If you're new to the project, see Why Create a New Shell? and the recent FAQ.

Changes

This section continues where I left off in What's Happened Since February?, published last month. That post summarized versions 0.6.pre16 through 0.6.pre22.

This section summarizes the last 3 releases, including 0.6.0.

I didn't write an announcement for the 0.6.0 release because the AMA on /r/linux last month generated a lot of feedback. Thanks for the support!

Contributors: let me know if I missed something notable.

0.6.pre23 on June 26th (changelog)

Highlights:

Implemented basic job control. The typical flow of vim , Ctrl-Z , and fg now works.

, , and now works. Marek Suppa implemented test -o to check if an option is set.

to check if an option is set. Implemented the -b -c -p -S flags to test .

flags to . Fixed bugs reported by Quentin L'Hours: Word are now joined correctly when $IFS is empty and when it's unset. Literal words like echo are no longer split.



I also triaged spec test failures. See the metrics at the end of this post.

0.6.0 on July 1st (changelog)

OSH now has a manual, and a list of known differences with other shells. From now on, it will be published with every release.

Changes related to strict options:

Moved them under shopt , rather than set -o e.g. shopt -s strict-array turns an option on, and -u turns it off.

, rather than Added shopt -s all:strict to turn on all strict options at once. Goal: OSH helps you write better shell scripts, like a ShellCheck that also has knowledge of what hapens at runtime.

to turn on all strict options at once. Turned on strict-arith by default.

by default. Documented strict options in the manual.

Other:

printf : Implement hex and octal formatting.

: Implement hex and octal formatting. Fixed bugs reported by Quentin L'Hours: &> and &>> no longer take a file descriptor on the left. Boundary cases for ${var%*} and ${var#*} .



0.7.pre1 on July 17th (changelog)

Highlights:

Overhauled and implemented the rest of array and associative array semantics, including literals like A=(['key']=value ['k2']=v2) . These semantics still need to be documented, but I'm happy with how it turned out. It relates to the issue I raised in Parsing Bash is Undecidable almost 3 years ago!

semantics, including literals like . Implemented the hash builtin, along with $PATH caching. (I didn't see any notable speedups as a result.)

builtin, along with caching. (I didn't see any notable speedups as a result.) Implemented test -O and -G .

and . Joshua Nelson Made the prompt width calculation Unicode-aware. Changed source to look up entries in $PATH . Run files without a shebang with #!/bin/sh . Added location information for ~ , which improves error messages. Started property-based testing using Hypothesis. Tested OSH on FreeBSD. Fixed an incompatibility in the configure script, and in our libc binding. Initial implementation of $PROMPT_COMMAND , which many bash users use to customize their interactive experience.

Wilke Schwiedop committed an initial implementation of find and xargs . This isn't released yet, but I expect it to play an important role in the Oil language, which will subsume the functionality of these two tools. Why? find is a recursive expression language , like many other shell sublanguages. xargs starts processes and splits strings, just like a shell.

and . This isn't released yet, but I expect it to play an important role in the Oil language, which will subsume the functionality of these two tools. Why? Fixed bugs reported by Quentin L'Hours: exit behavior across sourced modules. The default status for exit and return . Waiting for singleton pipelines like ! true .

Matt Singletary and Stephen Gregoratto revised the man page. (Still needs to be released and installed.)

Some work on the Oil language. I decided to start with the expression language. Details forthcoming.

What's Next?

There's still plenty more to do. Here are some highlights of major project areas:

(1) OSH language. After recent changes and enhancements to arrays, printf , and brace expansion, it feels like it's "converging". I expect the upcoming overhaul of assignment builtins to be one of the last major changes.

(2) Oil language. I have a concrete design for the language, but I only recently started implementing it. I also started a design-docs directory. It will be a lot of work, but we're finally getting to the main point of the project!

(3) Performance. Still needs a lot of work. Python lets us work quickly and safely, but the code needs to be much faster. I have an idea for a blog post: How to Rewrite Oil in C++, Rust, or D. If you're interested in that, leave a comment.

(4) Documentation. We now have a short manual, but the quick reference and help builtin need a lot of work.

(5) The Interactive Shell. I want to make the shell more immediately appealing. People use shell interactively more than they write programs with it.

Display completion descriptions from various sources, as zsh and fish do.

Color in error messages (optional).

How to Help

It's good that Oil is getting more contributions, because I can't do this all by myself. I can use more help!

I suggested some ways to help in the recent FAQ: How to Help

I continue to keep issue labels up to date: #good-first-issue and #help-wanted #open-problem: Apply your unique expertise here! #should-run-this: Shell programs we should run. Try running your own programs too.

I updated the README.md, which gives an overview of the repo. It looks big, but the parts are modular and well tested.

Oil is a plain Python program, with plain C extensions. That means it's quick to modify and test. The main thing to get used to is the use of multiple code generators, which makes the build system complicated. This is documented on the Contributing wiki page.

Please ask questions on oilshell.zulipchat.com.

I'm making more fine-grained announcements there as well. For example, I just wrote a wiki page documenting internal data structures and algorithms:

Conclusion

The metrics below show that everything is pointing in the right direction.

Oil is making solid progress, but there's still a lot more to do!

Appendix: Metrics for the 0.6.0 Release

The last time I reviewed metrics was in the February release of OSH 0.6.pre15. Let's compare it with version 0.6.0 rather than 0.7.pre1, since I like round numbers.

New features and bug fixes made more than 114 new spec test cases pass. I also marked failures as known differences, which led to the failure count dropping:

We have ~1400 new significant lines of code:

cloc for 0.6.pre15: 11,041 lines of Python and C, ~200 lines of ASDL (excluding testdata).

lines of Python and C, lines of ASDL (excluding testdata). cloc for 0.6.0: 12,447 lines of Python and C, 239 lines of ASDL

And ~2800 new lines of physical code:

Native Code and Bytecode Metrics

The amount of native code was reduced:

This is largely due to forking and modifying posixmodule.c . I did this to backport PEP 475 to Python 2, which fixed our EINTR issues.

The binary size stayed almost the same:

The bytecode size increased significantly, out of proportion to the increase in lines of code:

It appears this is partly due to bloat in the code generated by our ASDL implementation. I'll keep an eye on this.

I hope the MyPy / C++ translation will make these metrics obsolete, but I'll continue to keep track of them until that's done.