blog | oilshell.org

Please Try the OSH 0.1 Release

I've released version 0.1 of OSH, a bash-compatible shell:

To try it, follow the instructions in INSTALL.txt, which is also available in the tarball.

There are more links at oilshell.org/release/0.1.0/, some of which I explain in this post.

What's New?

The changes fall under two major categories:

(1) New features and bug fixes motivated by real bash scripts. Before the last release, I ran highly portable shell scripts, as well as my own bash scripts. This time, I'm running bash scripts found in the wild.

The most prominent example is the setup.sh script at the foundation of the Nix build process. Issue 26 has details, which I've summarized in Appendix B below.

(2) Build Fixes. OSH should be installable on all Linux distros, as long as the CPU architecture is x86 and x86_64 . If you find that this isn't the case, let me know on issue 16.

(Future work: use QEMU to test OSH on other architectures and non-Linux OSes.)

How Can I Help?

Please run OSH on open source shell scripts that you're familiar with. They could be portable scripts, or scripts written for bash, dash, busybox ash, or another shell.

This is as easy as running

$ ./myscript.sh $ osh ./myscript.sh

and then comparing the output. Something like this might be useful:

$ ./myscript.sh >stdout.txt 2 >stderr.txt $ echo $? >status.txt

If you find a difference, file a bug on Github. I give priority to scripts that I can run myself, which is why I did a lot of work on issue 26 regarding Nix.

If you'd like to fix an issue you've found, the Contributing page has instructions on building, testing, and modifying OSH. Leave a comment or e-mail me if you need help.

Contributors

Linus Heckemann packaged OSH for Nix, ran it on Nix build scripts, and fixed build issues (changelog).

Earnestly on Github reported build issues, which led to several fixes.

on Github reported build issues, which led to several fixes. Several of my friends reported build issues and bugs via e-mail: Ari Russo, Eric Higgins, Paul Oppenheim, and Tony Hoffman.

(Let me know if I forgot you, and I'll edit this post.)

Caveats and Conclusion

It's still too big and too slow. However, I think I'm making good progress on the functionality of the shell, as shown by the test results and changelog below.

After OSH runs more real bash scripts, I plan to optimize it.

Going forward, I plan to use time-based releases, perhaps with 4- to 8-week intervals.

Appendix A: Project Metrics

Three months ago, I gave an update on project metrics. Here's another update.

Summary of spec tests results for OSH 0.1:

836 total tests, 662 passing, 123 failing.

For comparison, the 0.0 release on 7/23 had this summary:

752 total tests, 586 passing, 113 failing.

There are more failing tests because of increased test coverage. For example, OSH now has tests for Unicode string manipulation, but Unicode hasn't been properly implemented yet.

Source Code Metrics

The files in /release/0.1.0/metrics/ show that we have almost 14K lines of code in OSH proper.

Including all Python dependencies, it's ~29K lines, and C dependencies are another 164K lines of code. The C dependencies are what I call OVM.

I'm tracking the last two numbers so that I can reduce them over time!

Appendix B: Details on Changes

Here's a summary of the changelog.

Language Enhancements and Fixes

Implemented ${x/pat/replace} . This operation accepts glob patterns , not just strings. Related: Glob Matching Can Be Simple and Fast Too.

. Implemented prefix and suffix stripping, e.g. ${x#prefix }. These operations also accept globs. Notably, they also require submatches combined with greedy/non-greedy matching — see Regular Expressions: The Virtual Machine Approach. This means we can't use the libc regex API, but we can use Python's. Unfortunately this means we're using backtracking rather than the simple and fast linear-time algorithm. Note: These operations require knowledge of Unicode, which isn't yet implemented.

}. Parsed and implemented += .

. The local and declare keywords now accept flags like -a . (They are builtins in bash, but OSH statically parses them as part of the language.)

and keywords now accept flags like . (They are builtins in bash, but OSH statically parses them as part of the language.) Basic support for ${!varRef} .

. Implemented <& . Trivia: it's the same as >& but the default LHS descriptor is 0 instead of 1 .

. Trivia: it's the same as but the default LHS descriptor is instead of . Fix several bugs with set -e ( errexit ), with tests.

Shell Builtin Enhancements and Fixes

Implemented the test / [ builtin. The last post went into detail on its problems.

/ builtin. The last post went into detail on its problems. Various enhancements and error checks in cd .

. Better error messages in source and eval .

and . type -t . Display the type of a command word.

. Display the type of a command word. shopt (-p -o -s nullglob) . Print and set shell options.

. Print and set shell options. set -f ( noglob ). An option to disable globbing.

( ). An option to disable globbing. read (-r -n) . Read a certain number of bytes; read without escaping.

Runtime Enhancements

Implemented the ${FUNCNAME[@]} array for introspecting the call stack. This appears to be bash-specific.

Build Fixes

Fix the detection of the readline library.

Remove hard-coded values from pyconfig.h and port them to our own configure script. This fixes CentOS/RedHat and 32-bit x86 builds.

Other

Better error messages in some cases.

Things I Worked On That Aren't in the Tarball