blog | oilshell.org

OSH 0.6.pre15 Does Not Crave Chaos and Destruction

As promised two weeks ago, here's a new OSH release:

Please try it both interactively and on batch programs! This post gives further detail on how to test OSH, and what's useful to test.

To build and run it, follow the instructions in INSTALL.txt. Please report bugs on Github.

If you're new to the project, see Why Create a New Shell?.

Two Colorful Metaphors

The strict-argv feature in this release was motivated by a recent thread on the help-bash mailing list. An excerpt regarding error messages:

... that's just bash's way of lulling newbies into complacency so that their scripts blow up LATER instead of IMMEDIATELY. Obviously a script that APPEARS to work has a better chance of going into production in a broken state. This maximizes the chances of chaos and destruction. Bash craves these things. The suffering of fools is its lifeblood. — Greg Wooledge, maintainer of BashFAQ

This suggests some new slogans:

Oil: The suffering of fools is not its lifeblood.

its lifeblood. Oil: The pleasant operation of Unix machines is its lifeblood.

Let me know if you have other ideas :-)

Feedback from the BayLISA talk confirmed that error messages are important, so it's good to know that Oil is on the right track. The release process includes a script that tickles all parse errors and all runtime errors. They're better than bash error messages, but could use another pass of polish. Some location info is missing.

Speaking of metaphors, I learned of the Augean stables in a recent lobste.rs thread. I also got some good ideas for an Oil logo from Rory O'Kane.

Recent Changes

This section summarizes the git changelog for 0.6.pre14 and 0.6.pre15:

User Contributions And Testing

I released OSH twice in the last two weeks due to meaningful contributions and testing:

Issue 224: Building OSH on OS X works again. I removed non-portable linker flags. See the caveats portability below.

Issue 225: Ctrl-C no longer crashes the release build. (TODO: it should properly erase a command that hasn't been entered.)

Issue 226: An alias can now contain a pipeline, as well as several other constructs. Many more spec tests pass.

can now contain a pipeline, as well as several other constructs. Many more spec tests pass. Issue 230: Implemented the $HOSTNAME variable.

Thanks to wolverian , RobSykes , tyxieblub (on Github), and tekknolagi (on Reddit) for the bug reports!

In addition, tyxieblub improved bash-compatible prompt support:

\w and \W expand to proper variants of the current working directory.

and expand to proper variants of the current working directory. \h and \H expand to proper variants of the hostname.

A New Strict Option

Inspired by help-bash threads like the one above, I implemented set -o strict-argv .

When it's on, OSH will fail when a command evaluates to an empty argv array, which can happen due to the elision of unquoted, empty words:

osh$ set -o strict-argv osh$ if $unquoted_empty_var; then echo true; fi Line 1 of ' ' if $unquoted_empty_var; then echo true; fi ^~~~~~~~~~~~~~~~~~~ fatal: Command evaluated to an empty argv array

In this case, the user probably meant test -n "$empty_var" .

. Another case that users hit in practice is $(echo x >/dev/null) — an unquoted command sub that evaluates to the empty string.

A future post will describe other OSH-specific options:

strict-errexit

strict-control-flow

strict-word-eval

I plan to add strict-ALL to opt into all of them in a single line.

I've already written about providing more parse errors. The strict options are about providing more runtime errors.

Features That Allowed Dogfooding

Although I've already announced the 0.6.pre13 release, I hope that these additional details suggest areas to test. OSH needs more testing on real shell scripts!

OSH now runs Python's virtualenv. I discovered that backticks are parsed differently than $() , and virtualenv relies on this odd behavior. I fixed it by adding a lexer mode and another pass of parsing. (Complete list of lexer modes).

OSH now runs git-prompt.sh . C-escaped string literals within arguments to variable expansions are now parsed properly. For example, in "${sep:-$'\t'}" , the TAB literal $'\t' is an argument to the :- operator. The read builtin now sets variables with dynamic scope.

OSH runs many (but not all) bash-completion plugins. compgen -W parses its argument as code , evaluates it, and splits it with $IFS . compgen -X <pattern> properly composes with other flags.



In addition, I implemented a "devtools" feature that uncovered many bugs:

If something like OSH_HIJACK_SHEBANG=/usr/local/bin/osh is set in the environment, then OSH will run #!/bin/sh and #!/bin/bash scripts with that shell interpreter. This allows deep testing of scripts that invoke other scripts, without editing code.

Bugs Fixed While Dogfooding

These changes also went in the 0.6.pre13 release:

Issue 202: I fixed an off-by-one error in code to find a free file descriptor, which okayzed originally pointed out. I also discovered that I should use an fnctl() flag to do this in the kernel , rather than in user space.

originally pointed out. I also discovered that I should use an flag to do this in the , rather than in user space. Fixed the behavior of pushd and popd , including the update of $PWD .

and , including the update of . I hit a classic pitfall with errexit in Oil's own shell test harnesses. I fixed the harness and filed issue 179 to help others avoid this pitfall in OSH. In summary, the strict-errexit flag mentioned above could be even stricter. Functions and external commands should behave differently with respect to errors.

with in Oil's own shell test harnesses. OSH can now look inside aliases for completion plugins. For example, if you have alias ll='ls -l' and no plugin defined for ll , OSH will use the plugin registered for ls . Note that zsh does this, but bash doesn't.

for completion plugins. Fixed translation of a[i++]=foo and backticks, since they unfortunately require another pass of parsing. These are two places where we break the rules I described in the last post. Not only is dynamic parsing like alias bad, making two passes of static parsing is also bad. It makes it harder to generate good error messages (although a solution for this problem is coming.)

and backticks, since they unfortunately require another pass of parsing.

Also:

I learned that the kernel will return ENOEXEC when an executable file doesn't contain native code (ELF) or doesn't start with a shebang line. All shells handle this error by interpreting the file themselves, as if it were a shell script.

ASDL Schemas Are Now Compiled to MyPy

I've mentioned a few times that Oil is too big and too slow. This is still an open problem, but I believe that the first step to make it faster is add static types to the code.

Luckily, the MyPy project exists, and has undergone heavy development since I last tried it in 2016. Oil is also more mature, which means I can list every place that it uses metaprogramming/reflection, and generate textual code instead.

In other words, the tradeoff between type checking vs. metaprogramming has changed in the past 2+ years.

This release made progress by compiling ASDL schemas into Python code that passes mypy --strict . We now run MyPy on Travis as well. There's still much more code to annotate with types, but this is a first step.

(Recall that ASDL schemas define the lossless syntax tree, as well as important runtime data structures.)

What To Test and How to Test It

I hope that the long list above gave you a feeling for what OSH can do. It runs real programs, but there are undoubtedly missing features and undiscovered bugs.

You can test it on both interactive and batch programs. I started a How To Test OSH page on the wiki. Summary:

bin/osh behaves like bin/sh or bin/bash , which makes it easy to test.

behaves like or , which makes it easy to test. The equivalent of ~/.bashrc is ~/.config/oil/oshrc . You can make an ~/.oshrc symlink if that becomes annoying to type.

is . You can make an symlink if that becomes annoying to type. Running completion scripts requires our temporary fork of the bash-completion project, under the dev/osh-release-0.6.pre15 branch. I removed some exotic bash features and simplified the code. The next post will explain this in detail.

branch. I removed some exotic bash features and simplified the code. The next post will explain this in detail. OSH has unique features that aid in testing: OSH_HIJACK_SHEBANG=/usr/local/bin/osh for deep testing (mentioned above). The --debug-file flag to show log messages and tracing.



Feedback is welcome in any of these ways:

Portability Notes

A shell has two main dependencies: libc and readline (a line-editing library). In both cases, OSH has minor GNU dependencies. They should be removed, but for now here are two notes.

Extended Globs Rely on GNU libc

OSH now builds again on OS X, but bash-completion plugins won't work. They use extended globs, and OSH relies on GNU libc to parse that syntax. In contrast, bash has its own implementation of extended globs — which is slow, according to the manual.

OS X and distros like Alpine Linux do not use GNU libc, so extended globs and completion scripts that use them won't work.

The work described in issue 192 should fix this, and it involves some fun computer science. On the other hand, I'm not sure I know how to do it, and it might be OK to do something hacky like bash. Please chime in if you're familiar with the implementation of regular expressions.

The Oil language won't have extended globs, since they're redundant with regexes.

Completion Relies on GNU Readline

Hitting TAB for completion doesn't work on OS X.

I've made some changes to CPython's readline wrapper, which probably broke libedit support on OS X. On the other hand, the readline support in Python 2.7 my Mac seems broken anyway, but I didn't look into it further.

If you have any expertise in this area, let me know.

What's Next?

I'll publish my draft of The Interactive Shell Needs a Principled Parser.

I plan to release version 0.6.0 after: More end user testing and feedback. Let me know if you have problems testing it. Integrating some UI enhancements I've already prototyped.

after:

Appendix: Selected Metrics

Let's compare the November release of O.6.pre8 with the current release. I reviewed metrics for the former in Hollowing Out the Python Interpreter.

65 more spec tests now pass, e.g. for completion builtins and other interactive features:

There was a slight regression on the million lines of shell, due to backtick parsing mentioned above:

wild tests for 0.6.pre8: 231 parse failures, 3 translation failures

parse failures, translation failures wild tests for 0.6.pre15: 232 parse failures, 10 translation failures.

But all the new interactive features cost us less than 1000 significant lines of code:

cloc for 0.6.pre8: 10,174 lines of Python and C, 149 lines of ASDL

lines of Python and C, lines of ASDL cloc for 0.6.pre15: 11,041 lines of Python and C, ~200 lines of ASDL (excluding testdata).

The story is similar when including whitespace and comments (physical lines):

For reference, bash has ~88K significant lines of code and ~124K physical lines of code (line count script).

I'm happy that the OSH source code has stayed small. It gives me confidence that I can make it both faster and smaller without rewriting all the code.

Native Code and Bytecode Metrics

I hope the MyPy / C++ translation will make these metrics obsolete, but here are some minor updates:

There was a slight increase in native code:

along with a bigger increase in binary size:

This was caused by removing linker flags to remove unused code. The flags apparently aren't portable, and this metric isn't very important now. OSH needs to be drastically sped up in some other fashion.

The bytecode size went up nearly 200 KB: