blog | oilshell.org

Frequently Asked Questions (2019)

(Last updated 2020-10-12)

I'm writing this FAQ for an upcoming AMA on Reddit's /r/linux, but I expect it to be useful for all readers. (AMA stands for "ask me anything"; it's like a group interview.)

Update: AMA: I spent 3 years creating a new bash-compatible shell called Oil

Why is the Project Named "Oil"?

Because Oil is a compliment to "metal"! Shell is an essential lubricant for software systems. Programs in C, C++, Java, Go, Rust, etc. are analogous to "metal". I also use shell to glue together Python, R, and JavaScript. Because oil is a short word, like cat , awk , or sed .

The name often reminds people of the company "Shell Oil". Of course, it has nothing to do with that.

I expect associations to change over time. Python creator Guido van Rossum was fighting "snakes" for a long time, but now the name stands on its own.

8/2020 update: "Oil" may also remind you that the upgrade process is relatively smooth! It runs your existing shell scripts.

Why is it Written in Python?

The very first post on the blog has some color on this. I started the first serious of version Oil in C++, but progress was too slow.

Python let me implement a large portion of bash in five to seven times less code. Small programs are easier to write and maintain, and they have fewer bugs.

Current line counts [1]:

Physical Lines Significant Lines bash 142K 101K osh 26K 14K Approximate ratio 5.5x 7.2x

Squeezing a backward-compatible shell into 14K lines of code (excluding whitespace and comments) is a great result! It gives me confidence that new features won't bloat the shell.

To be fair, a big chunk of the Python interpreter sits below that 14K lines. But I didn't have to write that code. And I think of it as having an 87K line "code budget" to work with (101K minus 14K).

The main downside of using Python is that the OSH interpreter is too slow. But now that OSH can run thousands of lines of bash scripts, speeding it up is the top priority. I aim to do this without rewriting the whole program by hand, using "metaprogramming" and DSLs.

More on Python:

Is it Written in Python 2?

See the FAQ in Building Oil with the OPy Bytecode Compiler.

Why Use It?

OSH is ready for early adopters to test. I started using it myself in February.

At the moment, there are a couple reasons to use it:

(1) As a development aid while writing shell scripts.

It likely runs your shell scripts. If it doesn't, please file a bug. Caveat: Batch programs are more likely to run unmodified. Interactive ones like .bashrc and completion scripts may require minor patches.

and completion scripts may require minor patches. It has more precise error messages than any shell, and they can be easily improved further. The last post has an example.

than any shell, and they can be easily improved further. The last post has an example. It can prevent coding errors. It's more strict by default, and you can opt into additional strict modes (e.g. strict-argv ). This February release announcement shows that example, and there are more that need to be documented. You can think of it as a ShellCheck that also has knowledge of what happens at runtime.

(2) To support a new project to replace bash. The wiki page Shell Programs That Run Under OSH is editable and I would love your help in making it longer.

I hope to get feedback from the AMA for a more featureful 0.7.0 release. Although I'll still be happy if 0.7.0 improves only in speed.

Why Not Use It?

As mentioned, it's still too slow. December 2019 update: oil-native shows how we'll optimize Oil.

shows how we'll optimize Oil. The documentation is sparse. December 2019 update: there's a new /release/$VERSION/doc/ tree, but it's still missing content.

tree, but it's still missing content. It's mainly tested on Linux right now. It runs on OS X, but completion probably doesn't work.

There's no job control. I use tmux instead. July 2019 Update: Oil 0.6.pre23 implemented basic job control.

If you're interested in helping with any of these things, feel free to join oilshell.zulipchat.com.

What Happened to the Oil Language?

I described a new, legacy-free shell language over 2 years ago, but I haven't worked on it much since then.

Instead, I worked on running existing bash scripts because I think that's more important. There are many alternative shells, but I don't think they will replace bash because of its strong network effect.

I still believe that many shell features hinge on a better language, including:

A better API for shell autocompletion.

Better hooks for command history.

The ability to to manipulate structured data.

"Declarative" features to express system configuration and build configuration.

In the last post, I mentioned that I made some progress on Oil's expression language.

With the caveat that I just started implementing it, the two sections below give you a flavor for the language. In short, the syntax and semantics resemble Python and JavaScript. You'll be able to use the var keyword and then write a "normal looking" expression. There are also some elements of Ruby, Perl, and Awk.

Shell Expressions

The following POSIX shell code tests if a number is between 5 and 10:

n=$(( 3 + 4 )) if [ 5 -le $n ] && [ $n -le 10 ]; then echo "$n is between 5 and 10" fi

# Output: 7 is between 5 and 10

I'd guess that 50-90% of shell users couldn't write that correctly the first time. Reading it afterward is an additional challenge.

Here's how you do it in the ksh dialect that bash supports:

(( n = 3 + 4 )) if (( 5 <= n && n <= 10 )); then echo "$n is between 5 and 10" fi

# Output: 7 is between 5 and 10

This syntax is OK (ignoring some extra parentheses). But the semantics are bad in at least three ways.

There are edge cases that are confusing to developers. What happens when n doesn't look like an integer? What happens when n is 0x0A , 0123 , x , or x+8 ? The semantics are inefficient. The lack of true integers in shell means that the interpreter must do many string→int→string conversions. I (re-)discovered in January that this sublanguage is dangerously insecure. What if n is $(echo pwned) , or a[$(echo 42 >pwned.txt)] ?

I didn't have time to blog about the security issue, but the gist of it is that there's a hidden eval every time you use shell arithmetic.

A statement as simple as echo $((x + 1)) leads to the evaluation of arbitrary code in x ! An adversary that controls x also controls your system.

Oil Expressions

Oil will have more familiar syntax and simpler, safer semantics:

var n = 3 + 4 # 'var' for real ints, floats, dicts, etc. if (5 <= n and n <= 10) { # 'and' in expressions, && in commands echo "$n is between 5 and 10" }

# Output: 7 is between 5 and 10

That's all I want to say about the Oil language right now. That's because I don't know if I'll be able to finish it!

If you're interested in a saner and more powerful shell language, the last post has some notes on how to get involved. And feel free to ask questions on oilshell.zulipchat.com.

Links

How Do I Try it?

The home page links to the latest release. The wiki page How To Test OSH has details.

How Can I Help?

I wrote about this in the last post. Feedback is appreciated!

More Answers

I wrote the last FAQ in January 2018. It's by far the most popular page on this site.

Other blog posts with answers are labeled #FAQ and #why-a-new-shell.

Notes

[1] The line-counts/run.sh script in the oilshell/blog-code repo counts lines of bash code.