blog | oilshell.org

Oil Doesn't Confuse Flags and Files (Code and Data)

In last week's recap, I said I'd write a post tagged #real-problems when Oil addresses a problem with shell.

Here's another one: I just implemented shopt -u dashglob , which excludes files beginning with a dash (hyphen) from globs. This option addresses a decades-old problem with security implications.

Read on to see the problem, as well as the reasoning behind the simple solution.

A Typical Example

First, an attacker creates a file named -rf on the file system. Typical vectors are tarballs, git repos, and the like.

attacker$ touch -- -rf

Then an admin can be fooled into recursively deleting a tree, rather than the intended current directory:

admin$ rm *

Here are two traditional solutions:

admin$ rm -- * admin$ rm ./*

However these solutions require "opting in" on every line. Even if you have the knowledge, you might not do it 100% of the time. Or your co-worker may not write scripts in this style.

In Oil, the common/short thing should be right thing, so it excludes -rf by default:

oil-admin$ rm *

Usage

dashglob is Turned Off by Oil

This is a good time to review the difference between OSH and Oil.

When you run bin/osh , the shell behaves compatibly, like POSIX and bash. A glob will include a file like -rf .

, the shell behaves compatibly, like POSIX and bash. A glob will a file like . When you run bin/oil , it's like running bin/osh with shopt -s oil:all . The binary is the same, but an option group is enabled. The group oil:all disables the dashglob option, so globs will exclude files like -rf .

, it's like running with . The binary is the same, but an option group is enabled.

Other notes:

The dashglob name is consistent with bash's dotglob option, which controls whether globs include files like .vimrc , .oilrc , etc.

name is consistent with bash's option, which controls whether globs include files like , , etc. dashglob is also off in the less aggressive groups strict:all and oil:basic . I'll write more about them later.

What if I Want Globs to Include -rf ?

There are a few ways to return all files. You can simply re-enable dashglob globally:

oil$ shopt -s dashglob oil$ rm -- *

To change the behavior for a single command, you will be able to pass shopt a block (not implemented):

shopt -s dashglob { rm -- * } rm *

You can also use the more traditional solution:

oil$ rm ./*

Background

I've read about this issue dozens of times over the years. Here's a recent lobste.rs thread about it, where I linked David A. Wheeler's page Filenames and Pathnames in Shell: How to do it Correctly. That page recommends the explicit ./* to avoid the problem.

I knew I wanted Oil to address the problem without opting in, but I wasn't sure how. I filed issue 552 to keep track of it.

Over the next several weeks I got great feedback from Mateusz Czapliński, David A. Wheeler, and others. We considered a number of solutions, and ended up with this simple one. Appendix B explains why other solutions were rejected.

What Other Problems Can Be Fixed in Shell?

As I wrote in August, you can influence the Oil language! I take feedback from both experienced shell users and programmers who avoid shell.

So let us know if there are any other common shell problems that Oil should mitigate. Limitations of ShellCheck could be a source of inspiration.

Again, I've tagged these posts #real-problems. It's absolutely a goal for Oil to deal safely with untrusted data, including all the problems described in detail by Wheeler.

Related Work

nullglob is also on in Oil, and addresses the find . -name *.jpg issue.

is also on in Oil, and addresses the issue. I'm working on a new doc Simple Word Evaluation in Unix Shell, which describes shopt -s simple_word_eval . Again, the short thing should be the right thing.

. Again, the short thing should be the right thing. set -e / errexit is another hairy issue. I'd like users to test the solutions that Oil has so far. They still need work and documentation. See point #3 about the "ignored errexit problem" in this section of the last post, as well as the linked Reddit comments.

/ is another hairy issue. I'd like users to test the solutions that Oil has so far. They still need work and documentation. See point #3 about the "ignored problem" in this section of the last post, as well as the linked Reddit comments. CSTR is a proposal for a serialization format that solves more quoting problems. In Oil, you should be able to safely print paths to stdout and read them back in. They shouldn't be mangled in any way. The output should be text, not binary.

Appendix A: GLOBIGNORE= in bash

While discussing the -rf problem, I discovered that bash has a mechanism to omit such files:

GLOBIGNORE='-*'

Although I don't recall any script that does this. For example, I don't see it in a grep.app search for GLOBIGNORE.

But you may want to set GLOBIGNORE in your bash scripts if you deal with untrusted filenames.

I filed issue 609 to implement it in Oil. It will be useful for people who want to run the same program under Oil and bash.

It's labeled #help wanted, and would be a great way to start contributing to Oil. See this recent call for help for other ideas.

Appendix B: Assumptions That Oil Doesn't Make

I like the dashglob solution because it's consistent with dotglob and GLOBIGNORE , and it doesn't involve heuristics or false assumptions.

In particular, Oil doesn't assume: