Beware of shell globs
Written on .
Shell globs allow one to specify set of filenames with wildcard characters. This is really useful, but they have some rather unintuitive functions that could surprise you, and even cause big problems if you're unlucky enough.
Let's say you have a directory full of files, and maybe the names were even chosen by the user. It might look like this if you give funny people access to your system:
$ ls -l total 4 -rw-r--r-- 1 petr petr 0 Nov 26 16:06 * drwxr-xr-x 2 petr petr 4096 Nov 26 16:06 big_project -rw-r--r-- 1 petr petr 0 Nov 26 16:06 -r
Oh, what are all the empty files doing there? I don't need those files at all. I guess I'll just remove them. rm * should work. It will remove all files and leave my big_project alive as rm doesn't remove directories by default.
$ rm big_project rm: cannot remove 'big_project': Is a directory
Yeah, this looks safe. Rm by default doesn't remove directories. So let's execute rm, there should be nothing to surprise us, right?
$ rm * $ ls -l -rw-r--r-- 1 petr petr 0 Nov 26 16:06 -r
Wait, what? Where did the directory go? We verified rm won't delete directories by default, so what happened?
Can you figure it out? Why did our directory disappear?
# Setup
mkdir big_project && touch -- {\-r,\*} big_project/important-file-{1,2}
Some hints:
- Why did this file survive?
- When is the glob * expanded?
- How does the command look with * expanded?
The answer, while unintuitive, is pretty clear once we understand how shell globs work. Shell expands globs before the actual command is even called, so the program can't tell flags and arguments apart.
So what shell did when we called rm * was that it expanded to this:
$ rm \* big_project -r
Do you see the -r there? Yeah, this isn't a file (which is why the file -r wasn't removed), but a flag. Flag that says that rm should delete directories and all their content.
For rm, this is completely valid input and doesn't have a way to tell if you meant a file or a flag. And since shell doesn't escape -, it treats it as flag rather than file.
How can we protect against this? This isn't as easy as quoting parameters, since shell will expand variables within "", but won't expand globs. And you want the glob to expand, you don't want to write lengthy for loop for simple file removal.
Most basic unix tools you use every day (but not all of them, don't take this as granted) have a -- switch, that tells the tool that everything after this is not a switch, but a file. So rm in this case should be called like this: rm -- *, which would be absolutely safe for this situation.
If you're using tool which doesn't support this, prefixing the star with something is a very good defense. Just doing something like rm ./* will make all files look like ./-r, which is a filename, not a switch or flag.
Linux filenames are fun, as they can be everything (including newline) except null byte, so we have endless options here. Imagine what would happen if admin was deleting folder content with rm -rf * and someone left file named ~ there. (Modern shells escape this, it won't work :-)
If you ever write a script that operates on untrusted files, always make sure the command will do exactly the thing you wanted it to do. If possible, always use a tool like shellcheck that will warn you of those little surprises that shell is full of.