More resources

The best book I know of for console hacks is Linux Server Hacks: 100 Industrial-Strength Tips and Tools . It's loaded with fiendishly creative scripts and shortcuts. (Almost all of the hacks apply to any Unix-derived OS, including Mac OS X.)

. It's loaded with fiendishly creative scripts and shortcuts. (Almost all of the hacks apply to any Unix-derived OS, including Mac OS X.) It turns out that my colleague Eric Eide did a masters thesis on an adaptive interface to a Unix shell, Valet. Valet uses a mixture of system knowledge and heuristics to "detect and correct the kinds of mistakes that experienced users make most frequently: typographical errors, file location errors, and minor syntactic errors." Neat!

Logging and mining console activity

Before you can exploit the principle of frequency, you need an unbiased record of what it is that you do most frequently. You might even be surprised by what your most frequent tasks are.

Fortunately, shells like bash already have some of that data in the form of the history command. The history command prints a list of recent commands. bash stores this history (by default) in ~/.bash_history .

You should periodically examine your frequently used commands, and find ways to execute them quickly. In many cases, a one-letter alias (e.g. e for emacs ) is surprisingly effective.

To find frequent commands, you'll need a script to compute counts for the contents of the history file. For this, I created a perl script called frequency :

my %counts = () ; while (my $cmd = <STDIN>) { chomp $cmd ; if (!$counts{$cmd}) { $counts{$cmd} = 1 ; } else { $counts{$cmd}++ ; } } foreach $k (keys %counts) { my $count = $counts{$k} ; print "$count $k

" ; }

It counts the number of times that each line occurs on stdin, and then dumps out a report.

For example:

$ cat ~/.bash_history | frequency | sort -rn | head 78 ls 25 recent 17 ls -l 16 mutt 12 cd matt.might.net 12 cd 10 echo hi 9 emacs .bash_profile 7 cd articles/ 5 sudo perl -MCPAN -e 'install DBD::mysql' ;

This example is clearly representative of about a day's worth of my recent activity, since my history only goes back for 500 lines. To get a truly representative sample, you'll probably want to increase your history size. Inside your .bash_profile , you should add:

export HISTFILESIZE=10000 # Record last 10,000 commands export HISTSIZE=10000 # Record last 10,000 commands per session

At the same time, while it's useful to look at the full commands, it's more instructive to look at just the commands themselves. For this, you can use cut :

$ cat ~/.bash_history | cut -d" " -f1 | frequency | sort -rn | head 84 ls 67 cd 48 recent 47 history 37 sudo 26 emacs 22 echo 15 mutt 15 ecmd 13 r

It shouldn't be any surprise to an experienced console user that nearly a third of all my commands are to list files and change directories. Since these commands are so common, I'll focus on optimizing those two. That's where my biggest "bang for buck" comes from.

Logging directories to MySQL

Switching directories is a common console task. In any given week, however, perhaps 80-90% of a your time will be spent in a small working set directories. This frequency provides a chance to optimize.

What you should do is log the directories visited, and then create a script for quickly switching to one of the most visited directories.

To log directories, you can use (1) a MySQL database to store the data, and (2) the PROMPT_COMMAND environment variable to insert a script into the console that runs every time a console prompt displays.

(Yes, this is an abuse of the PROMPT_COMMAND variable, but that's OK.)

Setting up the table is straightforward:

CREATE TABLE dircounts (path VARCHAR(255) PRIMARY KEY NOT NULL, count INT NOT NULL DEFAULT 0, time TIMESTAMP NOT NULL)

Then, you need a script, markdir.pl , to mark the current directory:

use strict; use warnings; use DBI; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Connection failed: $DBI::errstr"; my $qpwd = $dbh->quote($ENV{'PWD'}) ; $dbh->do("INSERT INTO dircounts(path,count) ". "VALUES ($qpwd,1) ". "ON DUPLICATE KEY UPDATE count=count+1") ; $dbh->disconnect();

Before each time the prompt is displayed, bash evaluates the contents of the environment variable PROMPT_COMMAND . This variable is just the right hook to monitor the current directory; for instance, in ~/.bash_profile , we can add:

export LASTDIR="/" function prompt_command { # Record new directory on change. newdir=`pwd` if [ ! "$LASTDIR" = "$newdir" ]; then /path/to/markdir.pl fi export LASTDIR=$newdir } export PROMPT_COMMAND="prompt_command"

Damping the frequency

You should damp the counts over time, so that if you spend one week accessing a directory a lot, but never again thereafter, it disappears from the frequent directories set quickly. Otherwise, you could have a defunct directory that sticks around for a long time after you've finished a major project.

To do this, I recommend a cronjob that runs a damping script once a week:

use strict; use warnings; use DBI; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Could not connect to database: $DBI::errstr"; my $qpwd = $dbh->quote($ENV{'PWD'}) ; # Cut all counts in half: $dbh->do("UPDATE dircounts SET count = count/2"); $dbh->disconnect();

The script exponentially decays counts, so that as soon as you stop using a directory regularly, its relative importance fades quickly.

Run crontab -e and then add:

0 0 * * 0 /path/to/damping/script

A frequency-adaptive change directory

With a database of all directory visits, it's easy to write a perl script, f , that lists these directories and permits changing to them with just a couple keystrokes:

#!/usr/bin/env perl use strict; use warnings; use DBI; my $target = shift ; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Could not connect to database: $DBI::errstr"; my $sth = $dbh->prepare("SELECT path ". "FROM dircounts ". "ORDER BY count DESC LIMIT 7") ; $sth->execute() ; my $count = 1 ; if (!$target) { print "+ change to directory with: f <number>

" ; } while (my @row = $sth->fetchrow_array) { if ($target) { if ($count == $target) { $sth->finish() ; $dbh->disconnect(); system(". cdto '@row'") ; exit ; } } else { my $home = $ENV{'HOME'} ; my $path = "@row" ; $path =~ s/$home/~/ ; print "${count}: $path

" ; } $count++ ; } $dbh->disconnect();

For example:

$ f + change to recent directory with: f <number> 1: ~ 2: ~/family/bertrand 3: ~/ucombinator/private/grants 4: ~/ucombinator/private/papers 5: ~/ucombinator/private/projects 6: ~/matt.might.net 7: ~/talks

A recency-adaptive change directory

Another useful perl script, r , lists the most recently accessed directories.

#!/usr/bin/env perl use strict; use warnings; use DBI; my $target = shift ; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Could not connect to database: $DBI::errstr"; my $span = 1000 * 60 * 60 * 7 ; my $q = "SELECT path ". "FROM dircounts ". "WHERE UNIX_TIMESTAMP(time) > UNIX_TIMESTAMP(now()) - $span ". "ORDER BY time DESC LIMIT 7" ; my $sth = $dbh->prepare($q) ; $sth->execute() ; my $count = 1 ; if (!$target) { print "+ change to directory with: r <number>

" ; } while (my @row = $sth->fetchrow_array) { if ($target) { if ($count == $target) { $sth->finish() ; $dbh->disconnect(); system(". cdto '@row'") ; exit ; } } else { my $home = $ENV{'HOME'} ; my $path = "@row" ; $path =~ s/$home/~/ ; print "${count}: $path

" ; } $count++ ; } $dbh->disconnect();

List recent files on directory change

In addition to changing directories, my other most common task was listing directory contents. Looking at the history file, I saw that an ls usually comes right after a cd .

This indicates that automatically listing directory contents after a directory switch should save time.

But, instead of listing all of the directory contents, it's more useful list only the most recently used files.

To do this, we need to augment PROMPT_COMMAND to check if the directory has changed, and if so, to run a command like ls -t | head -7 , which tells us the 7 most recently modified files in the directory:

export LASTDIR="/" function prompt_command { # Record new directory on change. newdir=`pwd` if [ ! "$LASTDIR" = "$newdir" ]; then /path/to/markdir.pl # List contents: ls -t | head -7 fi export LASTDIR=$newdir } export PROMPT_COMMAND="prompt_command"

Recency is actually a good approximation for frequency. Seeing the seven most recently modified files in a directory tends to quickly jog my mind and remind me where I left off.

Switching to the last directory on login

Another comman pattern in my work-flow is to open a second terminal window and then move to the directory I'm currently in. Once again, this change directory step should be eliminated by having new terminals open to the most recently used directory. Modifying ~/.bash_profile yet again provides the desired behavior.

# Change to most recently used directory: if [ -f ~/.lastdir ]; then cd "`cat ~/.lastdir`" fi export LASTDIR="/" function prompt_command { # Remember where we are: pwd > ~/.lastdir # Record new directory on change. newdir=`pwd` if [ ! "$LASTDIR" = "$newdir" ]; then /path/to/markdir.pl # List contents: ls -t | head -7 fi export LASTDIR=$newdir } export PROMPT_COMMAND="prompt_command"

The cdto script

The scripts above use a cdto command, which changes to a new directory from within a script. To accomplish this, cdto actually writes to the file ~/.lastdir , and starts a new shell.

#!/bin/bash echo + Switching to $* echo + Press CTRL-D to return to `pwd` echo $* > ~/.lastdir bash --login

The advantage of this is that a user can quickly change to a new directory, do work and "pop" back to the old directory by exiting the shell.

Recent files in emacs

Emacs can remember recent files as well. Add the following to ~/.emacs :

(require 'recentf) (recentf-mode 1) (setq recentf-max-menu-items 25) (global-set-key "\C-x\ \C-r" 'recentf-open-files)

Then C-x C-r to see recently opened files.

Conclusion

The principle of frequency isn't profound; but it is effective. Just because the console is a powerful way to compute doesn't mean we shouldn't look for ways to make it more efficient. Logging activity and then mining that data is a low-cost way for console users to discover their most frequent tasks. And, once those tasks are discovered, users can devise methods (or additional shell scripts) to make them go faster.

Update: Easier ways

One of the beautiful things about Unix is that you can use it for a decade and still find new things to learn every day.

Multiple readers have pointed out that much of these scripts could be replaced with simple, elegant shell hacks.

For instance, you can implement frequency as:

$ sort | uniq -c | sort -g

And, you can count commands with:

$ history | cut -c8- | sort | uniq -c | sort -rn | head

For jumping between directories, try autojump.

Related pages