Introduction

... a means of interacting with a computer program where the user issues commands to the program in the form of successive lines of text (command lines) ... The interface is usually implemented with a command line shell, which is a program that accepts commands as text input and converts commands to appropriate operating system functions.

A command line interface (CLI) is a way to control your computer by typing in commands rather than clicking on buttons in a graphical user interface (GUI). Most computer users are only doing basic things like clicking on links, watching movies, and playing video games, and GUIs are fine for such purposes.



But to do industrial strength programming - to analyze large datasets, ship a webapp, or build a software startup - you will need an intimate familiarity with the CLI. Not only can many daily tasks be done more quickly at the command line, many others can only be done at the command line, especially in non-Windows environments. You can understand this from an information transmission perspective: while a standard keyboard has 50+ keys that can be hit very precisely in quick succession, achieving the same speed in a GUI is impossible as it would require rapidly moving a mouse cursor over a profusion of 50 buttons. It is for this reason that expert computer users prefer command-line and keyboard-driven interfaces.

make or rename 100 folders or files en masse

find all files of a given extension or any file that was created within the last week

log onto a computer remotely and access its files with ssh

copy files to your computer directly over the network (no external hard drive necessary!) with rsync

run a Perl or Python script

run one of the many programs that are only available on the command line

see all processes running on your computer or the space occupied by your folders

see or change the permissions on a file

parse a text file in any way imaginable (count lines, swap columns, replace words, etc.)

soundly encrypt your files or communications with gpg2

run your own web server on the Amazon cloud with nginx

unix

linux

100 Useful Unix Commands

Getting Started: Opening the Terminal

Applications > Utilities

The Definitive Guides to Unix, Bash, and the Coreutils

$ man bash $ info coreutils

$

The Unix Filestructure

/

/dir1/dir2/dir3/some_file

tree

/Users/username

http://www.example.com/abc/xyz

The Great Trailing Slash Debate

dir1/

Where Are You? - Your Path and How to Navigate through the Filesystem

HOME

/Users/username

.

./

/

$ ./myprogram

..

../

~

~/

$ pwd

$ cd /some/path

$ cd

$ mkdir

/Users/username

/Users

$ cd /Users

$ cd ..

$ cd /Users/username/..

$ cd /Users/username/../username/..

cd

$ echo $HOME # print the variable HOME /Users/username $ cd # cd is equivalent to cd $HOME $ pwd # print working directory shows us where we are /Users/username

$ unset HOME # unset HOME erases its value $ echo $HOME $ cd /some/path # cd into /some/path $ cd # take us HOME? $ pwd /some/path

/some/path

/Users/username

HOME

Gently Wading In - The Top 10 Indispensable Unix Commands

pwd ls cd mkdir echo cat cp mv rm man

man

pwd

$ man pwd

pwd

ls

$ man ls

command -f command --flag

$ man man

$ man woman No manual entry for woman

ls

$ cd $ ls $ ls -1 $ ls -hl $ ls -al

ls

ls -1

-h

-l

ls -hl

ls -h -l

$ whoami $ groups

--------- rwx------ rwxrwx--- rwxrwxrwx

chmod

drwxrwxrwx

-a

ls -al

ls -hl

ls

$ ls . dir1 .. dir2/*.txt dir3/A*.html

$ mkdir -p squirrel/mouse/fox

$ cd squirrel $ cd mouse $ cd fox $ ls

ls

$ ls squirrel/mouse/fox/ # use autocomplete to form this path quickly

Single Line Comments in Unix

#

$ # This is a comment. $ # If we put the pound sign in front of a command, it won't do anything: $ # ls -hl

#

The Primacy of Text Files, Text Editors

Exercise

nano

$ nano file.txt

1 x 4 b z 9

echo and cat

echo

cat

$ echo joe $ echo "joe"

$ cat file.txt

$ cat file.txt file2.txt

$ echo -n "joe" # suppress newline $ echo -e "joe\tjoe

joe" # interpret special chars ( \t is tab,

newline ) $ cat -n file.txt # print file with line numbers

cp, mv, and rm

cp

mv

rm

cp

$ cp file1 file2 $ cp -R dir1 dir2

-R

Question

$ cp -R dir1 ../../

Answer

mv

$ mv file1 file2

$ mv file1 dir1/dir2/file2

$ mv file1 dir1/dir2/

rm

$ rm file # removes a file $ rm -r dir # removes a file or directory $ rm -rf dir # force removal of a file or directory # (i.e., ignore warnings)

Variables in Unix

$ a=3 # This syntax is right (no whitespace) $ a = 3 # This syntax is wrong (whitespace) -bash: a: command not found

$ a=3 $ echo a a $ echo $a 3

$ a=3 # a can be an integer $ echo $a 3

$ a=joe # or a can be a string $ echo $a joe

$ a="joe joe" # Use quotes if you want a string with spaces $ echo $a joe joe

echo

$ a=3; b=4 $ echo $a $b 3 4 $ echo $a$b # mesh variables together as you like 34 $ echo "$a$b" # use quotes if you like 34 $ echo -e "$a\t$b" # the -e flag tells echo to interpret \t as a tab 3 4

$ var=5 $ joe=hello $var -bash: 5: command not found

$ joe="hello $var" $ echo $joe hello 5

$ joe='hello $var' $ echo $joe hello $var

$ d=dir1/dir2/dir3 $ ls $d $ cd $d

$ d=.. # this variable stores the directory one above us (relative path) $ cd $d/.. # cd two directories up

Escape Sequences

echo

\

$ a=3 $ echo $a 3 $ echo \$a $a $ echo "\$a" # use quotes if you like $a

$ echo \\\$a # escape the slash and the dollar sign \$a

$ echo ${a} 3

a=$11

a=$1

1

a=${11}

$ a=3 $ echo $a # variable a equals 3 3 $ echo $apple # variable apple is not set $ echo ${a}pple # this describes the variable a plus the string "pple" 3pple

Global Variables in Unix

HOME

$ set

HOME

PS1

TMPDIR

EDITOR

DISPLAY

HOME

/Users/username

PS1

$ PS1=':-) '

TMPDIR

sort

$TMPDIR

TMPDIR

EDITOR

DISPLAY

PERL5LIB

PYTHONPATH

R_LIBS

MATLABPATH

AWKPATH

LD_LIBRARY_PATH

# via: https://techcommons.stanford.edu/topics/git/show-git-branch-bash-prompt parse_git_branch() { git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/()/' } # add git branch to prompt export PS1="\[\033]0;\h:\W\007\]\h:\W\$(parse_git_branch)] "

The PATH

PATH

PATH

PATH

ls

which

$ which ls # on my work computer /bin/ls

$ which ls # on my home Mac /usr/local/Cellar/coreutils/8.20/libexec/gnubin/ls

ls

ls

ls

ls

PATH

PATH

PATH=/bin/dir1:/bin/dir2:/bin/dir3

/bin/dir1

/bin/dir2

/bin/dir1

PATH

$ echo $PATH

PATH

PATH

PATH

/bin

/usr/bin

/usr/local/bin

/mydir/newcommand

$ /mydir/newcommand

/mydir

PATH

$ PATH=/mydir:$PATH # add /mydir to the front of PATH - highest priority $ PATH=$PATH:/mydir # add /mydir to the back of PATH - lowest priority $ newcommand # now invoking newcommand is this easy

PATH

PATH

$ unset PATH # not advisable $ ls # now ls is not found -bash: ls: No such file or directory

ls

$ wget ftp://ftp.gnu.org/gnu/coreutils/coreutils-8.9.tar.xz $ tar -xvf coreutils-8.9.tar.xz

$ git clone git://git.sv.gnu.org/coreutils

Links

alias

$ ln -s /path/to/target/file mylink

mylink --> /path/to/target/file

ls -hl

$ rm mylink

ln

$ ln -s /path/to/target/file

file --> /path/to/target/file

PATH

PATH

PATH

PATH

$ ls -hl myprogram current -> version3 version1 version2 version3

~/MYLINK --> /some/long/and/complicated/path/to/an/often/used/directory

$ cd MYLINK

$ cd /some/long/and/complicated/path/to/an/often/used/directory

What is Scripting?

echo

cat

mkdir

cd

ls

$ mkdir tmp $ cd tmp $ pwd /Users/oliver/tmp $ touch myfile.txt # the command touch creates an empty file $ ls myfile.txt $ ls myfile_2.txt # purposely execute a command we know will fail ls: cannot access myfile_2.txt: No such file or directory

$ nano myscript.sh

# a first script mkdir tmp cd tmp pwd touch myfile.txt ls ls myfile_2.txt

create a directory called "tmp"

go into that directory

print our current path in the file system

make a new file called "myfile.txt"

list the contents of the directory we're in

specifically list the file "myfile_2.txt" (which doesn't exist)

$ ./myscript.sh -bash: ./myscript.sh: Permission denied

$ ls -hl myscript.sh -rw-r--r-- 1 oliver staff 75 Oct 12 11:43 myscript.sh

chmod

$ chmod u+x myscript.sh # add executable(x) permission for the user(u) only $ ls -hl myscript.sh -rwxr--r-- 1 oliver staff 75 Oct 12 11:43 myscript.sh

$ ./myscript.sh /Users/oliver/tmp/tmp myfile.txt ls: cannot access myfile_2.txt: No such file or directory

tmp/myfile.txt

$ ls myfile.txt myscript.sh tmp $ ls tmp myfile.txt

cd

$ pwd /Users/oliver/tmp

/Users/oliver/tmp/tmp

/Users/oliver/tmp

File Suffixes in Unix

.txt - for text files

- for text files .html - for html files

- for html files .sh - for shell scripts

- for shell scripts .pl - for Perl scripts

- for Perl scripts .py - for Python scripts

- for Python scripts .cpp - for c++ code

ls

find

$ ls *.txt

$ find . -name "*.txt"

Update

#!/usr/bin/env python def yo(): print('hello')

>>> import hello

>>> import hi Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named hi

The best way to handle this revolves around the core question of whether the file should be a command or a library. Libraries have to have the extension, and commands should not, so making a tiny command wrapper that handles parsing options and then calls the API from the other, imported one is correct.

PATH

PATH

#!/usr/bin/env python def yo(): print('hello')

PATH

#!/usr/bin/env python import hello hello.yo()

Hat tip

The Shebang

$ cat myscript_1.sh # a bash script echo "hello kitty"

$ cat myscript_1.pl # a Perl script print "hello kitty

";

bash

$ bash ./myscript_1.sh # use bash for bash scripts hello kitty

perl

$ perl ./myscript_1.pl # use Perl for Perl scripts hello kitty

$ ./myscript_1.pl # this won't work ./myscript_1.pl: line 1: print: command not found

$ bash ./myscript_1.pl # let's purposefully do it backwards ./myscript_1.pl: line 1: print: command not found

$ perl ./myscript_1.sh String found where operator expected at ./myscript_1.sh line 1, near "echo "hello kitty"" (Do you need to predeclare echo?) syntax error at ./myscript_1.sh line 1, near "echo "hello kitty"" Execution of ./myscript_1.sh aborted due to compilation errors.

bash

perl

$ which perl /usr/bin/perl

$ which bash /bin/bash

perl

bash

/bin

#!

$ cat myscript_1.sh #!/bin/bash echo "hello kitty"

$ cat myscript_1.pl #!/usr/bin/perl print "hello kitty

";

$ ./myscript_1.sh hello kitty $ ./myscript_1.pl hello kitty

perl

$ cat myscript_1.pl #!/usr/bin/env perl print "hello kitty

";

env

/usr/bin

env

perl

perl

PATH

perl

/usr/bin/perl

PATH

perl

/usr/bin

env

perl

cat

$ g++ MyProgram.cpp -o MyProgram

cat

ls

$ cat $( which ls ) # don't do this!

bash

sh

The Bourne shell ( sh ) is a shell, or command-line interpreter, for computer operating systems. The shell is a command that reads lines from either a file or the terminal, interprets them, and generally executes other commands. It is the program that is running when a user logs into the system ... Commands can be typed directly to the running shell or can be put into a file and the file can be executed directly by the shell

sh

/bin/sh

$ sh ./myscript

$ sh

sh

$ ./myscript

sh

sh

bash

/bin/bash

The Bourne-Again SHell ( bash ) a Unix shell written by Brian Fox for the GNU Project as a free software replacement for the Bourne shell. bash is an sh-compatible command language interpreter that executes commands read from the standard input or from a file ... There are some subtle differences between bash and traditional versions of sh

sh

bash

$ ps -p $$

chmod

u - user

- user g - group

- group o - other/world

r - read

- read w - write

- write x - execute

chmod entity+permissiontype

chmod entity-permissiontype

$ chmod u+x myfile # make executable for you $ chmod g+rxw myfile # add read write execute permissions for the group $ chmod go-wx myfile # remove write execute permissions for the group # and for everyone else (excluding you, the user)

$ chmod a-rwx myfile # remove all permissions for you, the group, # and the rest of the world

chmod

$ chmod 777 myfile # grant all permissions (rwxrwxrwx) $ chmod 755 myfile # reserve write access for the user, # but grant all other permissions (rwxr-xr-x)

ssh

chmod

ssh

ssh

ssh username@host

$ ssh username@myhost.university.edu

ssh

ssh

$ ssh username@myhost.university.edu "ls -hl"

ssh

~/.ssh/config

ssh

ssh

$ mkdir -p ~/.ssh $ cd ~/.ssh $ ssh-keygen -t rsa -f localkey

~/.ssh/localkey.pub

~/.ssh/localkey

ssh

$ ssh myusername@myserver.com

Host Myserver HostName myserver.com User myusername IdentityFile ~/.ssh/localkey

cat

~/.ssh/authorized_keys

ssh

$ ssh Myserver

Settings > SSH Keys > Add SSH Key

ssh

ssh

Saving to a File; Stdout and Stderr

$ echo joe > junk.txt # save to file $ cat junk.txt joe

$ echo joe >> junk.txt # append to already-existing file $ cat junk.txt joe joe

$ ./myscript.sh > out.txt mkdir: cannot create directory ‘tmp’: File exists ls: cannot access myfile_2.txt: No such file or directory

$ cat out.txt /Users/oliver/tmp/tmp myfile.txt

2>

$ # save the output into out.txt and the error into err.txt $ ./myscript.sh > out.txt 2> err.txt $ cat out.txt /Users/oliver/tmp/tmp myfile.txt $ cat err.txt mkdir: cannot create directory ‘tmp’: File exists ls: cannot access myfile_2.txt: No such file or directory

./myjob --instance 758 > out758.o 2> out758.e

cout << "some output" << endl; cerr << "some error" << endl;

print STDOUT "some output

"; print STDERR "some error

";

import sys sys.stdout.write("some output

") sys.stderr.write("some error

")

More on Stdout and Stderr; Redirection

1> # save stdout to (plain old > also works) 2> # save stderr to

$ ./myscript.sh 1> out.o 2> out.e $ ./myscript.sh > out.o 2> out.e # these two lines are identical

&1 # standard out stream &2 # standard error stream

#!/bin/bash # version 1 echo "hello kitty"

#!/bin/bash # version 2 echo "hello kitty" > somefile.txt

#!/bin/bash # version 3 echo "hello kitty" > &1

#!/bin/bash # version 4 echo "hello kitty" > &2

#!/bin/bash # version 5 echo "hello kitty" > 1

version 1 - echo "hello kitty" to stdout

- echo "hello kitty" to stdout version 2 - echo "hello kitty" to the file somefile.txt

- echo "hello kitty" to the file somefile.txt version 3 - same as version 1

- same as version 1 version 4 - echo "hello kitty" to sterr

- echo "hello kitty" to sterr version 5 - echo "hello kitty" to the file named 1

$ # output saved to file but error printed to console $ ./hellokitty.sh > junk.txt hello kitty

$ ./somescript.sh 2> &1 # redirect stderr to stdout

Conditional Logic

$ a=joe $ if [ $a == "joe" ]; then echo hello; fi hello

$ a=joe $ if [ $a == "joe" ]; then echo hello; echo hello; echo hello; fi hello hello hello

if [ condition ]; then ... ; fi

then

fi

#!/usr/bin/env perl my $a="joe"; if ( $a eq "joe" ) { print "hello

"; print "hello

"; print "hello

"; }

if

else

elif

#!/bin/bash a=joe if [ $a == "joe" ]; then echo hello; elif [ $a == "doe" ]; then echo goodbye; else echo "ni hao"; fi

case

It is like a struggle between a tiger and a shark,

each is supreme in his own element,

but helpless in that of the other.

J.J. Thomson, 1925

File Test Operators; Return or Exit Status

if

if

$ touch emptyfile # create an empty file $ echo joe > nonemptyfile # create a non-empty file

$ file=emptyfile $ if [ -e $file ]; then echo "exists"; if [ -s $file ]; then echo "non-0"; fi; fi exists

$ file=nonemptyfile $ if [ -e $file ]; then echo "exists"; if [ -s $file ]; then echo "non-0"; fi; fi exists non-0

$?

$ echo joe joe $ echo $? # query exit code of previous command 0

$ cat test_exitcode.sh #!/bin/bash sleep 10

$ ./test_exitcode.sh; # let it run $ echo $? 0

$ ./test_exitcode.sh; # interrupt it ^C $ echo $? 130

if

$ ./test_exitcode.sh $ if [ $? == 0 ]; then echo "program succeeded"; else echo "program failed"; fi program succeeded

$ ./test_exitcode.sh; ^C $ if [ $? == 0 ]; then echo "program succeeded"; else echo "program failed"; fi program failed

Question

$ if echo joe; then echo joe; fi joe joe

echo joe

if

$ outputdir=nonexistentdir # set output dir equal to a nonexistent dir $ if ! cd $outputdir; then echo "couldnt cd into output dir"; fi -bash: pushd: nonexistentdir: No such file or directory couldnt cd into output dir

$ mkdir existentdir # make a test directory $ outputdir=existentdir $ if ! cd $outputdir; then echo "couldnt cd into output dir"; fi $ # no error - now we're in the directory existentdir

!

cd

cd

if ! cd $outputdir; then echo "[error] couldn't cd into output dir"; exit; fi

cd

&&

||

if

$ touch file{1..4} $ ls file1 file2 file3 file4

&&

$ ( ls file1 ) && ( ls file2 ) && ( ls file3 ) && ( ls file4 ) file1 file2 file3 file4

$ ( ls file1 ) && ( ls file2 ) && ( ls fileX ) && ( ls file4 ) file1 file2 ls: cannot access fileX: No such file or directory

||

$ ( ls file1 ) || ( ls file2 ) || ( ls file3 ) || ( ls file4 ) file1

$ ( ls fileX ) || ( ls fileY ) || ( ls fileZ ) || ( ls file4 ) ls: cannot access fileX: No such file or directory ls: cannot access fileY: No such file or directory ls: cannot access fileZ: No such file or directory file4

Basic Loops

$ for i in 1 2 3; do echo $i; done 1 2 3

for variable in list; do ... ; done

$ for i in 1 2 hello; do echo $i; done 1 2 hello

$ for i in {1..10}; do echo -n "$i "; done; echo 1 2 3 4 5 6 7 8 9 10

$ echo {1..10}

$ for i in *.txt; do echo $i; done

$ ls *.txt

do

done

$ mkdir -p myfolder{1..3}/{X,Y}

$ j=0; for i in myfolder*/*; do echo "*** "$i" ***"; touch ${i}/a_${j}.txt ${i}/b_${j}.txt; ((j++)); done

((j++))

j

$i

$ for i in myfolder*/*/b*; do echo "*** "$i" ***"; ln -s $i; done

b_0.txt -> myfolder1/X/b_0.txt b_1.txt -> myfolder1/Y/b_1.txt b_2.txt -> myfolder2/X/b_2.txt b_3.txt -> myfolder2/Y/b_3.txt b_4.txt -> myfolder3/X/b_4.txt b_5.txt -> myfolder3/Y/b_5.txt

$ for i in myfolder*/X/a*.txt; do echo "*** "$i" ***"; j=$( echo $i | sed 's|\.txt|\.html|' ); echo $j; mv $i $j; echo; done

$ for i in $( echo $PATH | tr ":" " " ); do echo "*** "$i" ***"; ls $i | head; echo; done | less

PATH

$ for i in *.txt; do { echo $i; gzip $i & }; done

while condition; do ... ; done

$ x=1; while ((x <= 3)); do echo $x; ((x++)); done 1 2 3

$ cat junk.txt 1 2 3

$ while read x; do echo $x; done < junk.txt 1 2 3

Arguments to a Script

#!/bin/bash echo hello

$ chmod 755 hellokitty.sh $ ./hellokitty.sh hello

#!/bin/bash echo hello $1

$ ./hellokitty.sh kitty hello kitty

$1

$2

#!/bin/bash echo $0 echo hello $1 $4

$ ./hellokitty.sh my sweet kitty cat ./hellokitty.sh hello my cat

$n

$0

command --flag1 1 --flag2 1 --flag3 5

command -f1 1 -f2 1 -f3 5

case

if

#!/bin/bash helpmessage = "This script showcases how to read arguments" ### get arguments # while input array size greater than zero while (( $# > 0 )) ; do if [ " $1 " == "-h" -o " $1 " == "-help" -o " $1 " == "--help" ] ; then shift ; echo " $helpmessage " exit ; elif [ " $1 " == "-f1" -o " $1 " == "--flag1" ] ; then # store what's passed via flag1 in var1 shift ; var1 = $1 ; shift elif [ " $1 " == "-f2" -o " $1 " == "--flag2" ] ; then shift ; var2 = $1 ; shift elif [ " $1 " == "-f3" -o " $1 " == "--flag3" ] ; then shift ; var3 = $1 ; shift # if unknown argument, just shift else shift fi done ### main # echo variable if not empty if [ ! -z $var1 ] ; then echo "flag1 passed " $var1 ; fi if [ ! -z $var2 ] ; then echo "flag2 passed " $var2 ; fi if [ ! -z $var3 ] ; then echo "flag3 passed " $var3 ; fi

$# is the size of our input argument array

is the size of our input argument array shift pops an element off of our array (the same as in Perl)

pops an element off of our array (the same as in Perl) exit exits the script

exits the script -o is logical OR in unix

is logical OR in unix -z checks if a variable is empty

$ ./test_args --flag1 x -f2 y --flag3 zzz flag1 passed x flag2 passed y flag3 passed zzz

shift

$1

shift

$1

$ ./test_args --flag3 x --flag1 zzz flag1 passed zzz flag3 passed x

$ ./test_args --flag2 asdf flag2 passed asdf

@ARGV

sys.argv

Bash Perl Python Description $0 $0 sys.argv[0] Name of Script Itself $* String Containing All Input Arguments ("$@") @ARGV sys.argv Array or List Containing All Input Arguments [1] $1 $ARGV[0] sys.argv[1] First Argument $2 $ARGV[1] sys.argv[2] Second Argument

Multi-Line Comments, Multi-Line Strings in Bash

if

# multi-line comment if false; then echo hello echo hello echo hello fi

cat <<_EOF_ Usage: $0 --flag1 STRING [--flag2 STRING] [--flag3 STRING] Required Arguments: --flag1 STRING This argument does this Options: --flag2 STRING This argument does that --flag3 STRING This argument does another thing _EOF_

Source and Export

Question

$ cat ./test_src.sh #!/bin/bash myvariable=54 echo $myvariable

$ ./test_src.sh 54 $ echo $myvariable

$ source ./test_src.sh 54 $ echo $myvariable 54

$ . ./test_src.sh # this is the same as "source ./test_src.sh" 54

$ cat ./test_src_2.sh #!/bin/bash echo $myvariable

$ ./test_src_2.sh

export

$ cat ./test_src.sh #!/bin/bash export myvariable=54 # export this variable echo $myvariable

$ ./test_src.sh 54 $ ./test_src_2.sh

source

$ source ./test_src.sh 54 $ ./test_src_2.sh 54

source

source

export

Dotfiles (.bashrc and .bash_profile)

$ touch .test

ls

ls -al

bash - .bashrc

vim - .vimrc

git - .gitconfig

HOME

PATH

/some/path/to/prog

PATH

export PATH=/some/path/to/prog:$PATH

export

PATH

PATH

PATH

PATH=/apps/python/2.7.6/bin:$PATH # use this version of Python PATH=/apps/R/3.1.2/bin:$PATH # use this version of R PATH=/apps/gcc/4.6.0/bin/:$PATH # use this version of gcc export PATH

PATH

HOME

$ ls -1 .dotfiles/bash/ bash_aliases_local bash_aliases_share bash_functions_share bash_inirun_local bash_paths_local bash_settings_local bash_settings_share bash_welcome_local bash_welcome_share

source

# the directory where bash configuration files reside INIT_DIR = " ${ HOME } /.dotfiles/bash" # to make local configurations, add these files into this directory: # bash_aliases_local # bash_paths_local # bash_settings_local # bash_welcome_local # this line, e.g., protects the functionality of rsync by only turning on the below if the shell is in interactive mode # In particular, rsync fails if things are echo-ed to the terminal [[ " $- " ! = *i* ]] && return # bash welcome if [ -e " ${ INIT_DIR } /bash_welcome_local" ] ; then cat ${ INIT_DIR } /bash_welcome_local elif [ -e " ${ INIT_DIR } /bash_welcome_share" ] ; then cat ${ INIT_DIR } /bash_welcome_share fi #--------------------LOCAL------------------------------ # aliases local if [ -e " ${ INIT_DIR } /bash_aliases_local" ] ; then source " ${ INIT_DIR } /bash_aliases_local" echo "bash_aliases_local loaded" fi # settings local if [ -e " ${ INIT_DIR } /bash_settings_local" ] ; then source " ${ INIT_DIR } /bash_settings_local" echo "bash_settings_local loaded" fi # paths local if [ -e " ${ INIT_DIR } /bash_paths_local" ] ; then source " ${ INIT_DIR } /bash_paths_local" echo "bash_paths_local loaded" fi #---------------SHARE----------------------------- # aliases share if [ -e " ${ INIT_DIR } /bash_aliases_share" ] ; then source " ${ INIT_DIR } /bash_aliases_share" echo "bash_aliases_share loaded" fi # settings share if [ -e " ${ INIT_DIR } /bash_settings_share" ] ; then source " ${ INIT_DIR } /bash_settings_share" echo "bash_settings_share loaded" fi # functions share if [ -e " ${ INIT_DIR } /bash_functions_share" ] ; then source " ${ INIT_DIR } /bash_functions_share" echo "bash_functions_share loaded" fi

scp

rsync

PATH

.gitconfig -> .dotfiles/.gitconfig .vimrc -> .dotfiles/.vimrc

Working Faster with Readline Functions and Key Bindings

$ set -o emacs # Set emacs-style key bindings (this is the default) $ set -o vi # Set vi-style key bindings

Cntrl-a - jump cursor to beginning of line

- jump cursor to beginning of line Cntrl-e - jump cursor to end of line

- jump cursor to end of line Cntrl-k - delete to end of line

- delete to end of line Cntrl-u - delete to beginning of line

- delete to beginning of line Cntrl-w - delete back one word

- delete back one word Cntrl-y - paste (yank) what was deleted with the above shortcuts

- paste (yank) what was deleted with the above shortcuts Cntrl-r - reverse-search history for a given word

- reverse-search history for a given word Cntrl-c - kill the process running in the foreground; don't execute current line on the command line

- kill the process running in the foreground; don't execute current line on the command line Cntrl-z - suspend the process running in the foreground

- suspend the process running in the foreground Cntrl-l - clear screen. (this has an advantage over the unix command clear in that it works in the Python, MySQL, and other shells)

- clear screen. (this has an advantage over the unix command in that it works in the Python, MySQL, and other shells) Cntrl-d - end of transmission (in practice, often synonymous with quit - e.g., exiting the Python or MySQL shells)

- end of transmission (in practice, often synonymous with quit - e.g., exiting the Python or MySQL shells) Cntrl-s - freeze screen

- freeze screen Cntrl-q - un-freeze screen

$ bind -P # show all Readline Functions and their key bindings $ bind -l # show all Readline Functions

forward-word - jump cursor forward a word

- jump cursor forward a word backward-word - jump cursor backward a word

- jump cursor backward a word history-search-backward - scroll through your bash history backward

- scroll through your bash history backward history-search-forward - scroll through your bash history forward

Meta-f - jump forward one word

- jump forward one word Meta-b - jump backward one word

bind

Cntrl-forward-arrow - forward-word

- forward-word Cntrl-backward-arrow - backward-word

- backward-word up-arrow - history-search-backward

- history-search-backward down-arrow - history-search-forward

# make cursor jump over words bind '"\e[5C": forward-word' # control+arrow_right bind '"\e[5D": backward-word' # control+arrow_left # make history searchable by entering the beginning of command # and using up and down keys bind '"\e[A": history-search-backward' # arrow_up bind '"\e[B": history-search-forward' # arrow_down

Tip

More on Key Bindings, the ASCII Table, Control-v

The American Standard Code for Information Interchange (ASCII) is a character-encoding scheme originally based on the English alphabet that encodes 128 specified characters—the numbers 0-9, the letters a-z and A-Z, some basic punctuation symbols, some control codes that originated with Teletype machines, and a blank space—into the 7-bit binary integers.

Binary Oct Dec Hex Abbr [a] [b] [c] Name 000 0000 000 0 00 NUL ␀ ^@ \0 Null character 000 0001 001 1 01 SOH ␁ ^A Start of Header 000 0010 002 2 02 STX ␂ ^B Start of Text 000 0011 003 3 03 ETX ␃ ^C End of Text 000 0100 004 4 04 EOT ␄ ^D End of Transmission 000 0101 005 5 05 ENQ ␅ ^E Enquiry 000 0110 006 6 06 ACK ␆ ^F Acknowledgment 000 0111 007 7 07 BEL ␇ ^G \a Bell 000 1000 010 8 08 BS ␈ ^H \b Backspace 000 1001 011 9 09 HT ␉ ^I \t Horizontal Tab 000 1010 012 10 0A LF ␊ ^J

Line feed 000 1011 013 11 0B VT ␋ ^K \v Vertical Tab 000 1100 014 12 0C FF ␌ ^L \f Form feed 000 1101 015 13 0D CR ␍ ^M \r Carriage return 000 1110 016 14 0E SO ␎ ^N Shift Out 000 1111 017 15 0F SI ␏ ^O Shift In 001 0000 020 16 10 DLE ␐ ^P Data Link Escape 001 0001 021 17 11 DC1 ␑ ^Q Device Control 1 (oft. XON) 001 0010 022 18 12 DC2 ␒ ^R Device Control 2 001 0011 023 19 13 DC3 ␓ ^S Device Control 3 (oft. XOFF) 001 0100 024 20 14 DC4 ␔ ^T Device Control 4 001 0101 025 21 15 NAK ␕ ^U Negative Acknowledgement 001 0110 026 22 16 SYN ␖ ^V Synchronous idle 001 0111 027 23 17 ETB ␗ ^W End of Transmission Block 001 1000 030 24 18 CAN ␘ ^X Cancel 001 1001 031 25 19 EM ␙ ^Y End of Medium 001 1010 032 26 1A SUB ␚ ^Z Substitute 001 1011 033 27 1B ESC ␛ ^[ \e Escape 001 1100 034 28 1C FS ␜ ^\ File Separator 001 1101 035 29 1D GS ␝ ^] Group Separator 001 1110 036 30 1E RS ␞ ^^ Record Separator 001 1111 037 31 1F US ␟ ^_ Unit Separator 111 1111 177 127 7F DEL ␡ ^? Delete

$ ^[[A

$ ^[[B

$ ^[[5D

# make cursor jump over words bind '"\e[5C": forward-word' # control+arrow_right bind '"\e[5D": backward-word' # control+arrow_left # make history searchable by entering the beginning of command # and using up and down keys bind '"\e[A": history-search-backward' # arrow_up bind '"\e[B": history-search-forward' # arrow_down

bind -l

Note

Terminal > Preferences > Settings > Keyboard

Aliases, Functions

alias

alias c="cat"

$ cat file.txt

$ c file.txt

alias

alias cp="cp -R"

alias mkdir="mkdir -p"

alias c="cat" alias s="less -S" alias l="ls -hl" alias lt="ls -hlt" alias ll="ls -al" alias rr="rm -r" alias r="readlink -m" alias ct="column -t" alias ch="chmod -R 755" alias chh="chmod -R 644" alias grep="grep --color" alias yy="rsync -azv --progress" # remove empty lines with white space alias noempty="perl -ne '{print if not m/^(\s*)$/}'" alias awkf="awk -F'\t'" alias length="awk '{print length}'"

$ git add . $ git commit -m "some message" $ git push

source

# stage all changes, commit, then push gup () { local mymessage = "next update" ; # if $1 not zero length if [ ! -z " $1 " ] ; then mymessage = $1 fi git add . git commit -m " $mymessage " git push }

local

git add

git commit

extract () { if [ -f $1 ] ; then case $1 in *.tar.bz2 ) tar xvjf $1 ;; *.tar.gz ) tar xvzf $1 ;; *.bz2 ) bunzip2 $1 ;; *.rar ) unrar x $1 ;; *.gz ) gunzip $1 ;; *.tar ) tar xvf $1 ;; *.tbz2 ) tar xvjf $1 ;; *.tgz ) tar xvzf $1 ;; *.zip ) unzip $1 ;; *.Z ) uncompress $1 ;; *.7z ) 7z x $1 ;; * ) echo "don't know how to extract ' $1 '..." ;; esac else echo "' $1 ' is not a valid file!" fi }

case

if

$ extract my_compressed_file.gz

$ extract my_compressed_file.bz2

$ extract my_compressed_tarball.tar.gz

The Top 20 Indispensable Unix Commands

pwd ls cd mkdir echo cat cp mv rm man head tail less more sort grep which chmod history clear

which

chmod

clear

head and tail

head

tail

$ head myfile.txt # print the first 10 lines of the file $ head -1 myfile.txt # print the first line of the file $ head -50 myfile.txt # print the first 50 lines of the file

$ tail myfile.txt # print the last 10 lines of the file $ tail -1 myfile.txt # print the last line of the file

cat

head

tail

head

cat

$ cat hello.txt hello hello hello

$ cat kitty.txt kitty kitty kitty

$ cat hello.txt kitty.txt hello hello hello kitty kitty kitty

head

$ head -2 hello.txt kitty.txt ==> hello.txt <== hello hello ==> kitty.txt <== kitty kitty

$ head *

less and more

cat

less

$ less myfile.txt # view the file page by page

less

/

less

$ less -S myfile.txt # allow horizontal scrolling

cat myfile.txt | column -t | less -S

column

cat myfile.txt | column -s' ' -t | less -S

less

more

more

less

more

grep

sort

grep

grep

$ grep apple myfile.txt # return lines of file with the text apple

grep

$ grep -n apple myfile.txt # include the line number $ grep -i apple myfile.txt # case insensitive matching $ grep --color apple myfile.txt # color the matching text

$ grep -A1 apple myfile.txt # return lines with the match, # as well as 1 after

$ grep -B2 apple myfile.txt # return lines with the match, # as well as 2 before

$ grep -C3 apple myfile.txt # return lines with the match, # as well as 3 before and after.

grep

$ grep -v apple myfile.txt # return lines that don't contain apple

$ grep -R apple mydirectory/ # search for apple in any file in mydirectory

$ grep -m 2 apple myfile.txt

grep

egrep

$ egrep "apple|orange" myfile.txt # return lines with apple OR orange

(Image credit: O'Reilly Media)

sort

$ cat testsort.txt vfw 34 awfjo a 4 2 f 10 10 beb 43 c f 2 33 f 1 ?

$ sort testsort.txt a 4 2 beb 43 c f 1 ? f 10 10 f 2 33 vfw 34 awfjo

sort

-n

-r

sort -kn,m

sort -k2,2

$ sort -k2,2n testsort.txt f 1 ? f 2 33 a 4 2 f 10 10 vfw 34 awfjo beb 43 c

Question

$ sort -k1,1r -k2,2n testsort.txt vfw 34 awfjo f 1 ? f 2 33 f 10 10 beb 43 c a 4 2

Answer

sort

-u

$ sort -u testsort.txt # sort uniquely

-T

$ sort -T /my/tmp/dir testsort.txt # sort using a designated tmp directory

sort

TMPDIR

sort

history

history

$ history

(reverse-i-search)`':

$ history | grep apple

$ history | tail

HISTFILE - the path to the history file

- the path to the history file HISTSIZE - how many commands to store in the history list (memory)

- how many commands to store in the history list (memory) HISTFILESIZE - how many commands to store in the history file

HISTFILE

~/.bash_history

history

$ history | tail $ tail ~/.bash_history

# append history to history file as you type in commands shopt -s histappend export PROMPT_COMMAND='history -a'

If set, the history list is appended to the history file when the shell exits, rather than overwriting the history file.

shopt -s histappend

To append every line to history individually set:

PROMPT_COMMAND='history -a'

With these two settings, a new shell will get the history lines from all previous shells instead of the default 'last window closed'>history (the history file is named by the value of the HISTFILE variable)

shopt

history -a

PROMPT_COMMAND

$ history -c # clear the history list in memory

$ echo > ~/.bash_history

ls

cd

alias n="history | tail -2 | head -1 | tr -s ' ' | cut -d' ' -f3- | awk '{print \"# \"\$0}' >> notes"

$ ./long_hard-to-remember_command --with_lots --of_flags > poorly_named_file $ n

$ cat notes # ./long_hard-to-remember_command --with_lots --of_flags > poorly_named_file

Piping in Unix

|

$ cat file.txt | sort | less

cat

sort

sort

less

history

$ history | tail # show the last 10 lines of history $ history | grep apple # find commands in history containing "apple"

cat

$ cat -n file.txt | head -37 | tail -1 # print row 37 $ cat -n file.txt | awk 'NR==37' # print row 37

awk

$ cat test.txt 1 c 3 c 2 t 1 c

$ cat test.txt | sort -u | tee tmp.txt | wc -l 3

$ cat tmp.txt 1 c 2 t 3 c

tee

sort

wc -l

$ cat file.txt | cut -f2 | sort -u | wc -l

$ head -1 file.txt | tr "\t" "

" | nl -b a

$ bind -P | grep history-search-backward

$ cat file.txt | awk '$1==1' | cut -f2-4

$ history | grep "|" | less

$ history | tail -1 1022 history | tail -1

$ echo Hello Hello $ history | tail -2 | head -1 1023 echo Hello

$ history | tail -2 | head -1 | tr -s ' ' 1023 echo Hello

-s

cut

cut

$ history | tail -2 | head -1 | tr -s ' ' | cut -d' ' -f3- | awk '{print "# "$0}' # echo Hello

Command Substitution

$ d=$( pwd )

$ d=`pwd`

$ len=$( cat file.txt | wc -l )

cd

# save scripts directory in the variable d d=$( dirname $( readlink -m $0 ) )

readlink

dirname

head

$ head *.txt

$ head $( ls *.txt ) $ head $( find . -maxdepth 1 -name "*.txt" )

PATH

$ cat $( which myscript.pl )

seq

$ for i in $( seq 1 3 ); do echo $i; done 1 2 3

Question

$ for i in $( cat file.txt | cut -f3 | sort -u ); do echo $i; done

$ for i in $( ls *.txt | grep -v apple ); do echo $i; done

Answer

echo

$ touch {1..3}.txt

$ for i in {1..3}.txt; do j=$( echo $i | sed 's|\.txt|\.html|' ); cmd="mv $i $j"; echo "Run command: $cmd"; echo $cmd | bash; done

bash

PATH

$ for i in $( echo $PATH | tr ":" "

" | head -3 ); do echo "*** "$i" ***"; echo "This folder has "$( ls $i | wc -l )" elements and is "$( du -sh $i | cut -f1 )" large"; echo; done *** /usr/local/sbin *** This folder has 0 elements and is 4.0K large *** /usr/local/bin *** This folder has 7 elements and is 32K large *** /usr/sbin *** This folder has 130 elements and is 8.6M large

Process Substitution

... process substitution is a form of inter-process communication that allows the input or output of a command to appear as a file. The command is substituted in-line, where a file name would normally occur, by the command shell. This allows programs that normally only accept files to directly read from or write to another program.

$ cat <( head -1 file.txt ) <( tail file.txt )

cat

<( )

$ head -1 file.txt; tail file.txt

column -t

$ cat <( head -1 file.txt ) <( tail file.txt ) | column -t

$ { head -1 file.txt; tail file.txt; } | column -t

column -t

awk

$ cat test.txt | awk '{print $2"\t"$2}'

$ paste <( cat test.txt | cut -f2 ) <( cat test.txt | cut -f2 )

$ cat <( head -1 temp.txt ) <( cat temp.txt | sed '1d' | sort -k1,1n )

#!/bin/bash myfile = $1 len = $( cat $myfile | wc -l ) echo "Your file is $len lines long."

$ ./test_script 1.txt Your file is 11 lines long.

$ gzip 1.txt

$ ./test_script <( gunzip --stdout 1.txt.gz ) Your file is 11 lines long.

Interlude - Bell Laboratories 1982 Unix Video

Processes and Running Processes in the Background

$ ps # show process info $ ps -f # show verbose process info

$ sleep 60 & [1] 6846 $ sleep 30 & [2] 6847

$ ps -f UID PID PPID C STIME TTY TIME CMD 501 5166 5165 0 Wed12PM ttys008 0:00.20 -bash 501 6846 5166 0 10:57PM ttys008 0:00.01 sleep 60 501 6847 5166 0 10:57PM ttys008 0:00.00 sleep 30

sleep

tree

$ pstree -+= 00001 root /sbin/launchd |-+= 00132 oliver /sbin/launchd | |-+= 00252 oliver /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal | | \-+= 05165 root login -pf oliver | | \-+= 05166 oliver -bash | | |--= 06846 oliver sleep 30 | | |--= 06847 oliver sleep 60 | | \-+= 06848 oliver pstree | | \--- 06849 root ps -axwwo user,pid,ppid,pgid,command

sleep

pstree

ps

$ ps -A # show all process info $ ps -fA # show all process info (verbose)

sleep

jobs

$ jobs [1]- Running sleep 60 & [2]+ Running sleep 30 &

sleep

bg

fg

$ sleep 60 # pause job with control-z ^Z [1]+ Stopped sleep 60 $ jobs # see jobs [1]+ Stopped sleep 60 $ bg # set running in background [1]+ sleep 60 & $ jobs # see jobs [1]+ Running sleep 60 & $ fg # bring job from background to foreground sleep 60

$ sleep 60 & [1] 7161 $ jobs [1]+ Running sleep 60 & $ kill %1 $ [1]+ Terminated: 15 sleep 60

%n

grep

$ ps -Af | grep Terminal 501 252 132 0 11:02PM ?? 0:01.66 /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal 501 1653 1532 0 12:09AM ttys000 0:00.00 grep Terminal

grep

grep

awk

$ ps -Af | awk 'NR==1 || $2==252' UID PID PPID C STIME TTY TIME CMD 501 252 132 0 11:02PM ?? 0:01.89 /Applications/Utilities/Terminal.app/Contents/MacOS/Terminal

$ kill 252

$ script.py > out.o 2> out.e &

$ time script.py > out.o 2> out.e & $ nohup script.py > out.o 2> out.e &

$ for i in 1 2 3; do { echo "***"$i; sleep 60 & } done

sleep

$ top

top

htop

top

htop

sudo and the Root User

whoami

$ ls -hl /

$ mkdir /junk mkdir: cannot create directory ‘/junk’: Permission denied

$ sudo mkdir /junk

sudo

$ sudo rmdir /junk

$ sudo -i

$ whoami # check my user name oliver $ sudo -i # simulate login as root Password: $ whoami # check user name now root $ exit # logout of root account logout $ whoami # check user name again oliver

sudo

/usr/bin

sudo

awk

cat file.txt | awk '{ some code }'

$ cat test.txt 1 c 3 c 2 t 1 c

$ cat test.txt | awk '{print}' # print full line 1 c 3 c 2 t 1 c

$ cat test.txt | awk '{print $0}' # print full line 1 c 3 c 2 t 1 c

$ cat test.txt | awk '{print $1}' # print column 1 1 3 2 1

$ cat test.txt | awk '{print $2}' # print column 2 c c t c

$ cat test.txt | awk 'BEGIN{x=""}{x=x"@"$1; print x}' @1 @1@3 @1@3@2 @1@3@2@1

$ cat test.txt | awk 'BEGIN{x=""}{x=x"@"$1}END{print x}' @1@3@2@1

$ cat test.txt | awk '{x+=$1}END{print x}' # x+=$1 is the same as x=x+$1 7

$ cat test.txt | awk '{print $1"\t"$2}' # write tab explicitly 1 c 3 c 2 t 1 c

$ cat test.txt | awk '{OFS="\t"; print $1,$2}' # set output field separator to tab 1 c 3 c 2 t 1 c

$ cat test.txt | awk '{OFS="\t"; print $1,$2}' # print file as is 1 c 3 c 2 t 1 c

$ cat test.txt | awk '{OFS="\t"; print NR,$1,$2}' # print row num 1 1 c 2 3 c 3 2 t 4 1 c

$ cat test.txt | awk '{OFS="\t"; print NR,NF,$1,$2}' # print row & field num 1 2 1 c 2 2 3 c 3 2 2 t 4 2 1 c

$ cat test.txt | awk '{if (NR==3) {print $0}}' # print the 3rd row of your file 2 t

$ cat test.txt | awk '{if (NR==3) {print}}' # same thing, more compact syntax 2 t

$ cat test.txt | awk 'NR==3' # same thing, most compact syntax 2 t

$ cat test.txt | awk '{print NF}' | sort -u 2

$ cat test.txt | awk '{print $NF}' c c t c

cut

$ echo "a b" | awk '{print $1}' a

$ echo "a b" | awk -F"\t" '{print $1}' a b

$ x=hello $ cat test.txt | awk -v var=$x '{ print var"\t"$0 }' hello 1 c hello 3 c hello 2 t hello 1 c

$ cat test.txt | awk '{if ($1==1) {print > "file1.txt"} else {print > "file2.txt"}}'

$ cat file1.txt 1 c 1 c

$ cat file2.txt 3 c 2 t

Question

$ echo -e "a\ta

a\tc

a\tz

a\ta" a a a c a z a a

$ echo -e "a\ta

a\tc

a\tz

a\ta" | awk '$1==$2{print NR}' 1 4

sed

$ cat test_header.txt This is a header 1 asdf 2 asdf 2 asdf

$ cat test_header.txt | sed '1d' # delete the first line 1 asdf 2 asdf 2 asdf

$ cat test_header.txt | sed '1,3d' # delete lines 1-3 2 asdf

/pattern/d

$ cat test_comment.txt 1 asdf # This is a comment 2 asdf # This is a comment 2 asdf

$ cat test_comment.txt | sed '/^#/d' 1 asdf 2 asdf 2 asdf

s/A/B/

s/A/B/g

$ echo "hello kitty. goodbye kitty" | sed 's/kitty/X/' hello X. goodbye kitty

$ echo "hello kitty. goodbye kitty" | sed 's|kitty|X|' hello X. goodbye kitty

$ echo "hello kitty. goodbye kitty" | sed 's/kitty/X/g' hello X. goodbye X

$ echo "hello kitty. goodbye kitty" | sed 's|kitty||' hello . goodbye kitty

$ echo "hello kitty. goodbye kitty" | sed 's|kitty||g' hello . goodbye

$ touch file{1..3}.txt $ for i in file{1..3}.txt; do j=$( echo $i | sed 's|.txt|.html|'); mv $i $j; done $ ls file1.html file2.html file3.html

More awk Examples

$ for i in myfile*; do echo "*** "$i" ***"; cat $i | awk -v x=${i} '{print x"\t"$0}' >> files.concat.txt; done

$ cat test_markup.txt https://www.google.com http://www.nytimes.com http://en.wikipedia.org/wiki/Main_Page

$ cat test_markup.txt | awk '{print "<a href=\""$1"\">"$1"</a><br>"}' <a href="https://www.google.com">https://www.google.com</a><br> <a href="http://www.nytimes.com">http://www.nytimes.com</a><br> <a href="http://www.wikipedia.org">http://www.wikipedia.org</a><br>

$ for i in {1..500}; do a=$( cat input${i}.txt | cut -f1 | sort -u | wc -l ); b=$( cat output${i}.txt | cut -f1 | sort -u | wc -l ); echo -e $i"\t"$a"\t"$b; done | awk '$2!=$3'

$ for i in {1..500}; do a=$( cat input${i}.txt | cut -f1 | sort -u | wc -l ); b=$( cat output${i}.txt | cut -f1 | sort -u | wc -l ); echo -e $i"\t"$a"\t"$b; done | awk '$2!=$3' # print if col2 not equal to col3

$ cat myfasta.fa >GeneA ATGCTGAAAGGTCGTAGGATTCGTAG >GeneB ATGAACGTAA

$ cat myfasta.fa | awk '{if ($1 ~ />/) {printf $1"\t"} else {print length}}' | sed 's|>||' GeneA 26 GeneB 10

$ cat myfasta.fa | awk '{if ($0 ~ />/) {id=$0} else if (length > 20) {print id; print}}' >GeneA ATGCTGAAAGGTCGTAGGATTCGTAG

$ cat myfasta.fa | awk 'BEGIN{f=0}{if ($0~/^>/) {if (f) printf "

"; print $0; f=1} else printf $0}END{printf "

"}'

$ cat test.fa >a ACTGNNNAGGTNNNA >b NNACTGNNNAGGTNN

$ cat test.fa | awk 'BEGIN{flag=0}{if ($0 ~ />/) {if (flag==1) {print nstart"\t"nend;}; print; pos=0; flag=0} else {for (i=1; i<=length; i++) { pos++; c=substr($0,i,1); if (c=="N" && flag==0) {nstart=pos; nend=pos; flag=1;} else if (c=="N") {nend=pos} else if (c!="N" && flag==1) {print nstart"\t"nend; flag=0}}}}END{if (flag==1) {print nstart"\t"nend;}}'

$ cat test.fa | awk ' BEGIN { flag = 0 } { if ( $ 0 ~ />/ ) # ID line { if ( flag == 1 ) { print nstart "\t" nend ;}; # print ID, set position to 0, flag to 0 print ; pos = 0 ; flag = 0 ; } else # sequence line { # loop thro sequence for ( i = 1 ; i <= length ; i ++ ) { # increment position pos ++ ; # grab the i-th letter c = substr ( $ 0 , i , 1 ); # set position if char == N # flag will raise hi if we hit an N char if ( c == "N" && flag == 0 ) { nstart = pos ; nend = pos ; flag = 1 ; } else if ( c == "N" ) { nend = pos } else if ( c != "N" && flag == 1 ) { print nstart "\t" nend ; flag = 0 ; } } # for loop }} END { if ( flag == 1 ) { print nstart "\t" nend ;}} ' >a 5 7 12 14 >b 1 2 7 9 14 15

hello kitty kitty hello kitty kitty

1 2 3 hello kitty kitty hello kitty kitty

$ cat file.txt | awk -F"\t" '{if (NR==1) { for (i=1; i<=NF; i++) {printf i; for (j=1; j<=int(length($i)/8); j++) {printf "\t"}; if (i==NF) {printf "

"} else {printf "\t"}}; print $0 } else {print $0}}'

$ cat file.txt | awk -F"\t" ' { if ( NR == 1 ) # if first line { # loop thro the number of fields for ( i = 1 ; i <= NF ; i ++ ) { # print number with no tab or newline printf i ; # use the fact a tab is 8 characters # print extra tabs for long words for ( j = 1 ; j <= int ( length ( $ i ) / 8 ); j ++ ) { printf "\t" }; # if last field, print newline; otherwise, print tab if ( i == NF ) { printf "

" } else { printf "\t" } }; # print first line print $ 0 } else # if not first line, simply print { print $ 0 }} '

Regular Expressions and Globbing in Bash

tttxc234 wer1 asfwaffffffff2342525

xd2@joe.com carlos_danger@gmail.com hellokitty@yahoo.com

$ touch A1f A2a A3c A4d A5a B5x # create files $ ls # list everything A1f A2a A3c A4d A5a B5x $ ls A{2..5}* # list files beginning with A2 through A5 A2a A3c A4d A5a

Command Line Perl and Regex

cat file.txt | perl -ne '{ some code }'

$ cat test_table.txt x y z 1 2 3 a b c

$ cat test_table.txt | perl -ne 'BEGIN{print "<table border=\"1\">

";}{chomp($_); my @line=split("\t",$_); print "<tr>"; foreach my $elt (@line) { print "<td>$elt</td>"; } print "</tr>

";}END{print "</table>

";}'i

$ cat test_table.txt | perl -ne ' BEGIN { print "<table border=\"1\">

" ;}{ chomp ( $_ ); my @line = split ( "\t" , $_ ); print "<tr>" ; foreach my $elt ( @line ) { print "<td>$elt</td>" ; } print "</tr>

" ; } END { print "</table>

" ;} ' <table border="1"> <tr><td>x</td><td>y</td><td>z</td></tr> <tr><td>1</td><td>2</td><td>3</td></tr> <tr><td>a</td><td>b</td><td>c</td></tr> </table>

Syntax Equivalent Syntax What It Represents \d [0-9] Any digit \D [^0-9] Any character not a digit \w [0-9a-zA-Z_] Any "word character" \W [^0-9a-zA-Z_] Any character not a word character \s [ \t

\r\f] whitespace (space, tab, newline, carriage return,

form feed) \S [^ \t

\r\f] Any non-whitespace character . Any character except newline

Quantifiers, etc. What It Means * Match 0 or more times + Match 1 or more times ? Match 1 or 0 times {n} Match exactly n times {n,} Match at least n times {n,m} Match at least n but not more than m times ^ Match at the beginning of a line $ Match at the end of a line

$ cat test.txt 889 tttxc234 wer1 CAT asfwaffffffff2342525

$ cat test.txt | perl -ne '{chomp($_); if ($_ =~ m/(\d*)/) {print $_,"

";}}' 889 tttxc234 wer1 CAT asfwaffffffff2342525

some value =~ m/regular expression/

$ cat test.txt | perl -ne '{chomp($_); if ($_ =~ m/(\d+)/) {print $_,"

";}}' 889 tttxc234 wer1 asfwaffffffff2342525

$ cat test.txt | perl -ne '{chomp($_); if ($_ =~ m/(\D+)/) {print $_,"

";}}' tttxc234 wer1 CAT asfwaffffffff2342525

$ cat test.txt | perl -ne '{chomp($_); if ($_ =~ m/(\d+)(\D+)/) {print $_,"

";}}' $

$ cat test.txt | perl -ne '{chomp($_); if ($_ =~ m/(\D+)(\d+)/) {print $_,"

";}}' tttxc234 wer1 asfwaffffffff2342525

$ cat test.txt | perl -ne '{chomp($_); if ($_ =~ m/(\D+)(\d+)/) {print $1,"

";}}' tttxc wer asfwaffffffff

$ cat test.txt | perl -ne '{chomp($_); if ($_ =~ m/(\D+)(\d+)/) {print $2,"

";}}' 234 1 2342525

( )

$ cat mail.txt xd2@joe.com malformed.hotmail.com malformed@@hotmail.com carlos_danger@gmail.com hellokitty@yahoo.com

$ cat mail.txt | perl -ne '{chomp($_); if ($_ =~ m/(\w+)@(\w+)/) {print $_,"

";}}' xd2@joe.com carlos_danger@gmail.com hellokitty@yahoo.com

$ cat mail.txt | perl -ne '{chomp($_); if ($_ =~ m/(\w+)\@{1}(\w+)/) {print $_,"

";}}' xd2@joe.com carlos_danger@gmail.com hellokitty@yahoo.com

$ cat mail.txt | perl -ne '{chomp($_); if ($_ =~ m/(\w+)\@{2}(\w+)/) {print $_,"

";}}' malformed@@hotmail.com

Question

$ cat mail.txt | perl -ne '{chomp($_); if ( $_ =~ m/(\w+)(\@+)(\w+)/) {print $2,"

";}}'

Answer

$ cat mail.txt | perl -ne '{chomp($_); if ( $_ =~ m/(\w+)(\@+)(\w+)/) {print $2,"

";}}' @ @@ @ @

Text Editors Revisited: Vim

nano

The most useful software is sometimes rendered useless by poor or altogether missing documentation. Vim refuses to succumb to death by underdocumentation.

i enter insert mode I enter insert mode at the beginning of the line a enter insert mode one character after the cursor A enter insert mode at the end of the line o create new line (below cursor), enter insert mode at the beginning of it O create new line (above cursor), enter insert mode at the beginning of it

Ctrl-c return to normal mode Esc return to normal mode

:

:w save :q quit :wq save and quit :q! quit without saving

gg jump to top of file G jump to bottom of file Cntrl-u move up a chunk Cntrl-d move down a chunk W jump cursor forward on whitespace w jump cursor forward on special characters B jump cursor backward on whitespace b jump cursor backward on special characters 27gg go to line 27 :27 go to line 27

dd delete line (and store in default register) yy copy or yank line (into default register) d0 delete everything from cursor to beginning of line (into default register) d$ delete everything from cursor to end of line (into default register) p paste (from default register) after cursor P paste (from default register) before cursor

u undo Cntrl-r redo

/

n find next instance N find prev instance

nano

Example Problems on the Command Line

Question

Answer

ls

$ ls -t | head -1

Question

$ cat tmp.txt 1 aa 3 1 34 z 1 f 32 2 3r z 2 d cc 3 d 34 x e cc x 1 z

Answer

sort

$ cat tmp.txt | cut -f3 | sort -u | wc -l 5

Question

$ cat tmp.txt aaa bbb ccc ddd eee fff ggg hhh iii jjj

Hint

awk

Answer

$ cat tmp.txt | awk '{print NR"\t"NR%2"\t"$0}' 1 1 aaa 2 0 bbb 3 1 ccc 4 0 ddd 5 1 eee 6 0 fff 7 1 ggg 8 0 hhh 9 1 iii 10 0 jjj

$ cat tmp.txt | awk 'NR%2==1' aaa ccc eee ggg iii

Question

$ cat file1.txt 1 200 324 95 10 a b c

$ cat file2.txt 3 1 200 324 95

Hint

Answer

$ cat file1.txt file2.txt file2.txt | sort | uniq -u 10 a b c

cat

uniq -u

>17 dna:chromosome chromosome:GRCh37:17:1:81195210:1 REF AAGCTTCTCACCCTGTTCCTGCATAGATAATTGCATGACAATTGCCTTGTCCCTGCTGAA TGTGCTCTGGGGTCTCTGGGGTCTCACCCACGACCAACTCCCTGGGCCTGGCACCAGGGA GCTTAACAAACATCTGTCCAGCGAATACCTGCATCCCTAGAAGTGAAGCCACCGCCCAAA GACACGCCCATGTCCAGCTTAACCTGCATCCCTAGAAGTGAAGGCACCGCCCAAAGACAC GCCCATGTCCAGCTTATTCTGCCCAGTTCCTCTCCAGAAAGGCTGCATGGTTGACACACA

Question

Hint

Answer

grep -v ">"

fold

$ cat chr17_300bp.fa | grep -v ">" | fold -w 1 | head A A G C T T C T C A

uniq -c

$ cat chr17_300bp.fa | grep -v ">" | fold -w 1 | sort | uniq -c 73 A 100 C 63 G 64 T

,height,weight,salary,age 1,106,111,111300,62 2,124,91,79740,40 3,127,176,15500,46

Question

1 height 106 2 height 124 3 height 127 1 weight 111 2 weight 91 3 weight 176 1 salary 111300 2 salary 79740 3 salary 15500 1 age 62 2 age 40 3 age 46

Hint

Answer

$ cat example_data.txt | cut -d"," -f2- height,weight,salary,age 106,111,111300,62 124,91,79740,40 127,176,15500,46

$ cat example_data.txt | cut -d"," -f2- | sed 's|,|\t|g' height weight salary age 106 111 111300 62 124 91 79740 40 127 176 15500 46

$ cat example_data.txt | cut -d"," -f2- | sed 's|,|\t|g' | datamash transpose height 106 124 127 weight 111 91 176 salary 111300 79740 15500 age 62 40 46

awk '{for (i = 2; i <= NF; i++){print i-1"\t"$1"\t"$i}}'

$ cat example_data.txt | cut -d"," -f2- | sed 's|,|\t|g' | datamash transpose | awk '{for (i = 2; i <= NF; i++){print i-1"\t"$1"\t"$i}}' 1 height 106 2 height 124 3 height 127 1 weight 111 2 weight 91 3 weight 176 1 salary 111300 2 salary 79740 3 salary 15500 1 age 62 2 age 40 3 age 46

Question

Answer

$ sudo find / | wc -l 98912

$ sudo find / | grep " " | wc -l 304

$ sudo find / | grep " " | head -5 /sys/bus/pnp/drivers/i8042 aux /sys/bus/pnp/drivers/i8042 aux/bind /sys/bus/pnp/drivers/i8042 aux/00:06 /sys/bus/pnp/drivers/i8042 aux/uevent /sys/bus/pnp/drivers/i8042 aux/unbind

$ sudo find / | grep " " | awk -F"/" '$NF ~ / /' | wc -l 70

awk

$ sudo find / -name "* *" | wc -l 70

Example Bash Scripts

$ cat $( which myscript )

PATH

#!/bin/bash # cat a file in your path file = $( which $1 2>/dev/null ) if [[ -f $file ]] ; then cat $file else echo "file \" $1 \" does not exist!" fi

/dev/null

which

#!/bin/bash # vim a file in your path file = $( which $1 2>/dev/null ) if [[ -f $file ]] ; then vim $file else echo "file \" $1 \" does not exist!" fi

[start] Some output ... [end]

#!/bin/bash # 1st arg is logs directory logs = $1 # example: ./checkerror project/logs tot = 0 # number of files scanned # loop thro .o files in logs directory for i in ${ logs } /*.o* ; do # get file name filename = $( basename $i ) ; # check for [start] and [end] tags if ! grep "\[start\]" $i > /dev/null ; then echo -e "

[error] no start tag: " ${ filename } ; elif ! grep "\[end\]" $i > /dev/null ; then echo -e "

[error] no end tag: " ${ filename } ; else # print a dot for each error-free file echo -n "." fi # increment total (( tot++ )) done echo -e "

[summary] $tot files scanned"

/dev/null

grep

basename

PATH

#!/bin/bash echo "[checking dependencies]" # define dependencies dependencies = "bwa blastn bedtools" for i in $dependencies ; do if ! which $i > /dev/null ; then echo "[error] $i not found" exit fi done

PATH

which

PATH

Using the Python Shell to do Math, and More

$ python

$ python # open the Python shell >>> 234+234 468 >>> import math >>> math.log(234,3) 4.96564727304425 >>> math.log(234,2) 7.870364719583405 >>> math.pow(2,8) 256.0 >>> math.cos(3.141) -0.9999998243808664 >>> math.sin(3.141) 0.0005926535550994539

,height,weight,salary,age 1,106,111,111300,62 2,124,91,79740,40 3,127,176,15500,46

1 height 106 2 height 124 3 height 127 1 weight 111 2 weight 91 3 weight 176 1 salary 111300 2 salary 79740 3 salary 15500 1 age 62 2 age 40 3 age 46

>>> with open('example_data.txt', "r") as f: contents = f.read()

>>> contents ',height,weight,salary,age

1,106,111,111300,62

2,124,91,79740,40

3,127,176,15500,46

'

>>> contents.split('

') [',height,weight,salary,age', '1,106,111,111300,62', '2,124,91,79740,40', '3,127,176,15500,46', '']

>>> contents.split('

')[:-1] [',height,weight,salary,age', '1,106,111,111300,62', '2,124,91,79740,40', '3,127,176,15500,46']

>>> [x.split(',') for x in contents.split('

')[:-1]] [['', 'height', 'weight', 'salary', 'age'], ['1', '106', '111', '111300', '62'], ['2', '124', '91', '79740', '40'], ['3', '127', '176', '15500', '46']]

A

zip(*A)

>>> zip(*[x.split(',') for x in contents.split('

')[:-1]]) [('', '1', '2', '3'), ('height', '106', '124', '127'), ('weight', '111', '91', '176'), ('salary', '111300', '79740', '15500'), ('age', '62', '40', '46')]

>>> for j in zip(*[x.split(',') for x in contents.split('

')[:-1]])[1:]: print(j) ('height', '106', '124', '127') ('weight', '111', '91', '176') ('salary', '111300', '79740', '15500') ('age', '62', '40', '46')

>>> k = ('height', '106', '124', '127') >>> [str(y+1) + '\t' + k[0] + '\t' + z for y,z in enumerate(k[1:])] ['1\theight\t106', '2\theight\t124', '3\theight\t127']

>>> for j in zip(*[x.split(',') for x in contents.split('

')[:-1]])[1:]: ... print('

'.join([str(y+1) + '\t' + j[0] + '\t' + z for y,z in enumerate(j[1:])])) 1 height 106 2 height 124 3 height 127 1 weight 111 2 weight 91 3 weight 176 1 salary 111300 2 salary 79740 3 salary 15500 1 age 62 2 age 40 3 age 46

tmux

tmux

tmux

tmux

sessions

windows

panes

tmux

tmux

tmux

tmux

$ tmux

tmux

$ tmux ls 0: 4 windows (created Fri Sep 12 16:52:30 2014) [177x37]

$ tmux attach # attach to the first session $ tmux attach -t 0 # attach to a session by id

$ tmux kill-session -t 0 # kill a session by id

tmux

<Leader> c Create a new window <Leader> x Kill the current window <Leader> n Go to the next window <Leader> p Go to the previous window <Leader> l Go to the last-seen window <Leader> 0-9 Go to a window numbered 0-9 <Leader> w List windows <Leader> s Toggle between sessions <Leader> d Detach the session <Leader> , Set a window's title

tmux

Note

tmux

nohup

Installing Programs on the Command Line

htop

pstree

nginx

tmux

./configure; make; make install

$ brew search gpg2 # Mac $ yum search gpg2 # Fedora $ apt-cache search gpg2 # Ubuntu

$ brew install gpg2 # Mac $ sudo yum install gnupg2.x86_64 # Fedora $ sudo apt-get install gpgv2 # Ubuntu

gpg2

Bash in the Programming Ecosystem (or When Not to Use Bash)

result=$(echo "scale=2; 5 * 7 /3;" | bc)

Concluding Notes

$ man bash $ info coreutils

Acknowledgements and Postscript

Albert Lee

Vladimir Trifonov

Alex North-Keys

Guido Jansen

Postscript, March 2015

I don't like remembering syntax and I forget what I've learned quickly. Thus this article started out as a memory aid—miscellaneous command line-alia for which I wanted a reference done in my own particular way, prejudices included. Its waistline expanded to its present (large) girth and it got a little play on the internet in early 2015. If I hadn't appreciated the importance of editors and fact-checkers, I do now—as the editing of this page was nerve-wrackingly crowded-sourced in real time. I've since corrected many of the errors people pointed out as well as polished and added sections, and the article is better for it.



If you've read this far, you know that this article is not, speaking in rigid literals, an introduction to unix. It's not about the nitty-gritty of how operating system kernels work. And it's not even about UNIX®, which is still a trademark—"the worldwide Single UNIX Specification integrating X/Open Company's XPG4, IEEE's POSIX Standards and ISO C" (say what!?), according to the official website. These days UNIX® refers to a detailed specification rather than software. This article, in contrast, is about command line basics. But it's not about all command lines, which would include rubbish like Windows Powershell. It's about unix-like command lines—i.e., command lines descended from the ancestral unix progenitor:



(Image credit: Wikipedia: Unix-like)



This is what I mean when I use "unix" as a colloquial shorthand. However, to begin the article by embroiling readers in these quagmires of terminology would turn them away in the first paragraph.



There were other comments but the funniest one came from my dad: P.S. I see 2 keys at bottom of my keyboard that say, "command." Is this what all the fuss is about? I'm afraid if I touched them the computer would blow up like a roadside bomb! More comments, corrections, or suggestions? Let me know!