First of all, you can read this article in russian here.

1. Split string with variable expansions

*:

$ STRING="username:homedir:shell" $ echo "$STRING"|cut -d ":" -f 3 shell $ echo "${STRING##*:}" shell

significant

$ cat test.sh #!/usr/bin/env bash STRING="Name:Date:Shell" echo "using cut" time for A in {1..1000} do cut -d ":" -f 3 > /dev/null <<<"$STRING" done echo "using ##" time for A in {1..1000} do echo "${STRING##*:}" > /dev/null done

Results $ ./test.sh using cut real 0m0.950s user 0m0.012s sys 0m0.232s using ## real 0m0.011s user 0m0.008s sys 0m0.004s

$ cat test.sh #!/usr/bin/env bash echo "using cut" time for count in {1..1000} do cut -d ":" -f 7 </etc/passwd > /dev/null done echo "using ##" time for count in {1..1000} do while read do echo "${REPLY##*:}" > /dev/null done </etc/passwd done

Result $ ./test.sh $ ./test.sh using cut real 0m0.827s user 0m0.004s sys 0m0.208s using ## real 0m0.613s user 0m0.436s sys 0m0.172s No comments =) No comments =)

$ VAR="myClassName = helloClass" $ echo ${VAR##*= } helloClass

$ VAR="Hello my friend (enemy)" $ TEMP="${VAR##*\(}" $ echo "${TEMP%\)}" enemy

2. Bash autocompletion with tab

guess

$ alias TAsteriskLog="tail -f /var/log/asteriks.log" $ alias TMailLog="tail -f /var/log/mail.log" $ TA[tab]steriksLog $ TM[tab]ailLog

3. Bash autocompletion with tab — part 2

function LastLogin { STRING=$(last | head -n 1 | tr -s " " " ") USER=$(echo "$STRING"|cut -d " " -f 1) IP=$(echo "$STRING"|cut -d " " -f 3) SHELL=$( grep "$USER" /etc/passwd | cut -d ":" -f 7) echo "User: $USER, IP: $IP, SHELL=$SHELL" }

$ L[tab]astLogin User: saboteur, IP: 10.0.2.2, SHELL=/bin/bash

4.1. Sensitive data

$ echo "hello" hello $ history 2 2011 echo "hello" 2012 history 2 $ echo "my password secretmegakey" # there are two spaces before 'echo' my password secretmegakey $ history 2 2011 echo "hello" 2012 history 2

It is optional It is usually enabled by default, but you can configure this behavior in the following variable:



export HISTCONTROL=ignoreboth

4.2. Sensitive data in command line arguments

secret.sh PASSWORD=LOVESEXGOD

myapp.sh source ~/secret.sh sqlplus -l user/"$PASSWORD"@database:port/sid @mysqfile.sql

$ echo "secretpassword" > secret.key; chmod 600 secret.key

$ echo "string_to_encrypt" | openssl enc -pass file:secret.key -e -aes-256-cbc -a U2FsdGVkX194R0GmFKCL/krYCugS655yLhf8aQyKNcUnBs30AE5lHN5MXPjjSFML

$ echo 'U2FsdGVkX194R0GmFKCL/krYCugS655yLhf8aQyKNcUnBs30AE5lHN5MXPjjSFML' | openssl enc -pass file:secret.key -d -aes-256-cbc -a string_to_encrypt

5. The grep command

tail -f application.log | grep -i error

tail -f application.log | grep -i -P "(error|warning|failure)"

tail -f application.log | grep -v -i "info"

$ grep myuser /etc/passwd| cut -d ":" -f 7 $ grep -Po "^myuser(:.*){5}:\K.*" /etc/passwd

6. How to reduce log file size

echo "" > application.log

truncate --size=1M application.log

echo "$(tail -n 1000 application.log)" > application.log

7. Watch is watching for you!

watch <command>

8. Bash sequence

for srv in 1 2 3 4 5; do echo "server${srv}";done server1 server2 server3 server4 server5

for srv in server{1..5}; do echo "$srv";done server1 server2 server3 server4 server5

for srv in $(seq -w 5 10); do echo "server${srv}";done server05 server06 server07 server08 server09 server10

for file in *.txt; do name=$(basename "$file" .txt);mv $name{.txt,.lst}; done

for file in *.txt; do mv ${file%.txt}{.txt,.lst}; done

mkdir -p project/src/{main,test}/{java,resources}

Result project/ !--- src/ |--- main/ | |-- java/ | !-- resources/ !--- test/ |-- java/ !-- resources/

9. tail, multiple files, multiple users...

tail -f /var/logs/*.log

alias TFapplog='tail -f --pid=$(cat /opt/app/tmp/app.pid) /opt/app/logs/app.log'

10. Create file with specified size

dd if=/dev/zero of=out.txt bs=1M count=10

fallocate -l 10M file.txt

11. xargs

$ # lets print 5 arguments and send them to echo with xargs: $ echo 1 2 3 4 5 | xargs echo 1 2 3 4 5 $ # now let’s repeat, but limit argument processing by 3 per execution $ echo 1 2 3 4 5 | xargs -n 3 echo 1 2 3 4 5

echo 1 2 3 4 5 6 7 8 9 10| xargs -n 2 -P 3 echo

echo 1 2 3 4 5 6 7 8 9 10 | xargs -n 2 -P $(nproc) echo

12. sleep? while? read!

read -p "Press any key to continue " -n 1

read -p "Press any key to continue (auto continue in 30 seconds) " -t 30 -n 1

One evening, I was reading Mastering regular expressions by Jeffrey Friedl , I realized that even if you have all the documentation and a lot of experience, there could be a lot of tricks developed by different people and imprisoned for themselves. All people are different. And techniques that are obvious for certain people may not be obvious to others and look like some kind of weird magic to third person. By the way, I already described several such moments here (in russian) For the administrator or the user the command line is not only a tool that can do everything, but also a highly customized tool that could be develops forever. Recently there was a translated article about some useful tricks in CLI. But I feel that the translator do not have enough experience with CLI and didn't follow the tricks described, so many important things could be missed or misunderstood.Under the cut — a dozen tricks in Linux shell from my personal experience.Note: All scripts and examples in the article was specially simplified as much as possible — so maybe you can find several of tricks looks completely useless — perhaps this is the reason. But in any case, share your minds in the comments!People often useor evenjust to substract a part of string by pattern or with separators.Also, many people uses substring bash operation using ${VARIABLE:start_position:length}, that works very fast.But bash provides a powerful way to manipulate with text strings using #, ##,% and %% — it calledUsing this syntax, you can cut the needful by the pattern without executing external commands, so it will work really fast.The example below shows how get the third column (shell) from the string where values separated by colon «username:homedir:shell» usingor using variable expansions (we use themask and the ## command, which means: cut all characters to the left until the last colon found):The second option does not start child process (), and does not use pipes at all, which should work much faster. And if you are using bash subsystem on windows, where the pipes barely move, the speed difference will beLet's see an example on Ubuntu — execute our command in a loop for 1000 timesThe difference is several dozen times!Of course, the example above is too artificial. In real example we will not work with a static string, we want to read a real file. And for '' command, we just redirect /etc/passwd to it. In the case of ##, we have to create a loop and read file using internal '' command. So who will win this case?A couple more examples:Extract the value after equal character:Extract text in round brackets:bash-completion package is a part of almost every Linux distributive. You can enable it in /etc/bash.bashrc or /etc/profile.d/bash_completion.sh, but usually it is already enabled by default. In general, autocomplete is one of the first convenient moments on Linux shell that a newcomer first of all meets.But the fact that not everyone uses all of the bash-completion features, and in my opinion is completely in vain. For example not everybody knows, that autocomplete works not only with file names, but also with aliases, variable names, function names and for some commands even with arguments. If you dig into autocomplete scripts, which are actually shell scripts, you can even add autocomplete for your own application or script.But let’s come back to the aliases.You don't need to edit PATH variable or create files in specified directory to run alias. You just need to add them to profile or startup script and execute them from any place.Usually we are using lowercase letters for files and directories in *nix, so it could be very comfortable to create uppercase aliases — in that case bash-completion willyour command almost with a single letter:For more complicated cases, probably you would like to put your personal scripts to $HOME/bin.But we have functions in bash.Functions don't require path or separate files. And (attention) bash-completion works with functions too.Let’s create function LastLogin in(don't forget to reload .profile):In console (please note that function name have an uppercase first letter to speedup bash-completion):If you put space before any command in console, it will not appears in the command history, so if you need to put plain text password in the command, it is a good way to use this feature — look into example below,will not appears in history:You want to store some shell scripts in git to share them across servers, or may be it is a part of application startup script. And you want this script will connect to database or do anything else which requires credentials.Of course it is bad idea to store credentials in the script itself, because git is not secure.Usually you can use variables, which was already defined on the target environments, and your script will not contain the passwords itself.For example, you can create small script on each environments with 700 permissions and call it usingcommand from the main script:But it is not secure.If somebody else can login to your host, he can just executecommand and see your sqlplus process with the whole command line arguments including passwords. So, secure tools usually should be able to read passwords/keys/sensitive data directly from files.For example — securejust even have no options to provide password in command line. But he can read ssh key from the file (and you can set secure permissions on ssh key file).And non-secure wget have an option "--password" which allows you to provide password in command line. And all the time wget will be running, everybody can execute ps command and see password you have provided.In additional, if you have a lot of sensitive data, and you want to control it from git, the only way is encryption. So you put to every target environment only master password, and all other data you can encrypt and put to git. And you can work with encrypted data from command line, using openssl CLI interface. Here is an example to encrypt and decrypt from command line:File secret.key contains master key — a single line:Lets use aes-256-cbc to encrypt a string:You can put this encrypted string to any configuration file stored in git, or any other place — without secret.key it is almost impossible to decrypt it.To decrypt execute the same command just replace -e with -d:All should knows grep command. And be friendly with regular expressions. And often you can write something like:Or even like this:But don't forget that grep have a lot of wonderful options. For example -v, that reverts your search and shows all except «info» messages:Additional stuff:Option -P is very useful, because by default grep uses pretty outdated «basic regular expression:», and -P enables PCRE which even doesn't know about grouping.-i ignores case.--line-buffered parses line immediately instead of waiting to reach standard 4k buffer (useful for tail -f | grep).If you know regular expression well, with --only-matching/-o you can really do a great things with cutting text. Just compare next two commands to extract myuser's shell:The second command looks more compilcated, but it runs onlyinstead ofand, so it will take less time for execution.In *nix, if you delete the log file, that is currently used by an application, you can not just remove all logs, you can prevent application to write new logs until restart.Because file descriptor opens not the file name, but iNode structure, and the application will continue to write to file descriptor to the file, which have no directory entry, and such file will be deleted automatically after application stop by file system().So, how to clear log file without deleting it:Or we can use truncate command:Mention, thatcommand will delete the rest of the file, so you will lost the latest log events. Check another example how to store last 1000 lines:There is a situation when you are waiting for some event being finished. For example, while another user logins to shell (you continuously executecommand), or someone should copy file to your machine using scp or ftp and you are waiting for the completion (repeating ls dozens of times).In such cases, you can useBy default, will be executed every 2 seconds with pre-clear of screen until Ctrl+C pressed. You can configure how often should be executed.It is very useful when you want to watch live logs.There is very useful construct to create ranges. For example instead of something like this:You can write the following:Also you can usecommand to generate formatted ranges. For example we can useto create values whitch will be automatically adjusted by width (00, 01 instead of 0, 1):Another example with command substitution — rename files. To get filename without extension we are using '' command:Also even more short with '%':P.S. Actually for renaming the files you can try '' tool which has a lot of options.Another example — lets create structure for new java project:I have mentionedto read files and watching multiple live logs. But it doesn't provided by default, and permissions to install something not always available.But standard tail can do it too:Also lets remember about users, which are using 'tail -f' aliases to watch application logs.Several users can watch log files simultaneously using 'tail -f'. Some of them are not very accurate with their sessions. They could leave 'tail -f' in background for some reason and forget about it.If the application was restarted, there are these running 'tail -f' processes which are watching inexistent log file can hang for several days or even months.Usually it is not a big problem, but not neatly.In case, if you are using alias to watch the log, you can modify this alias with --pid option:In that case, allwill be automatically terminated when the target application will be restarted.was one of the most popular tool to work with block and binary data. For example create 1 MB file filled with zero will be:But I recommend useOn file systems, which support allocate function (xfs, ext4, Btrfs...),will be executed instantly, unlike the dd tool. In additional, allocate means real allocation of blocks, not creation a spare file.Many people know popularcommand. But not all of them use two following options, which could greatly, improve your script.First — you can get very long list of arguments to process, and it could exceed command line length (by default ~4 kb).But you can limit execution using -n option, sowill run command several times, sending specified number of arguments at a time:Going ahead. Processing long list could take a lot of time, because it runs in a single thread. But if we have several cores, we can tellto run in parallel:In the example above, we tellto process list in 3 threads; each thread will take and process 2 arguments per execution. If you don't know how many cores you have, lets optimize this using "":Some time you need to wait for several seconds. Or wait for user input with read:But you can just add timeout option tocommand, and your script will be paused for specified amount of seconds, but in case of interactive execution, user can easily skip waiting.So you can just forget about sleep command.I suspect that not all my tricks looks interesting, but it seemed to me that a dozen are a good number to fill out.At this I say goodbye, and I will be grateful for participating in the survey.Of course feel free to discuss the above and share your cool tricks in the comments!