To successfully work with the Linux sed editor and the awk command in your shell scripts, you have to understand regular expressions or in short regex. Since there are many engines for regex, we will use the shell regex and see the bash power in working with regex.

First, we need to understand what regex is; then we will see how to use it.

What is regex

For some people, when they see the regular expressions for the first time, they said what are these ASCII pukes !!

Well, A regular expression or regex, in general, is a pattern of text you define that a Linux program like sed or awk uses it to filter text.

We saw some of those patterns when introducing basic Linux commands and saw how the ls command uses wildcard characters to filter output.

Types of regex

Many different applications use different types of regex in Linux, like the regex included in programming languages (Java, Perl, Python,) and Linux programs like (sed, awk, grep,) and many other applications.

A regex pattern uses a regular expression engine that translates those patterns.

Linux has two regular expression engines:

The Basic Regular Expression (BRE) engine.

engine. The Extended Regular Expression (ERE) engine.

Most Linux programs work well with BRE engine specifications, but some tools like sed understand some of the BRE engine rules.

The POSIX ERE engine comes with some programming languages. It provides more patterns, like matching digits and words. The awk command uses the ERE engine to process its regular expression patterns.

Since there are many regex implementations, it’s difficult to write patterns that work on all engines. Hence, we will focus on the most commonly found regex and demonstrate how to use it in the sed and awk.

Define BRE Patterns

You can define a pattern to match text like this:

$ echo "Testing regex using sed" | sed -n '/regex/p'

$ echo "Testing regex using awk" | awk '/regex/{print $0}'

You may notice that the regex doesn’t care where the pattern occurs or how many times in the data stream.

The first rule to know is that regular expression patterns are case sensitive.

$ echo "Welcome to LikeGeeks" | awk '/Geeks/{print $0}'

$ echo "Welcome to Likegeeks" | awk '/Geeks/{print $0}'

The first regex succeeds because the word “Geeks” exists in the upper case, while the second line fails because it uses small letters.

You can use spaces or numbers in your pattern like this:

$ echo "Testing regex 2 again" | awk '/regex 2/{print $0}'

Special characters

regex patterns use some special characters. And you can’t include them in your patterns, and if you do so, you won’t get the expected result.

These special characters are recognized by regex:

.*[]^${}\+?|()

You need to escape these special characters using the backslash character (\).

For example, if you want to match a dollar sign ($), escape it with a backslash character like this:

$ cat myfile There is 10$ on my pocket

$ awk '/\$/{print $0}' myfile

If you need to match the backslash (\) itself, you need to escape it like this:

$ echo "\ is a special character" | awk '/\\/{print $0}'

Although the forward-slash isn’t a special character, you still get an error if you use it directly.

$ echo "3 / 2" | awk '///{print $0}'

So you need to escape it like this:

$ echo "3 / 2" | awk '/\//{print $0}'

Anchor characters

To locate the beginning of a line in a text, use the caret character (^).

You can use it like this:

$ echo "welcome to likegeeks website" | awk '/^likegeeks/{print $0}'

$ echo "likegeeks website" | awk '/^likegeeks/{print $0}'

The caret character (^) matches the start of the text:

$ awk '/^this/{print $0}' myfile

What if you use it in the middle of the text?

$ echo "This ^ caret is printed as it is" | sed -n '/s ^/p'

It’s printed as it is like a normal character.

When using awk, you have to escape it like this:

$ echo "This ^ is a test" | awk '/s \^/{print $0}'

This is about looking at the beginning of the text, what about looking at the end?

The dollar sign ($) checks for the end a line:

$ echo "Testing regex again" | awk '/again$/{print $0}'

You can use both the caret and dollar sign on the same line like this:

$ cat myfile this is a test This is another test And this is one more

$ awk '/^this is a test$/{print $0}' myfile

As you can see, it prints only the line that has the matching pattern only.

You can filter blank lines with the following pattern:

$ awk '!/^$/{print $0}' myfile

Here we introduce the negation which you can do it by the exclamation mark !

The pattern searches for empty lines where nothing between the beginning and the end of the line and negates that to print only the lines have text.

The dot character

We use the dot character to match any character except the newline (

).

Look at the following example to get the idea:

$ cat myfile this is a test This is another test And this is one more start with this

$ awk '/.st/{print $0}' myfile

You can see from the result that it prints only the first two lines because they contain the st pattern while the third line does not have that pattern, and the fourth line starts with st, so that also doesn’t match our pattern.

Character classes

You can match any character with the dot special character, but what if you match a set of characters only, you can use a character class.

The character class matches a set of characters if any of them found, the pattern matches.

We can define the character classes using square brackets [] like this:

$ awk '/[oi]th/{print $0}' myfile

Here we search for any th characters that have o character or i before it.

This comes handy when you are searching for words that may contain upper or lower case, and you are not sure about that.

$ echo "testing regex" | awk '/[Tt]esting regex/{print $0}'

$ echo "Testing regex" | awk '/[Tt]esting regex/{print $0}'

Of course, it is not limited to characters; you can use numbers or whatever you want. You can employ it as you want as long as you got the idea.

Negating character classes

What about searching for a character that is not in the character class?

To achieve that, precede the character class range with a caret like this:

$ awk '/[^oi]th/{print $0}' myfile

So anything is acceptable except o and i.

Using ranges

To specify a range of characters, you can use the (-) symbol like this:

$ awk '/[e-p]st/{print $0}' myfile

This matches all characters between e and p then followed by st as shown.

You can also use ranges for numbers:

$ echo "123" | awk '/[0-9][0-9][0-9]/'

$ echo "12a" | awk '/[0-9][0-9][0-9]/'

You can use multiple and separated ranges like this:

$ awk '/[a-fm-z]st/{print $0}' myfile

The pattern here means from a to f, and m to z must appear before the st text.

Special character classes

The following list includes the special character classes which you can use them:

[[:alpha:]] Pattern for any alphabetical character, either upper or lower case. [[:alnum:]] Pattern for 0–9, A–Z, or a–z. [[:blank:]] Pattern for space or Tab only. [[:digit:]] Pattern for 0 to 9. [[:lower:]] Pattern for a–z lower case only. [[:print:]] Pattern for any printable character. [[:punct:]] Pattern for any punctuation character. [[:space:]] Pattern for any whitespace character: space, Tab, NL, FF, VT, CR. [[:upper:]] Pattern for A–Z upper case only.

You can use them like this:

$ echo "abc" | awk '/[[:alpha:]]/{print $0}'

$ echo "abc" | awk '/[[:digit:]]/{print $0}'

$ echo "abc123" | awk '/[[:digit:]]/{print $0}'

The asterisk

The asterisk means that the character must exist zero or more times.

$ echo "test" | awk '/tes*t/{print $0}'

$ echo "tessst" | awk '/tes*t/{print $0}'

This pattern symbol is useful for checking misspelling or language variations.

$ echo "I like green color" | awk '/colou*r/{print $0}'

$ echo "I like green colour " | awk '/colou*r/{print $0}'

Here in these examples, whether you type it color or colour it will match, because the asterisk means if the “u” character existed many times or zero time that will match.

To match any number of any character, you can use the dot with the asterisk like this:

$ awk '/this.*test/{print $0}' myfile

It doesn’t matter how many words between the words “this” and “test”, any line matches, will be printed.

You can use the asterisk character with the character class.

$ echo "st" | awk '/s[ae]*t/{print $0}'

$ echo "sat" | awk '/s[ae]*t/{print $0}'

$ echo "set" | awk '/s[ae]*t/{print $0}'

All three examples match because the asterisk means if you find zero times or more any “a” character or “e” print it.

Extended regular expressions

The following are some of the patterns that belong to Posix ERE:

The question mark

The question mark means the previous character can exist once or none.

$ echo "tet" | awk '/tes?t/{print $0}'

$ echo "test" | awk '/tes?t/{print $0}'

$ echo "tesst" | awk '/tes?t/{print $0}'

We can use the question mark in combination with a character class:

$ echo "tst" | awk '/t[ae]?st/{print $0}'

$ echo "test" | awk '/t[ae]?st/{print $0}'

$ echo "tast" | awk '/t[ae]?st/{print $0}'

$ echo "taest" | awk '/t[ae]?st/{print $0}'

$ echo "teest" | awk '/t[ae]?st/{print $0}'

If any of the character class items exists, the pattern matching passes. Otherwise, the pattern will fail.

The plus sign

The plus sign means that the character before the plus sign should exist one or more times, but must exist once at least.

$ echo "test" | awk '/te+st/{print $0}'

$ echo "teest" | awk '/te+st/{print $0}'

$ echo "tst" | awk '/te+st/{print $0}'

If the “e” character not found, it fails.

You can use it with character classes like this:

$ echo "tst" | awk '/t[ae]+st/{print $0}'

$ echo "test" | awk '/t[ae]+st/{print $0}'

$ echo "teast" | awk '/t[ae]+st/{print $0}'

$ echo "teeast" | awk '/t[ae]+st/{print $0}'

if any character from the character class exists, it succeeds.

Curly braces

Curly braces enable you to specify the number of existence for a pattern, it has two formats:

n: The regex appears exactly n times.

n,m: The regex appears at least n times, but no more than m times.

$ echo "tst" | awk '/te{1}st/{print $0}'

$ echo "test" | awk '/te{1}st/{print $0}'

In old versions of awk, you should use –re-interval option for the awk command to make it read curly braces, but in newer versions, you don’t need it.

$ echo "tst" | awk '/te{1,2}st/{print $0}'

$ echo "test" | awk '/te{1,2}st/{print $0}'

$ echo "teest" | awk '/te{1,2}st/{print $0}'

$ echo "teeest" | awk '/te{1,2}st/{print $0}'

In this example, if the “e” character exists one or two times, it succeeds; otherwise, it fails.

You can use it with character classes like this:

$ echo "tst" | awk '/t[ae]{1,2}st/{print $0}'

$ echo "test" | awk '/t[ae]{1,2}st/{print $0}'

$ echo "teest" | awk '/t[ae]{1,2}st/{print $0}'

$ echo "teeast" | awk '/t[ae]{1,2}st/{print $0}'

If there are one or two instances of the letter “a” or “e”, the pattern passes; otherwise, it fails.

Pipe symbol

The pipe symbol makes a logical OR between 2 patterns. If one of the patterns exists, it succeeds; otherwise, it fails, here is an example:

$ echo "Testing regex" | awk '/regex|regular expressions/{print $0}'

$ echo "Testing regular expressions" | awk '/regex|regular expressions/{print $0}'

$ echo "This is something else" | awk '/regex|regular expressions/{print $0}'

Don’t type any spaces between the pattern and the pipe symbol.

Grouping expressions

You can group expressions so the regex engines will consider them one piece.

$ echo "Like" | awk '/Like(Geeks)?/{print $0}'

$ echo "LikeGeeks" | awk '/Like(Geeks)?/{print $0}'

The grouping of the “Geeks” makes the regex engine treats it as one piece, so if “LikeGeeks” or the word “Like” exist, it succeeds.

Practical examples

We saw some simple demonstrations of using regular expression patterns. It’s time to put that in action, just for practicing.

Counting directory files

Let’s look at a bash script that counts the executable files in a folder from the PATH environment variable.

$ echo $PATH

To get a directory listing, you must replace each colon with space.

$ echo $PATH | sed 's/:/ /g'

Now let’s iterate through each directory using the for loop like this:

mypath=$(echo $PATH | sed 's/:/ /g') for directory in $mypath; do done

Great!!

You can get the files on each directory using the ls command and save it in a variable.

#!/bin/bash path_dir=$(echo $PATH | sed 's/:/ /g') total=0 for folder in $path_dir; do files=$(ls $folder) for file in $files; do total=$(($total + 1)) done echo "$folder - $total" total=0 done

You may notice some directories don’t exist, no problem with this, it’s OKay.

Cool!! This is the power of regex—these few lines of code count all files in all directories. Of course, there is a Linux command to do that very easily, but here we discuss how to employ regex on something you can use. You can come up with some more useful ideas.

Validating E-mail address

There are a ton of websites that offer ready to use regex patterns for everything, including e-mail, phone number, and much more, this is handy, but we want to understand how it works.

[email protected]

The username can use any alphanumeric characters combined with dot, dash, plus sign, underscore.

The hostname can use any alphanumeric characters combined with a dot and underscore.

For the username, the following pattern fits all usernames:

^([a-zA-Z0-9_\-\.\+]+)@

The plus sign means one character or more must exist followed by the @ sign.

Then the hostname pattern should be like this:

([a-zA-Z0-9_\-\.]+)

There are special rules for the TLDs or Top-level domains, and they must be not less than 2 and five characters maximum. The following is the regex pattern for the top-level domain.

\.([a-zA-Z]{2,5})$

Now we put them all together:

^([a-zA-Z0-9_\-\.\+]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$

Let’s test that regex against an email:

Awesome!!

This was just the beginning of the regex world that never ends. I hope you understand these ASCII pukes 🙂 and use it more professionally.

I hope you like the post.

Thank you.