How do you audit your Linux environment? How do you track after changes to your files? What kind of processes are running on your system at any given moment? What uses the most resources? Valid questions, all. Special contributor Dedoimedo gives us the straight scoop on “audit.”. Editor’s note: Call for submissions: are you an OS expert? Can you provide some special insight, some tips and tricks, or just plain illuminate an obscure feature in your OS of choice? We’d like to publish it.

We’ve monitored system activity with sar. We used iostat, vmstat and dstat to collect statistics

on resource usage. We learned about pacct for process accounting. We also worked with top in

batch mode and atop. Finally, we worked with some high-end debugging tools like OProfile and

strace. Links below. But we did not use any utility for file auditing.

Today, we do that. The name of the game: Linux audit.

What Linux audit be?



Linux audit is a built-in mechanism in the kernel, which allows you to intercept changes to

monitored files and write them to a log on the disk. The mechanism consists of three major

components: auditd , the audit daemon (background service),

auditctl , the program used to control the auditing system, and

ausearch , a tool to query audit daemon logs.

To get audit configured, you will need to install the audit (or auditd) package, if it’s not

already installed. Once installed, you will probably want to setup audit to run on startup,

using a system command-line tool like chkconfig, for instance.

You can also manually control the service via /etc/init.d.

Why not use version control?



That’s a good question. The big difference between audit and version control is that the

first is mandatory. Version control works only if people want to cooperate. Let’s see an

example.

Take subversion (svn), for instance. You can have all kinds of files and directories under

version control. Indeed, every time you update your files from the central repository, you

will override any local changes done to your files. While svn blame will tell you about file

edits against the subversion server, it won’t tell you anything about local changes.

Furthermore, this requires another layer of setup.

An alternative is configuration management, like cfengine. This could work, too. You will

have a static baseline to revert to, deleting any unwanted changes to your files. However,

you will not know, in between period runs, who made changes to your files – or why.

Both version control and configuration management are excellent resources, but they rely on

good will. They work extremely well in an environment where you have cooperative users

working toward a common goal. If you merely want to police your files against all and any

desired and undesired changes, then you want auditing.

So auditing it shall be.

Audit in action (auditctl)



Let’s see how this thing works. First, let’s make sure the service is running:

Now, we can start working on collecting audits. You can configure auditing rules on the fly

via auditctl, plus you can set some defaults using the configuration files.

/etc/audit/auditd.conf is used to configure log file location,

size and flushing (rotation), and various actions like mail and disk space events.

/etc/audit/audit.rules lists auditing rules that are loaded on

the service start, restart or reload.

But we will start with using auditctl from the command line. You will need root or sudo to be

able to conduct system-wide auditing. Let’s try an example, which is pretty much what you

will find in the man page. We will configure a watch using auditctl .

Or in text, if you can’t see the image:

auditctl -w /etc/hosts -p war -k hosts-file



What do we have here?

-w flag specifies the file to be watched.

-p sets the permission filter; in other words, what activityyou want to

audit. You have four options, r = read, w = write, x = execute, a = attribute change, which

could be owner, permissions or one of the special file attributes, depending on the

filesystem. -k is a filter, which you can later use to search

through your records in a more meaningful manner. Call it a tag if you will.

Other useful flags

There are many more flags for auditctl. For instance:

-W will remove the watch on a specified file.

-S allows you to trace specific system calls. You can specify

them individually or use all for all system calls. This is somewhat similar to -e flag for

strace, as we’ve seen in the separate tutorial.

Flags -a and -A allow you to add

rules to the end or the beginning or the rules list, respectively. Now, -d will delete a specific rule. -D will

delete all rules.

Adding rules

Adding rules uses a list,action pair, where you first specify

thelist entry, which could be one of the following: task, which is generated when a process

is created, entry, when you enter a system call, exit when you exit a system call, and so

forth. Action can be never and always.

We will see more examples later on. Now, we have a watch in place, let’s edit our /etc/hosts

file and then see what ausearch gives us.

ausearch



ausearch can be run any time you want, while auditd is running. The output produced is

extremely detailed and should not be read at face value. You may probably want to parse the

information, to grab only whatever interests you.

The basic usage is:

ausearch <filename>



But this will produce an ugly output. You can use the -i flag to make it more reasonable to

read; even so, you will have a lot of data to process.

What do we have here?

type tells us what kind of record we have. For file watch

rules, it will normally be PATH. For system calls, it will be SYSCALL.

msg is the record timestamp. name tells us

the filename for files. You also have file type, permissions, owner, group, device major and

minor numbers, as well as the inode number.

For system calls, it’s a little more complicated. syscall tells

us what system call was, uh, well, called. success tells us if

it worked. You also get the process id (pid) and the parent process id (ppid). There’s a long

set of real and effective user and group ids. You also have the console number.

exe specifies the command that was run. And so forth. Detailed and only

useful for experienced administrators knowing what they need.

This is very similar to going under /proc and burrowing into data for specific processes, but

the thing is, it’s done automatically and faster than a human could possibly log information.

Other useful flags

-k will search by specific keys (tags) only. Otherwise, why did

we use them in the first place? -m will filter only specific

types of records. -sc will match only specific system calls.

-p will only match specific processes.

You can also redirect your output into text editors or files.

More examples



There are quite a few examples in the man page. Let me be a monkey and repeat some of them

here, for the sake of education.

To see files opened by a specific user:

auditctl -a exit,always -S open -F auid=510



We will always monitor the exit from the open system call for the audit uid (auid) 510, which

might correspond to some user, like roger. auid is the original ID the user logged in, so if

he/she changes his effective UID while working, it won’t be masked in the log. We want to

monitor the exit from the system call, because if it fails, we will know the exact error.

Another example is:

auditctl -a exit,never -S mount



It says, don’t log any mount system calls.

Here, rather than collecting as much as we can, we suppress output. This could be useful if

you know you might have too many logs or the information is just not interesting. For

example, in the NFS environment, you may not want to do any auditing on the client side and

will instead focus on doing it on the server side. Besides, in this case, there are other

tools you may want to use. For example, pacct, access control via access.conf, traffic

logging, export options for your NFS areas, and more.

One more for ausearch:

ausearch -ts today -k hosts-file -x vi



-ts tells us the timestamp we want to search through our logs; today, only. -k narrows down

our search to the hosts-file key only. -x specifies the exact name of the executable run

against the watched file. In this regard, our example looks for any attempts to open or

modify the file with vi text editor today.

That’s enough, I think.

More stuff



There are more helper tools that can help you work with auditing. One such tool is

aureport , which you can use to generate summary reports of

collected logs. This is somewhat similar to -c flag in strace.

More reading



A handful of my system administration oriented articles.

10 super-cool Linux hacks

you did not know about

Linux cool hacks –

Part II

Linux super-duper admin tools:

lsof

Linux super-duper admin tools:

screen

Linux super-duper admin tools:

OProfile

Linux super-duper admin tools:

Strace

Some external links:

Linux

audit files to see who made changes to a file

Finally, man pages:

auditd

auditctl

ausearch

And that would be all folks!

Conclusion



You have just gained another powerful tool that you can use to make your Linux environment

smarter and better controlled. Alongside an ever-growing repertoire of powerful utilities,

audit will help you build your Linux street credit and make your work easier. audit is very

simple to use, although processing the collected logs can take time and effort. You need to

master all the little flags and understand how the system works before you can delve deeper.

If you have any useful tools you considered using but didn’t dare try, feel free to ping me

and I will testdrive them for you, trying to provide you with the best and most modular

advice while striving for strategic imperatives rather than tactical keyboard hammering.

Cheers.