From Fail2ban

Introduction

The problem

Brute-force break-in attempts are quite frequent against an SSH server and other password protected internet-services (such as ftp,pop,...). Automated scripts try multiple combinations of username/password (brute-force, dictionary attack) and sometimes changing the port to something other than the default can't be done. Furthermore, scouring your log files yourself is not only time consuming, but can be difficult too.

Fail2ban attempts to alleviate these issues by providing an automated way of not only identifying possible break-in attempts, but acting upon them quickly and easily in a user-definable manner.

The solution

Log files contain interesting information, especially about failed logins. This information can be used to ban an offensive host. This is exactly what Fail2ban does. It scans log files and detects patterns which correspond to possible breakin attempts and then performs actions. Most of the time, it consists of adding a new rule in a firewall chain and sending an e-mail notification to the system administrator.

Here is a list of the most important features available in Fail2ban:

client/server

multithreaded

Gamin support

autodetection of the date/time format

wildcard support in logpath option

option support for a lot of services (sshd, apache, qmail, proftpd, sasl, asterisk, etc)

support for several actions (iptables, tcp-wrapper, shorewall, mail notifications, etc)

The code has been completely rewritten since 0.6.x. Fail2ban is entirely written in Python and thus should work on most of the *nix systems.

Installation

Requirements

In order to use Fail2ban, the following software are required:

Python >= 2.4

Warning: Python 2.4 does not re-open the syslog socket when it encounters a failure. This is a Python bug. If you want to use "logtarget = SYSLOG", please use a newer version of Python like 2.5.

The following software are optional but recommended:

iptables

shorewall

tcp-wrappers

a working mail command

command Gamin the File Alteration Monitor



You will probably need at least one firewall software like iptables or shorewall. If you want a software which is not supported, please feel free to contact the author.

Gamin is a file alteration monitor. Gamin greatly benefits from a "inotify"-enabled kernel. Thus, active polling is no longer required to get the file modifications.

Installing from sources on a GNU/Linux system

You will need to obtain the latest version of the source code in order to compile Fail2ban yourself. Once you have done this, change to the directory where you downloaded the source code and execute the following:

tar xvjf fail2ban-x.x.x.tar.bz2

You will have the Fail2ban source code extracted to a directory under the current working directory. You must now move to the new directory.

Now run the installation script as root:

./setup.py install

Fail2ban should now be installed into /usr/share/fail2ban/ and /usr/bin/. You will also have to copy the content of config/ into /etc/fail2ban/ (not so in version 0.8.1). You are now ready to use the application.

Integrate fail2ban into your INIT-Process:

Go into the files-folder where you extracted the sources:

# cd /usr/local/src/fail2ban-0.8.1/files

and copy the init-script fitting your distribution to /etc/init.d.

# cp suse-initd /etc/init.d/fail2ban # chmod 755 /etc/init.d/fail2ban

the default-debian-script is missing in the master branch of source-files currently; but could be fetched from the debian branch: https://github.com/fail2ban/fail2ban/blob/debian/debian/fail2ban.init

Finally after copying/editing the init file, and adjusting the paths to fail2ban-client and fail2ban-server run chkconfig or update-rc.d or create the symbolic links manually...

chkconfig -a fail2ban

or

update-rc.d fail2ban defaults

or

ln -s /etc/init.d/fail2ban /etc/rc2.d/S20fail2ban ...

Integrate fail2ban into logrotate:

create file "/etc/logrotate.d/fail2ban":

/var/log/fail2ban.log { weekly rotate 7 missingok compress postrotate /usr/local/bin/fail2ban-client set logtarget /var/log/fail2ban.log >/dev/null endscript }

The path to your fail2ban-client might need to be adjusted (# whereis fail2ban-client)

Debian

Installing Fail2ban on a Debian based system is very straightforward, just install the package through Synaptic or

apt-get install fail2ban

Gentoo

Fail2ban is available in Portage. To install it, just run:

emerge fail2ban

The FAQ has a more detailed explanation of installing using distributions such as Debian, Red Hat and Gentoo

Fedora

Installing Fail2ban on a Fedora 9+ based system is usually straightforward. Execute the following code as the root user:

yum install fail2ban

There is a bug with the current (0.8.4-27) Fedora and RedHat distros [1] which will prevent jails from starting unless SELinux is disabled. Symptoms are that a server with no configuration changes will fail to start and the fail2ban log file (search for 'logtarget' in fail2ban.conf) will have an error message like:

fail2ban.jail : INFO Creating new jail 'ssh-iptables' fail2ban.comm : WARNING Invalid command: ['add', 'ssh-iptables', 'auto']

when starting the SSH jail. If this happens to you, edit jail.conf and change the line that says

backend = auto

to

backend = gamin

This requires gamin to be installed. If it isn't, install it via:

yum install gamin

Alternatively you can disable SELinux, but that is not recommended.

Mac OS X

See HOWTO Mac OS X Server (10.4)

or HOWTO Mac OS X Server (10.5)

Usage

Definitions

Before we begin, it is important to clarify some terms used in the following sections.

filter : a filter defines a regular expression which must match a pattern corresponding to a log-in failure or any other expression action : an action defines several commands which are executed at different moments jail : a jail is a combination of one filter and one or several actions. Fail2ban can handle several jails at the same time client : refers to the script fail2ban-client server : refers to the script fail2ban-server

Server

Fail2ban is composed of 2 parts: a client and a server. The server is multi-threaded and listens on a Unix socket for commands. The server itself knows nothing about the configuration files. Thus, at start-up, the server is in a "default" state in which no jails are defined. The following options are available for fail2ban-server:

-b start in background -f start in foreground -s <FILE> socket path -x force execution of the server -h, --help display this help message -V, --version print the version

fail2ban-server should not be used directly except in case of debugging. The option -s <FILE> is probably the most important one and is used to set the socket path. Thus, it is possible to run several instances of Fail2ban on different sockets. However, this should be not required because Fail2ban can run several jails concurrently.

If fail2ban-server crashes (does it?), it is possible that the socket file has not been removed correctly. The -x option tells the server to delete the socket file before start-up. If the socket file of a running server is removed, it is not possible to communicate with this server anymore.

The server handles the signals SIGTERM and SIGINT. When receiving one of these signals, fail2ban-server will quit nicely.

Client

fail2ban-client is the frontend of Fail2ban. It connects to the server socket file and sends commands in order to configure and operate the server. The client can read the configuration files or can simply be used to send a single command to the server using either the command line or the interactive mode (which is activated with the -i option). fail2ban-client can also start the server. The following options are available for fail2ban-client:

-c <DIR> configuration directory -s <FILE> socket path -d dump configuration. For debugging -i interactive mode -v increase verbosity -q decrease verbosity -x force execution of the server -h, --help display this help message -V, --version print the version

As for fail2ban-server, the option -s <FILE> can be used to set the socket path. Notice that this command line option overrides the socket option set in fail2ban.conf. The default configuration directory is /etc/fail2ban but can be override with the -c <DIR> option. The -x option is simply forwarded to fail2ban-server when starting the server.

A useful option for debugging is -d. This prints the configuration parsed by fail2ban-client. The output corresponds to the stream sent to the server. If the output of -d shows:

['set', 'loglevel', 1] ['set', 'logtarget', 'STDERR']

It is possible to achieve the same with:

$ fail2ban-client set loglevel 1 $ fail2ban-client set logtarget STDERR

Everything set in the configuration files can be configured manually. The configuration is just a simple and efficient way to configure the server. fail2ban-client only translates the configuration into a suite of commands. However, fail2ban-client has 2 more commands for its internal use. The first one is start. When typing:

$ fail2ban-client start

the client will first try to fork a server instance. The client then waits for the server to start-up by sending ping requests to it. Once the server responds to these requests, fail2ban-client parses the configuration and sends the corresponding commands to the server. The second one is reload. When typing:

$ fail2ban-client reload

the client will tell the server to stop all jails, parses the configuration files again and send the commands to the server. This is useful when a new configuration must be loaded without shutting down the server. This is also very useful when debugging the server. It is possible to start the server with fail2ban-server -f in one terminal and to load the configuration by typing fail2ban-client reload in an other one. Thus, client and server output will not be mixed up.

Any other commands are simply sent to the server without any specific treatment. However, most of the time, only the 2 above commands and stop will be used.

There is probably one last useful command: status [jail]. Without a jail name, the global status of the server is returned. If jail corresponds to an existing jail, the status of this jail is displayed.

A list with all commands is available here.

Configuration

The standard path for the configuration is in /etc/fail2ban. This can be set with the -c option of fail2ban-client. A typical configuration looks like this:

/etc/fail2ban/ ├── action.d │ ├── dummy.conf │ ├── hostsdeny.conf │ ├── iptables.conf │ ├── mail-whois.conf │ ├── mail.conf │ └── shorewall.conf ├── fail2ban.conf ├── fail2ban.local ├── filter.d │ ├── apache-auth.conf │ ├── apache-noscript.conf │ ├── couriersmtp.conf │ ├── postfix.conf │ ├── proftpd.conf │ ├── qmail.conf │ ├── sasl.conf │ ├── sshd.conf │ └── vsftpd.conf ├── jail.conf └── jail.local

Every .conf file can be overridden with a file named .local. The .conf file is read first, then .local, with later settings overriding earlier ones. Thus, a .local file doesn't have to include everything in the corresponding .conf file, only those settings that you wish to override.

Modifications should take place in the .local and not in the .conf. This avoids merging problem when upgrading. These files are well documented and detailed information should be available there.

General settings

The file fail2ban.conf contains general settings for the fail2ban-server daemon, such as the logging level and target. You can also specify here the socket path used for communication between the client and the server.

Jails

The most important file is probably jail.conf, which contains the declaration of your jails. By default, some sections are inserted as templates. You must enable the sections of interest and adapt to your local configuration. Here is an example of the ssh-iptables section:

[ssh-iptables] #enabled = false enabled = true filter = sshd action = iptables[name=SSH, port=ssh, protocol=tcp] # mail-whois[name=SSH, dest=yourmail@mail.com] #logpath = /var/log/sshd.log logpath = /var/log/auth.log maxretry = 5

With these settings a few things will happen: the section ssh-iptables is enabled; the filter sshd.conf in sub-directory filter.d will be processed; the action(s) described in iptables.conf (sub-directory action.d) will be executed if the outcome of the filter process is true. In this example, the additional action mail-whois.conf is commented out. the log file to be scanned by the filter is auth.log.

Filter and actions are combined to create jails. Only one filter is allowed per jail, but it is possible to specify several actions, on separate lines. For example, you can react to a SSH break-in attempt by first adding a new firewall rule, then retrieving some information about the offending host using whois and finally sending an e-mail notification. Or maybe you just want to received a notification on your Jabber account when someone accesses the page /donotaccess.html on your web server.

Fail2ban is not limited to SSH. It contains default filters and actions for many daemons and services. You can easily modify them or create new ones. If you take a look in the filter.d you will notice a few default filters that don't occur in the standard jail.conf that come with the sources. In this example we take the "sshd-ddos.conf". To integrate the filter into fail2ban edit your jail.conf:

[ssh-ddos] enabled = true port = ssh,sftp filter = sshd-ddos logpath = /var/log/messages maxretry = 2

Always remember to adjust $logpath to your log-file as mentioned above.

Jail Options

Every jail can be customized by tuning following options:

Jail Options Name Default Description filter Name of the filter to be used by the jail to detect matches. Each single match by a filter increments the counter within the jail logpath /var/log/messages Path to the log file which is provided to the filter maxretry 3 Number of matches (i.e. value of the counter) which triggers ban action on the IP. findtime 600 sec The counter is set to zero if no match is found within "findtime" seconds. bantime 600 sec Duration (in seconds) for IP to be banned for. Negative number for "permanent" ban.

Filters

The directory filter.d contains mainly regular expressions which are used to detect break-in attempts, password failures, etc. Here is an example for filter.d/sshd.conf with 3 possible regular expressions to match the lines of the logfile:

failregex = Authentication failure for .* from <HOST> Failed [-/\w]+ for .* from <HOST> ROOT LOGIN REFUSED .* FROM <HOST> [iI](?:llegal|nvalid) user .* from <HOST>

In the above example the default regex has been changed to allow the absence of user in a line such as:

Jan 10 07:02:37 homebrou sshd[18419]: Failed password for root from 222.76.213.151 port 55236 ssh2

If you're creating your own failregex expressions, here are some things you should know:

A failregex can have multiple lines, any one of which may match a line of the log file.

In every line of failregex, the part that matches the host name or IP address must be wrapped in a (?P<host> ... ) sandwich. This is a Python-specific regex extension that assigns the contents of the match to the name <host>. The <host> tag is how you tell fail2ban which host was connecting, so it has to be present in every line of failregex. If it's not, fail2ban will issue an error message about "No 'host' group".

As a convenience, you can use the predefined entity <HOST> in your regexes. <HOST> is an alias for (?:::f{4,6}:)?(?P<host>\S+) , which matches either a hostname or an IPv4 address (possibly embedded in an IPv6 address).

In the action scripts, the tag <ip> will be replaced by the IP address of the host that was matched in the <host> tag.

In order for a log line to match your failregex, it actually has to match in two parts: the beginning of the line has to match a timestamp pattern or regex, and the remainder of the line has to match your failregex. If the failregex is anchored with a leading ^ , then the anchor refers to the start of the remainder of the line, after the timestamp and intervening whitespace.

The pattern or regex to match the time stamp is currently not documented, and not available for users to read or set. See Debian bug #491253. This is a problem if your log has a timestamp format that fail2ban doesn't expect, since it will then fail to match any lines. Because of this, you should test any new failregex against a sample log line, as in the examples below, to be sure that it will match. If fail2ban doesn't recognize your log timestamp, then you have two options: either reconfigure your daemon to log with a timestamp in a more common format, such as in the example log line above; or file a bug report asking to have your timestamp format included.

As an example of the above points, run the following commands in your console and compare the results:

fail2ban-regex 'Jul 18 12:13:01 [1.2.3.4] authentication failed' 'authentication failed' fail2ban-regex 'Jul 18 12:13:01 [1.2.3.4] authentication failed' '\[<HOST>\] authentication failed' fail2ban-regex '18-07-2008 12:13:01 [1.2.3.4] authentication failed' '\[<HOST>\] authentication failed' fail2ban-regex '18-7-2008 12:13:01 [1.2.3.4] authentication failed' '\[<HOST>\] authentication failed'

The 1st command fails, with a "No 'host' group" error message. The 2nd command succeeds, and finds the host address 1.2.3.4. The 3rd command succeeds, and finds the host address 1.2.3.4 (at least with v0.8.4). The 4th command fails. It does tell you why---it says "...no valid date/time found for..."---and clearly it's not able to match the unusual timestamp format---which is not part of your failregex.

Actions

The directory action.d contains different scripts defining actions. The actions are executed at well-defined moments during the execution of Fail2ban: when starting/stopping a jail, banning/unbanning a host, etc.

Testing

Many things can be tested after configuration but the following commands can help to verify your settings:

# fail2ban-client -d

will dump the current configuration.

# fail2ban-regex "line" "failregex"

will test a single regular expression failregex (such as given in sshd.conf) with a single line of your logfile. Don't forget the double quotes around your line and failregex definitions.

fail2ban-regex accepts files too. Thus, it is easier to test and debug your own regular expressions.

# fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

You can use a combination of both:

# fail2ban-regex /var/log/auth.log "Failed [-/\w]+ for .* from <HOST>"

Limitations

Reaction time

First of all, remember that Fail2ban is a log parser. It cannot do anything before something is written in the log files. Lots of syslog daemons buffer their outputs. This can impact performance of Fail2ban. Thus, it could be good to disable buffering of your syslog daemon.

It is quite difficult to evaluate the reaction time. Fail2ban waits 1 second before checking for new logs to be scanned. This should be fine in most cases. However, it is possible to get more login failures than specified by maxretry.

Here is an excerpt of a /var/log/auth.log:

Jan 15 15:28:46 homebrou sshd[29778]: Invalid user recruit from 81.74.87.66 Jan 15 15:28:46 homebrou sshd[29778]: error: Could not get shadow information for NOUSER Jan 15 15:28:46 homebrou sshd[29778]: Failed password for invalid user recruit from 81.74.87.66 port 47672 ssh2 Jan 15 18:33:27 homebrou sshd[2156]: Did not receive identification string from 88.191.23.27 Jan 15 19:23:37 homebrou sshd[3418]: Invalid user test from 88.191.23.27 Jan 15 19:23:37 homebrou sshd[3418]: error: Could not get shadow information for NOUSER Jan 15 19:23:37 homebrou sshd[3418]: Failed password for invalid user test from 88.191.23.27 port 41017 ssh2 Jan 15 19:23:38 homebrou sshd[3420]: Invalid user test from 88.191.23.27 Jan 15 19:23:38 homebrou sshd[3420]: error: Could not get shadow information for NOUSER Jan 15 19:23:38 homebrou sshd[3420]: Failed password for invalid user test from 88.191.23.27 port 41096 ssh2 Jan 15 19:23:38 homebrou sshd[3422]: Invalid user test from 88.191.23.27 Jan 15 19:23:38 homebrou sshd[3422]: error: Could not get shadow information for NOUSER Jan 15 19:23:38 homebrou sshd[3422]: Failed password for invalid user test from 88.191.23.27 port 41162 ssh2 Jan 15 19:23:38 homebrou sshd[3424]: Invalid user test from 88.191.23.27 Jan 15 19:23:38 homebrou sshd[3424]: error: Could not get shadow information for NOUSER Jan 15 19:23:38 homebrou sshd[3424]: Failed password for invalid user test from 88.191.23.27 port 41209 ssh2 Jan 15 19:23:39 homebrou sshd[3426]: Invalid user test from 88.191.23.27 Jan 15 19:23:39 homebrou sshd[3426]: error: Could not get shadow information for NOUSER Jan 15 19:23:39 homebrou sshd[3426]: Failed password for invalid user test from 88.191.23.27 port 41267 ssh2 Jan 15 19:23:39 homebrou sshd[3428]: Invalid user test from 88.191.23.27 Jan 15 19:23:39 homebrou sshd[3428]: error: Could not get shadow information for NOUSER Jan 15 19:23:39 homebrou sshd[3428]: Failed password for invalid user test from 88.191.23.27 port 41323 ssh2 Jan 15 19:23:40 homebrou sshd[3430]: Invalid user test from 88.191.23.27 Jan 15 19:23:40 homebrou sshd[3430]: error: Could not get shadow information for NOUSER Jan 15 19:23:40 homebrou sshd[3430]: Failed password for invalid user test from 88.191.23.27 port 41376 ssh2 Jan 15 19:23:40 homebrou sshd[3433]: Invalid user test from 88.191.23.27 Jan 15 19:23:40 homebrou sshd[3433]: error: Could not get shadow information for NOUSER Jan 15 19:23:40 homebrou sshd[3433]: Failed password for invalid user test from 88.191.23.27 port 41433 ssh2 Jan 15 19:23:41 homebrou sshd[3435]: Invalid user test from 88.191.23.27 Jan 15 19:23:41 homebrou sshd[3435]: error: Could not get shadow information for NOUSER Jan 15 19:23:41 homebrou sshd[3435]: Failed password for invalid user test from 88.191.23.27 port 41484 ssh2 Jan 16 12:13:43 homebrou sshd[32249]: Did not receive identification string from 209.126.131.150

Looking at /var/log/fail2ban.log around the corresponding period of time:

2007-01-15 15:38:47,142 fail2ban.actions: WARNING [ssh-iptables] Unban 81.74.87.66 2007-01-15 19:23:41,175 fail2ban.actions: WARNING [ssh-iptables] Ban 88.191.23.27 2007-01-15 19:23:42,373 fail2ban.actions: WARNING [ssh-iptables] 88.191.23.27 already banned 2007-01-15 19:33:41,508 fail2ban.actions: WARNING [ssh-iptables] Unban 88.191.23.27 2007-01-16 12:29:50,496 fail2ban.actions: WARNING [ssh-iptables] Ban 209.126.131.150

Thus, you can see that between 19:23:37 and 19:23:41, i.e. within 4 seconds, 9 login (ssh) attempts (instead of only 3) from 88.191.23.27 have been recorded in auth.log before it has been banned by Fail2ban.

Possibility of DOS attack by a local user

Because syslog entries can be 'forged' by a regular user, e.g.:

logger -p auth.warning -t 'sshd[123]' 'Illegal user user1 from 1.2.3.4'

Fail2Ban should not be run on systems which provide ssh/CGI/PHP services to unknown users without any level of trust. They might be able to block other users from ssh and probably other access with a call to logger. A malicious user may also write via PHP's openlog()/syslog() to syslog.

N.B. chmod o-x /usr/bin/logger might provide at least obfuscation solution.

Logging systems supporting authentication (e.g. journal) might provide adequate resolution for this in the future.