Using fail2ban and redis under ansible's orchestration for common good.

Fail2ban

Fail2ban is a service that parses log files and can perform configured actions when a given regex is found. Usually it is used to ban offending IP addresses using firewall rules on linux machines. In situations that we have more than one server under our responsibility, it could be better to apply those actions not only to the server experienced the offending behaviour. For example we may refuse connections to our web server from the IP that just messed up with our mail server and so on.

For this demonstration we will configure only one jail and one action.

Jail configuration (template):

This is a simple ssh jail that parses /var/log/secure.log and detects failed logins:

jail.local.j2

[DEFAULT] # Ban hosts for one hour: bantime = 3600 # Override /etc/fail2ban/jail.d/00-firewalld.conf: banaction = redis - publisher [sshd] enabled = true

Action configuration (template):

For the action configuration we will use the following file. It is a copy from the default firewallcmd-ipset.conf with the variables actionban and actionunban commented out and changed to call a python script with a JSON string as argument.

redis-publisher.conf.j2

[INCLUDES] before = iptables - common.conf [Definition] actionstart = ipset create fail2ban - < name > hash :ip timeout <bantime> firewall - cmd -- direct -- add - rule ipv4 filter <chain> 0 - p <protocol> - m multiport -- dports <port> - m set -- match - set fail2ban - < name > src - j <blocktype> actionstop = firewall - cmd -- direct -- remove - rule ipv4 filter <chain> 0 - p <protocol> - m multiport -- dports <port> - m set -- match - set fail2ban - < name > src - j <blocktype> ipset flush fail2ban - < name > ipset destroy fail2ban - < name > # actionban = ipset add fail2ban-<name> <ip> timeout <bantime> -exist actionban = /usr/sbin/publisher .py "{\" action\ ": \" ban\ ", \" ip\ ": \" <ip>\ ", \" bantime\ ": \" <bantime>\ ", \" name \ ": \" < name >\ ", \" published_by\ ": \" \ "}" # actionunban = ipset del fail2ban-<name> <ip> -exist actionunban = /usr/sbin/publisher .py "{\" action\ ": \" unban\ ", \" ip\ ": \" <ip>\ ", \" bantime\ ": \" \ ", \" name \ ": \" < name >\ ", \" published_by\ ": \" \ "}" [Init] chain = INPUT_direct bantime = 600

Redis

Redis is an open source key-value data store, used as a database, cache and message broker. Since redis is using memory for data storing it is extremely fast. We can use it for our little project as message broker in Pub/Sub (publish/subscribe) mode to distribute messages from any of our servers to all of them. You can use almost any programming languge to integrate with redis. We will use python and the recommended "redis-py" library for this demo.

About configuration, we leave everything to default changing only the binding ip address (by default redis listens only on localhost).

Redis configuration (template):

redis.conf.j2

bind {{ ansible_default_ipv4.address }} protected -mode yes port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 daemonize no supervised no pidfile /var/run/redis_6379.pid loglevel notice logfile /var/log/redis/redis.log databases 16 save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir /var/lib/redis slave-serve-stale-data yes slave-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-disable-tcp-nodelay no slave-priority 100 appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto -aof-rewrite-percentage 100 auto -aof-rewrite-min-size 64 mb aof-load-truncated yes lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-size - 2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256 mb 64 mb 60 client-output-buffer-limit pubsub 32 mb 8 mb 60 hz 10 aof-rewrite-incremental-fsync yes

Ansible

Ansible is the most known orchestration - automation tool that we honored here with multiple articles. It is simple, agentless and works! You need to be familiar with ansible to follow this demo.

We will write an ansible-playbook that we can run against groups of servers (eg. servers that are exposed to the web). If a new server is added later, we can add it's hostname in our ansible-inventory and run the same playbook again.

A simple inventory file with the groups needed for this demo is the following:

hosts

[ REDIS - CLIENTS ] aphrodite artemis athena [ REDIS - SERVER ] aphrodite

Note that redis server "aphrodite" is also a redis client.

The ansible playbook need to:

Install all required packages depending on whether host is running redis-server or just client.

Configure all the services handling restarts.

Secure the environment configuring firewall and selinux (if we are enforcing selinux policys).

For now, let us finish with prerequisites.

What else we need?

Excluding the above, we need two scripts:

The first one can connect to redis server, publish a new message and exit. Of course this will be triggered by fail2ban when offensive behavour is detected. The second will be run as linux service, listen for messages and perform system actions.

A simple "publisher" python script is the following (template). It is just publishing its first argument on execution (sys.argv[1]).

publisher.py.j2

#!/usr/bin/env python3 import sys import redis redis_server = "{{ groups['REDIS-SERVER'][0] }}" redis_port = 6379 redis_db = 0 redis_channel = 'fail2ban' def encryptor (message): # Out of article's scope return message def main (): message = encryptor(sys.argv[1]) r = redis.StrictRedis( host = redis_server, port = redis_port, db = redis_db ) p = r.pubsub() r.publish(redis_channel, message) p.close() main()

The script for redis listener is a little more complex. It gets the JSON message from redis turning it to a python dictionary object and executes the same command that fail2ban would execute locally. We also added a basic logger that is writing in /var/log/fail2ban.log file.

redis_listener.py.j2

#!/usr/bin/env python3 import redis import logging import json import subprocess redis_server = "{{ groups['REDIS-SERVER'][0] }}" redis_port = 6379 redis_db = 0 redis_channel = 'fail2ban' logfile = '/var/log/fail2ban.log' r = redis.StrictRedis(host = redis_server, port = redis_port, db = redis_db) p = r.pubsub() p.subscribe(redis_channel) logging.basicConfig( format = '%(asctime)s %(levelname)s: %(message)s' , filename = logfile, level = logging.DEBUG ) def decryptor (message): # Out of article's scope return message def execute (message): if message[ 'action' ] == 'ban' : command = 'ipset add fail2ban-{name} {ip} timeout \ {bantime} -exist' .format( * * message) else : command = 'ipset del fail2ban-{name} {ip} -exist' .format( * * message) logging.debug( '{host} triggered: {command}' .format( host = message[ 'published_by' ], command = command )) return_code = subprocess.call(command, shell = True ) if not return_code == 0: logging.info( 'Command: {command} failed with rc={rc}' .format( command = command, rc = return_code )) else : logging.debug( 'Command: {command} executed succesfully' .format( command = command )) def main (): for record in p.listen(): if record[ 'type' ] == 'message' : message = json.loads(decryptor(record)[ 'data' ].decode( 'utf-8' )) execute(message) main()

Comment/Disclaimer:

This article is basically a proof of concept. If you intend to use it on a production environment, you need to be serious about message validation and encryption. The script "as is" could allow any user of your servers inject shell commands as redis messages and run them as root on all your servers.

Since the script will run as linux service we also need a simple systemd unit file:

redis_listener.service

[Unit] Description = Redis Listener: Daemon that listen , ban and unban Before = network-pre.target Wants = network-pre.target After = polkit.service Conflicts = iptables.service ip6tables.service ebtables.service ipset.service [Service] ExecStart = /usr/sbin/redis_listener.py - -nofork - -nopid ExecStop = /bin/kill -HUP $MAINPID # supress to log debug and error output also to /var/log/messages StandardOutput = null StandardError = null Type = idle [Install] WantedBy = multi-user.target

The playbook

fail2ban-distributed.yml

- hosts: REDIS-CLIENTS vars: remote_user: ansible tasks: - name: Upgrade all packages yum: name: '*' state: latest become: yes - name: Install epel repository yum: name: 'epel-release' state: latest become: yes - name: Install fail2ban , redis, python3, pip yum: name: '{{ item }}' state: latest with_items: - fail2ban - redis - python34-pip - python34 become: yes notify: - restart fail2ban - restart redis - name: using pip to install python libraries pip: name: redis executable: pip3 become: yes - name: Configure redis template: src: redis.conf.j2 dest: /etc/redis.conf owner: redis group: root mode: 0640 when: "'REDIS-SERVER' in group_names" become: yes notify: - restart redis - name: Enable redis port on redis server only for redis clients firewalld: zone: public rich_rule: rule family=ipv4 source address= {{ hostvars[item][ 'ansible_enp0s3' ][ 'ipv4' ][ 'address' ] }} port protocol=tcp port=6379 accept permanent: true state: enabled with_items: "{{ groups['REDIS-CLIENTS'] }}" when: "'REDIS-SERVER' in group_names" become: yes notify: - restart firewall - name: Install publisher python script template: src: publisher.py.j2 dest: /usr/sbin/publisher.py owner: root group: root mode: 0700 become: yes notify: - restart fail2ban - name: Install fail2ban action conf template: src: redis-publisher.conf.j2 dest: /etc/fail2ban/action.d/redis-publisher.conf owner: root group: root mode: 0644 become: yes notify: - restart fail2ban - name: Configure fail2ban SSH jail template: src: jail.local.j2 dest: /etc/fail2ban/jail.local owner: root group: root mode: 0644 become: yes notify: - restart fail2ban - name: Copy selinux policy file copy: src: my-python3.pp dest: /root/my-python3.pp owner: root group: root mode: 0644 become: yes - name: Copy unit file to systemd copy: src: redis_listener.service dest: /etc/systemd/system/redis_listener.service owner: root group: root mode: 0755 become: yes notify: - restart redis_listener - name: Install redis_listener template: src: redis_listener.py.j2 dest: /usr/sbin/redis_listener.py owner: root group: root mode: 0700 become: yes notify: - restart redis_listener - name: install selinux policy file command: semodule -i /root/my-python3.pp become: yes handlers: - name: restart fail2ban systemd: name: fail2ban.service state: restarted enabled: True become: yes - name: restart redis systemd: name: redis.service state: restarted enabled: True become: yes when: "'REDIS-SERVER' in group_names" - name: restart firewall systemd: name: firewalld.service state: restarted enabled: True become: yes when: "'REDIS-SERVER' in group_names" - name: restart redis_listener systemd: name: redis_listener.service state: restarted enabled: True become: yes

Comments:

The playbook was tested on CentOS Linux release 7.4.1708 servers with ansible 2.4.2.0. We tried playbook task names to be descriptive. Network interface names are hardcoded in the playbook (enp0s3 - the same for all servers). If your environment is less standardized you can configure interfaces as host variables in your inventory. On firewall configuration (module firewalld) check how we are looping over [REDIS-CLIENTS] group and running only on [REDIS-SERVER]. If you are running selinux with "enforcing" policy you will find out that connection to redis server will be refused on clients. To overcome this, you need to generate and apply a new selinux policy (check the quote in the end of the article). In playbook, my-python3.pp file mentioned is generated manually once and installed to all clients.

Distributed fail2ban in action

The following lines are copied from /var/log/fail2ban.log file from all servers after some failed logins from ip 192.168.16.49 to server aphrodite:

aphrodite:

2017-12-29 15:53:09,868 fail2ban.filter [1007]: WARNING Determined IP using DNS Lookup: konstantinos.epilis.gr = ['192.168.16.49']

2017-12-29 15:53:09,868 fail2ban.filter [1007]: INFO [sshd] Found 192.168.16.49

2017-12-29 15:53:11,389 fail2ban.filter [1007]: INFO [sshd] Found 192.168.16.49

2017-12-29 15:53:22,485 fail2ban.filter [1007]: INFO [sshd] Found 192.168.16.49

2017-12-29 15:53:25,687 fail2ban.filter [1007]: INFO [sshd] Found 192.168.16.49

2017-12-29 15:53:30,318 fail2ban.filter [1007]: WARNING Determined IP using DNS Lookup: konstantinos.epilis.gr = ['192.168.16.49']

2017-12-29 15:53:30,319 fail2ban.filter [1007]: INFO [sshd] Found 192.168.16.49

2017-12-29 15:53:30,999 fail2ban.actions [1007]: NOTICE [sshd] Ban 192.168.16.49

2017-12-29 15:53:31,075 DEBUG: aphrodite triggered: ipset add fail2ban-sshd 192.168.16.49 timeout 3600 -exist

2017-12-29 15:53:31,094 DEBUG: Command: ipset add fail2ban-sshd 192.168.16.49 timeout 3600 -exist executed succesfully



artemis:

2017-12-29 15:53:31,503 DEBUG: aphrodite triggered: ipset add fail2ban-sshd 192.168.16.49 timeout 3600 -exist

2017-12-29 15:53:31,511 DEBUG: Command: ipset add fail2ban-sshd 192.168.16.49 timeout 3600 -exist executed succesfully



athena:

2017-12-29 15:53:31,414 DEBUG: aphrodite triggered: ipset add fail2ban-sshd 192.168.16.49 timeout 3600 -exist

2017-12-29 15:53:31,424 DEBUG: Command: ipset add fail2ban-sshd 192.168.16.49 timeout 3600 -exist executed succesfully

... an hour later:



aphrodite:

2017-12-29 16:53:31,021 fail2ban.actions [1007]: NOTICE [sshd] Unban 192.168.16.49

2017-12-29 16:53:31,101 DEBUG: aphrodite triggered: ipset del fail2ban-sshd 192.168.16.49 -exist

2017-12-29 16:53:31,121 DEBUG: Command: ipset del fail2ban-sshd 192.168.16.49 -exist executed succesfully



artemis:

2017-12-29 16:53:31,529 DEBUG: aphrodite triggered: ipset del fail2ban-sshd 192.168.16.49 -exist

2017-12-29 16:53:31,535 DEBUG: Command: ipset del fail2ban-sshd 192.168.16.49 -exist executed succesfully



athena:

2017-12-29 16:53:31,439 DEBUG: aphrodite triggered: ipset del fail2ban-sshd 192.168.16.49 -exist

2017-12-29 16:53:31,447 DEBUG: Command: ipset del fail2ban-sshd 192.168.16.49 -exist executed succesfully