This blog post describes a set-up of a high-available web service with just two running servers. Quite a few examples on the internet make use of a third server as master, but it may create an unnecessary single-point-of-failure.

keepalived is normally used to provide a fail-over feature for load balancers, such as HAProxy. In this case, HAProxy does load balancing on your application servers and the setup requires at least 4 servers to make sense.

In this post, we skip the HAProxy layer as the setup is for our service, which has load-balancing built-in. What we need is a simple but robust fail-over for a high-availability service, no need to deploy more servers.

Network Setup

Server A is master - running the web service, server B is backup. If the master goes down, the backup will take over the web service for the master.

Let’s assume that our web service has a DNS A record 93.1.1.1 .

This is the floating IP address the current master server maps to itself. Once the master server A goes down, server B takes over and re-maps floating IP to itself. Besides that each server has its own public IP address on the en1 interface. The keep-alive mechanism works over the cross cable connected en0 interfaces of both servers, with static IP addresses. (If you don’t have this additional connection, you can easily use en1 wherever your read en0 .





Once the master server goes down situation looks like this:





The servers communicate with each other via en0 direct cable connection - using Virtual Router Redundancy Protocol (VRRP). It sends each second a keep-alive packet with advertisements. If there are 3 missing keep-alive packets, backup server will take over the floating IP address and designates itself as master.

In our setup, once the master server is back online, it starts sending advertisements to the backup server and re-takes the floating IP address back as it has higher priority.

This keep-alive mechanism is quite robust. Whenever the master server is not able to send keep-alives (e.g., power outage, network problem, OS problem, deadlock) the web service can be impaired as well so it makes sense to switch to the backup server. Floating IP address reassignment is done automatically by keepalived - by sending gratious ARP packet to the network. In this setup both servers must be in the same network segment.

Moreover Keepalived can check another services on the host with scripts. E.g., if the web service server is not running it can switch itself to fault state so backup server takes over the floating IP. More on this below or in the keepalived User Guide.

Installation

Keepalived is in the standard repositories, install it with yum / apt-get .

yum install keepalived

Server A configuration (Master)

The configuration file: /etc/keepalived/keepalived.conf

The constants in the configuration (IPs, interfaces) match the diagrams above.

! Configuration File for keepalived global_defs { notification_email { support@enigmabridge.com } notification_email_from server_a@enigmabridge.com smtp_server localhost smtp_connect_timeout 30 } vrrp_instance VI_1 { state MASTER # Server A is master interface en0 # VRRP is running on en0 interface virtual_router_id 151 priority 101 advert_int 1 dont_track_primary unicast_src_ip 10.0.0.2 # Server A IP unicast_peer { 10.0.0.3 # Server B IP } authentication { auth_type PASS auth_pass secret_password } virtual_ipaddress { 93.1.1.1/24 dev en1 } }

Server B configuration (Backup)

The changed directives here are:

state - by default, this server is the backup one.

priority - lower the priority number so after original master is back up again it takes the floating IP back

unicast_peer_ip - the IP address of the server B on the en0 interface

interface unicast_peer - ip address(es) of other servers, here we have only the server A (if not set, VRRP will do broadcasts)





! Configuration File for keepalived global_defs { notification_email { support@enigmabridge.com } notification_email_from server_b@enigmabridge.com smtp_server localhost smtp_connect_timeout 30 } vrrp_instance VI_1 { state BACKUP # Server B is backup by default interface en0 virtual_router_id 151 priority 100 advert_int 1 dont_track_primary unicast_src_ip 10.0.0.3 # server B IP unicast_peer { 10.0.0.2 # server A IP } authentication { auth_type PASS auth_pass secret_password } virtual_ipaddress { 93.1.1.1/24 dev en1 } }

Enable keepalived after start

CentOS 7:

systemctl enable keepalived.service

CentOS 6:

chkconfig keepalived on

Start keepalived

CentOS 7:

systemctl start keepalived.service

CentOS 6:

/etc/init.d/keepalived start

dont_track_primary

This configuration option is handy if two Keepalived instances are directly connected with a cross cable without any networking component.

If one server goes down, the en0 will be in the disconnected state on the other server - it’s like pulling out the cable from it. This confuses the other running server and may switch it to Fault state - not quite the thing we want as neither of the servers claims the floating address and the service becomes unavailable.

In logs in may look like this:

lis 21 13:53:36 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Down lis 21 13:53:36 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Transition to MASTER STATE lis 21 13:53:36 parrot NetworkManager[908]: <info> (en0): link disconnected lis 21 13:53:37 parrot Keepalived_vrrp[1763]: Kernel is reporting: interface en0 DOWN lis 21 13:53:37 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Entering FAULT STATE lis 21 13:53:37 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Now in FAULT state lis 21 13:53:39 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX lis 21 13:53:39 parrot NetworkManager[908]: <info> (en0): link connected lis 21 13:53:41 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Down lis 21 13:53:41 parrot NetworkManager[908]: <info> (en0): link disconnected lis 21 13:53:44 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Up 10 Mbps Full Duplex, Flow Control: RX/TX lis 21 13:53:44 parrot NetworkManager[908]: <info> (en0): link connected lis 21 13:53:44 parrot Keepalived_vrrp[1763]: Kernel is reporting: interface en0 UP lis 21 13:53:48 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Transition to MASTER STATE lis 21 13:53:48 parrot kernel: igb 0000:06:00.0 en0: igb: en0 NIC Link is Down lis 21 13:53:48 parrot NetworkManager[908]: <info> (en0): link disconnected lis 21 13:53:49 parrot Keepalived_vrrp[1763]: Kernel is reporting: interface en0 DOWN lis 21 13:53:49 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Entering FAULT STATE lis 21 13:53:49 parrot Keepalived_vrrp[1763]: VRRP_Instance(VI_1) Now in FAULT state

The dont_track_primary option will solve this Fault state problem on the peer down event.

Scripts monitoring (optional)

As mentioned above, keepalived can perform regular checks of the services and switch to fault state if not. Keepalived has embedded some checks already - HTTP_GET, SSL_GET or you can use your own check. If script returns 0 as the return value it means everything is OK. If it returns 1 the test fails and keepalived switches itself to fault state.

If your host provides some API the check script can check if API endpoint is operational.

vrrp_script chk_myscript { script "/usr/local/bin/check_api.py" interval 2 # check every 2 seconds fall 2 # require 2 failures for KO rise 2 # require 2 successes for OK timeout 10 # 10 second before failing due to timeout }

To enable this script checking, add the following code under the virtual_ipaddress block, inside the vrrp_instance directive.

track_script { chk_myscript }

Script example

The check script can for example look like the following one. It checks our API endpoint which is supposed to return JSON response with status field in it. If any parsing exception occurs, timeout 15 seconds, status field is missing or any other error is detected, script returns 1. As you guessed, this value means the check failed and the backup server should take over.

#!/usr/bin/env python import argparse import sys import requests import logging , coloredlogs logger = logging . getLogger ( __name__ ) coloredlogs . install ( level = logging . INFO ) CHECK_HOST = "127.0.0.1" CHECK_PORT = 11180 CHECK_TIMEOUT = 15 def main (): parser = argparse . ArgumentParser ( description = 'EnigmaBridge keepalived test' ) parser . add_argument ( '--host' , dest = 'host' , default = CHECK_HOST , help = 'Host to check' ) parser . add_argument ( '--port' , dest = 'port' , default = CHECK_PORT , type = int , help = 'port to check' ) parser . add_argument ( '--timeout' , dest = 'timeout' , default = CHECK_TIMEOUT , type = float , help = 'request timeout' ) args = parser . parse_args () host = args . host port = int ( args . port ) timeout = float ( args . timeout ) try : r = requests . get ( 'https:// % s: % d' % ( host , port ), timeout = timeout ) if r . status_code != 200 : raise ValueError ( 'Status code error: % s' % r . status_code ) js = r . json () if js is None : raise ValueError ( 'Json response is empty' ) if 'status' not in js : raise ValueError ( 'Status not in JSON' ) # Everything OK. sys . exit ( 0 ) except Exception as e : logger . info ( 'Exception: % s' % e ) sys . exit ( 1 ) if __name__ == '__main__' : main ()

Server A en0 static IP setup

It’s good to tell network manager not to mess with the names of network interfaces by adding NM_CONTROLLED=no to the networking scripts.

The networking script for server A en0 /etc/sysconfig/network-scripts/ifcfg-en0 :

TYPE=Ethernet BOOTPROTO=static HWADDR=d1:51:88:19:62:09 IPADDR=10.0.0.2 NETMASK=255.255.255.0 IPV4_FAILURE_FATAL=no IPV6INIT=no DEVICE=en0 ONBOOT=yes NM_CONTROLLED=no

Server B en0 static IP setup

The networking script for server B en0 /etc/sysconfig/network-scripts/ifcfg-en0 :

TYPE=Ethernet BOOTPROTO=static HWADDR=d1:51:88:19:60:09 IPADDR=10.0.0.3 NETMASK=255.255.255.0 IPV4_FAILURE_FATAL=no IPV6INIT=no DEVICE=en0 ONBOOT=yes NM_CONTROLLED=no

Fixed interface ordering

You may also want to fix the network interface ordering by its MAC address so it does not get changed on some system update.

In CentOS we do this by editing /etc/udev/rules.d/70-persistent-net.rules :

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="d1:51:88:19:62:09", ATTR{type}=="1", KERNEL=="eth*", NAME="en0" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="d1:51:88:19:62:10", ATTR{type}=="1", KERNEL=="eth*", NAME="en1"

Note: we decided not to use ethX as new names for the interfaces because there is some kind of race condition in the udev in CentOS operating system. If we change the order of more eth interfaces it may complain one is already taken. Choosing en is the safe option to avoid such races.

Keepalived: Invalid IP address, skipping…

It may happen keepalived logs something like this:

Nov 21 10:08:43 parrot Keepalived_vrrp[24465]: VRRP parsed invalid IP 93.1.1.1. skipping IP...

The IP looks good so why is keepalived complaining?

If you see this, make sure you have no weird white space characters in the configuration file. It may happen when you copy-paste a configuration snippet from a webpage, a notes application, or a text editor which added these white-space characters unrecognized by keepalived configuration file parser. In that case just re-write the configuration block virtual_ipaddress with vim on the server and restart. It solves this problem.

Configuration reload - service restart

After updating the keepalived configuration and restarting it with systemctl restart keepalived.service you may experience weird behavior of the service, for instance this can be found in logs:

lis 23 11:38:49 panda Keepalived_vrrp[8581]: receive an invalid ip number count associated with VRID! lis 23 11:38:49 panda Keepalived_vrrp[8581]: bogus VRRP packet received on en0 !!! lis 23 11:38:49 panda Keepalived_vrrp[8581]: VRRP_Instance(VI_1) Dropping received VRRP packet...

In this case we added a new IP address to the virtual_ipaddress block, restarted both services but it still kept sending these bogus packets.

The problem is restart with systemctl command does not work well for keepalived all the time and sometimes it may fail to reload configuration changes. Then one instances sends advertisements with 2 IP addresses and the other only with one which results in the described behaviour (the same problem occurs if the virtual IP is changed).

The solution is to stop keepalived instances on both servers, wait a few seconds and start them again. This solves the problem.

systemctl stop keepalived.service && sleep 2 && systemctl start keepalived.service

Scripts not working - fault state

It may happen the vrrp_script is not working properly. In that case Keepalived automatically fails to FAULT state without logging a proper error (we used Keepalived 1.2.13). Unfortunately there is also no log in /var/log/audit/audit.log. You can help yourself by attaching a strace to the running daemon to see what is going on:

strace -f -p 11222 [pid 12213] execve("/opt/enigmabridge/eb-keepalived/eb-controller-test.py", ["/opt/enigmabridge/eb-keepalived/"..., "--host", "panda.enigmabridge.com"], [/* 6 vars */]) = -1 EACCES (Permission denied)

The culprit is really a SELinux - at least in our case. Namely the executable check script has to have a type keepalived_unconfined_script_exec_t. If you dump the current SELinux file context you get:

semanage fcontext -l | grep -i keepali /var/run/keepalived. * regular file system_u:object_r:keepalived_var_run_t:s0 /usr/libexec/keepalived ( /. * ) ? all files system_u:object_r:keepalived_unconfined_script_exec_t:s0 /usr/lib/systemd/system/keepalived. * regular file system_u:object_r:keepalived_unit_file_t:s0 /usr/sbin/keepalived regular file system_u:object_r:keepalived_exec_t:s0

So you can either put the check script to the /usr/libexec/keepalived folder or add the tag to the check script. In our case the following worked:

semanage fcontext -a -t keepalived_unconfined_script_exec_t /opt/enigmabridge/eb-keepalived/eb-controller-test.py restorecon -Rv /opt/enigmabridge/eb-keepalived/eb-controller-test.py ls -lasZ /opt/enigmabridge/eb-keepalived/eb-controller-test.py -rwxr-xr-x. keepalived keepalived unconfined_u:object_r:keepalived_unconfined_script_exec_t:s0 /opt/enigmabridge/eb-keepalived/eb-controller-test.py

Extended SELinux policy

If your script also needs to write somewhere (e.g., /tmp folder) this may get a bit more complicated. The recommended approach is to inspect the audit log and create a new policy from that. There is a audit2allow tool that helps with generating SElinux policies from the audit logs (the name of the package is really the full path):

yum install /usr/bin/audit2allow

Then just inspect logs and create a policy files:

grep keepalived_t /var/log/audit/audit.log | audit2allow -M keepalived_t semodule -i keepalived_t.pp

Manual SELinux policy edit

You can also update policy file by hand keepalived_t.te :

module keepalived_t 1.0; require { type tmp_t; type keepalived_t; class dir { write add_name }; class file { write create open }; } #============= keepalived_t ============== allow keepalived_t tmp_t:dir { write add_name }; allow keepalived_t tmp_t:file { write create open };

And then rebuild the policy module and load it:

checkmodule -M -m -o keepalived_t.mod keepalived_t.te semodule_package -o keepalived_t.pp -m keepalived_t.mod semodule -i keepalived_t.pp

Testing

In order to test the configuration you can either stop the keepalived service or reboot the master server. Here I use the first method:

systemctl stop keepalived.service

Then server B log - switching to the master state and taking over the floating IP:

lis 22 14:51:07 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Transition to MASTER STATE lis 22 14:51:08 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Entering MASTER STATE lis 22 14:51:08 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) setting protocol VIPs. lis 22 14:51:08 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1 lis 22 14:51:08 parrot Keepalived_healthcheckers[3307]: Netlink reflector reports IP 93.1.1.1 added lis 22 14:51:13 parrot Keepalived_vrrp[3308]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1

The interface en1 now has assigned IP 93.1.1.1.

When we re-start the keepalived service on the server A:

systemctl start keepalived.service

Server A will start keepalived and takes over the floating IP:

lis 22 14:52:29 panda Keepalived[25936]: Starting VRRP child process, pid=25938 lis 22 14:52:29 panda Keepalived_vrrp[25938]: Netlink reflector reports IP 10.0.0.2 added lis 22 14:52:29 panda Keepalived_vrrp[25938]: Netlink reflector reports IP 93.1.1.2 added lis 22 14:52:29 panda Keepalived_vrrp[25938]: Registering Kernel netlink reflector lis 22 14:52:29 panda Keepalived_vrrp[25938]: Registering Kernel netlink command channel lis 22 14:52:29 panda Keepalived_healthcheckers[25937]: Configuration is using : 7915 Bytes lis 22 14:52:29 panda systemd[1]: Started LVS and VRRP High Availability Monitor. lis 22 14:52:29 panda Keepalived_healthcheckers[25937]: Using LinkWatch kernel netlink reflector... lis 22 14:52:29 panda Keepalived_vrrp[25938]: Registering gratuitous ARP shared channel lis 22 14:52:29 panda Keepalived_vrrp[25938]: Opening file '/etc/keepalived/keepalived.conf'. lis 22 14:52:29 panda Keepalived_vrrp[25938]: Truncating auth_pass to 8 characters lis 22 14:52:29 panda Keepalived_vrrp[25938]: Configuration is using : 64580 Bytes lis 22 14:52:29 panda Keepalived_vrrp[25938]: Using LinkWatch kernel netlink reflector... lis 22 14:52:29 panda Keepalived_vrrp[25938]: VRRP sockpool: [ifindex(4), proto(112), unicast(1), fd(10,11)] lis 22 14:52:30 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Transition to MASTER STATE lis 22 14:52:30 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election lis 22 14:52:31 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Entering MASTER STATE lis 22 14:52:31 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) setting protocol VIPs. lis 22 14:52:31 panda Keepalived_healthcheckers[25937]: Netlink reflector reports IP 93.1.1.1 added lis 22 14:52:31 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1 lis 22 14:52:36 panda Keepalived_vrrp[25938]: VRRP_Instance(VI_1) Sending gratuitous ARPs on en1 for 93.1.1.1

Server B switching back to backup state: