Oracle Cloud Infrastructure offers the choice, flexibility, control, and performance that your applications and workloads need. Load balancer is a basic building block of any High Available architecture. A load balancer improves resource utilization, facilitates scaling, and helps ensure high availability.

This paper outlines the process of setting up HAProxy based Load Balancer on OCI. The load balancer works in Active-Active mode. You can fully customize the load balancing configuration. For example, you can modify the SSL ciphers and protocol supported as required by your application, you can set up session persistence which is driven by the load balancer rather than application cookie based one and add any customization you like to have on the HAProxy load balancer.

Architecture Overview

Let’s take a quick look into how High Availability is achieved in this architecture.

Architecture

The environment consists on 2 HAProxy nodes and 2 Web Server nodes. Both the HAProxy nodes are assigned a Reserved Public IP and an additional secondary Private IP. The Reserved Public IP is assigned to the Primary Private IP in normal working mode. In the event of an HAProxy node failure, the Public IP will failover to the secondary Private IP of the other HAProxy node using Keepalived. When the failed node recovers, the Public IP will return to its original instance. Please remember that the functional Load Balancer will handle request on both the Public IPs till the failed node comes back on-line and the IP failback happens to the original node.

Keepalived is configured to use VRRP packets in uses couple of helper scripts to failover the Public IP. The helper scripts use OCI CLI and instance profile based authentication which is configured by IAM modules. Additional customization is also possible on keepalived service such as multiple health checks to ensure Keepalived correctly detect node/service failure, email notifications can be setup to alert you on a node/service failure.

DNS Records

A functional environment of this architecture requires the Customer to setup their DNS hosted zone to map both the HAProxy node’s Reserved Public IPs to the same A record (Hostname).

IAM Permissions

Keepalived runs as root user and hence the helper scripts. This walks you through creating a Dynamic Group and attaching a Policy to it for the helper scripts to function. However, you must launch and configure the service nodes using an admin user.

Network Configuration

The environment consists of a VCN with 2 Public Subnets for HAProxy LB nodes and 2 Private Subnets for Web Server nodes and includes the other VCN components.

Below are the details:

- VCN:

Name: myvcn

- Internet Gateway

Name: IGW

- NAT Gateway

Name: NGW

- Route Tables

- Security Lists

a. Name: LBSecList

Ingress Rules:

Egress Rules:

b. Name: WebSecList

Ingress Rules:

Egress Rules:

- Subnets:

Create Compute Instances

Create the following Instances:

- Assign Reserved Public IP address to Load Balancer nodes

We use Reserved Public IP address on the Load Balancer Nodes. You need to create 2 Reserved Public IP addresses and assign them to the Primary VNIC of the Load balancer nodes.

NOTE: If the instance already has Public IP assigned, you need to unassign the Public IP by selecting ‘No Public IP’ for Public IP Type in the Public IP Address section. NOTE: You need to do this on both the LB nodes hap1 & hap2

o Under Core Infrastructure, go to Compute and click Instances.

Click the instance to view its details.

o Under Resources, click Attached VNICs.

Select the Primary VNIC of the Node.

o Under Resources, click IP Addresses and select the primary Private IP.

o Click the Actions icon (three dots), and then click Edit.

o In the Public IP Address section, for Public IP Type, select the radio button for Reserved Public IP.

o Enter the following:

- Compartment: Select your compartment.

- Reserved Public IP: The reserved public IP you want to assign.

Select Create a new reserved public IP.

You may optionally provide a friendly name for it.

o Click Update.

Install and Configure Web Servers

Let’s configure the Web Servers first. You could host any application on the backends. Since the focus of this document is on fully customizable and High Available HAProxy Load Balancer, I am setting up a simple Web Servers using HTTPD server running in the backends.

You need to install and configure HTTPD on both the web server nodes.

$ sudo yum update -y $ sudo yum install -y httpd httpd-tools mod_ssl openssl

Make sure the Web Server listens on Port 80 and DocumentRoot is set to /var/www/html.

Sample Configuration:

$ sudo sed ‘/ *#/d; /^ *$/d’ /etc/httpd/conf/httpd.conf ServerRoot “/etc/httpd” Listen 80 Include conf.modules.d/*.conf User apache Group apache ServerAdmin root@localhost <Directory /> AllowOverride none Require all denied </Directory> DocumentRoot “/var/www/html” <Directory “/var/www”> AllowOverride None Require all granted </Directory> <Directory “/var/www/html”> Options Indexes FollowSymLinks AllowOverride None Require all granted </Directory> <IfModule dir_module> DirectoryIndex index.html </IfModule> <Files “.ht*”> Require all denied </Files> ErrorLog “logs/error_log” LogLevel warn <IfModule log_config_module> LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”” combined LogFormat “%h %l %u %t \”%r\” %>s %b” common <IfModule logio_module> LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %I %O” combinedio </IfModule> CustomLog “logs/access_log” combined </IfModule> <IfModule alias_module> ScriptAlias /cgi-bin/ “/var/www/cgi-bin/” </IfModule> <Directory “/var/www/cgi-bin”> AllowOverride None Options None Require all granted </Directory> <IfModule mime_module> TypesConfig /etc/mime.types AddType application/x-compress .Z AddType application/x-gzip .gz .tgz AddType text/html .shtml AddOutputFilter INCLUDES .shtml </IfModule> AddDefaultCharset UTF-8 <IfModule mime_magic_module> MIMEMagicFile conf/magic </IfModule> EnableSendfile on IncludeOptional conf.d/*.conf

- Update Firewalld/IPTables to allow web server ports

$ sudo firewall-cmd — permanent — add-service=httpd $ sudo firewall-cmd — reload

- Set up the default page for the web server

On hap1 node:

$ sudo echo “Node hap1 — Web Server” > /var/www/html/index.html

On hap2 node:

$ sudo echo “Node hap2 — Web Server” > /var/www/html/index.html

- Start and enable HTTPD service

$ sudo systemctl enable httpd.service $ sudo systemctl start httpd.service

Install and configure HAProxy Service

HAProxy needs to be installed and configured on both the hap1 and hap2 nodes. The configuration file is just the same on both the nodes.

$ sudo yum update -y $ sudo yum install -y haproxy

Backup the configuration file:

$ sudo cp -a /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig

- HAProxy LB configuration:

HAProxy configuration file:

$ sudo cat /etc/haproxy/haproxy.cfg # — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — - # Global settings # — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — - global # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats # — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — - defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend http bind *:80 default_backend http backend http mode http balance roundrobin cookie SERVERID insert indirect server web1 <web1 IP>:80 check cookie web1 server web2 <web2 IP>:80 check cookie web2

Replace <web1 IP> and <web2 IP> with the Private IP address of the Web Servers.

- Update Firewalld/IPTables with HAProxy port and VRRP protocol

$ sudo firewall-cmd — permanent — add-service=http $ sudo firewall-cmd — permanent — add-rich-rule=’rule protocol value=\”vrrp\” accept’

- Enable IP forwarding and binding to non-local IP addresses

$ sudo echo “net.ipv4.ip_forward = 1” >> /etc/sysctl.conf $ sudo echo “net.ipv4.ip_nonlocal_bind = 1” >> /etc/sysctl.conf $ sudo sysctl -p net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1

- Start and enable HAProxy

$ sudo systemctl enable haproxy $ sudo systemctl start haproxy

You should now be able to access the web servers via the HAProxy node’s Public IPs.

Create Secondary Private IP on HAProxy nodes

The secondary Private IP is used to assign the Public IP of the other HAProxy node during failover event. Keepalived uses the helper scripts do the IP failover and failback.

You need to set up secondary private IP on both the HAProxy nodes.

- Create secondary private IP

You can create it from Console/CLI/API. However, do not assign Public IP. Make a note of the private IP.

- Configure secondary IP on the OS

Creating a secondary private IP on the OCI API does not automatically create additional private IP on the OS. You need to manually set it up. We suggest persistent IP configuration as follows.

$ sudo cat /etc/sysconfig/network-scripts/ifcfg-ens3:1 DEVICE=”ens3:1" BOOTPROTO=none IPADDR=<IP Address> NETMASK=255.255.255.0 ONBOOT=yes ONPARENT=yes

Replace <IP Address> with the secondary IP got assigned from the previous step. Refer: https://docs.cloud.oracle.com/iaas/Content/Network/Tasks/managingIPaddresses.htm#linux

Install OCI CLI

$ sudo yum install -y python-oci-cli

You need not update the configuration with SSH keys as we are using Dynamic groups for authentication.

Configure Dynamic Groups for the LB nodes

The keepalived helper scripts used to failover the public IPs needs OCI CLI configured with credentials. The easiest way to configure the same is to set up IAM Dynamic Group with the least permissions.

Matching Rule for the dynamic group can be:-

instance.compartment.id = ‘<Compartment OCID>’

IAM policy for the Dynamic group required here is:

Allow dynamic-group haproxy_dynamic_group to manage public-ips in compartment <Compartment Name> Allow dynamic-group haproxy_dynamic_group to manage private-ips in compartment <Compartment Name>

Install and configure Keepalived

Our environment has 2 keepalived instances, one each for both the EIPs. Keepalived uses VRRP protocol for HA. Keepalived also uses Public IP migration scripts using OCI CLI to assign the Public IP to VNIC of the functional HAProxy instance in case of a failover. Hence OCI CLI needs to be installed and configured for root user.

$ sudo yum install -y keepalived

Keepalived configuration

- On node hap1

# cat /etc/keepalived/keepalived.conf vrrp_script chk_haproxy { script “pidof haproxy” interval 5 weight -4 fall 2 rise 1 } vrrp_instance vrrp_1 { interface ens3 virtual_router_id 1 state MASTER priority 201 unicast_src_ip <Self Primary IP Address> unicast_peer { <Peer IP Address> } authentication { auth_type PASS auth_pass <Password> } track_script { chk_haproxy } notify_master /etc/keepalived/ip_failback11.sh } vrrp_instance vrrp_2 { interface ens3 virtual_router_id 2 state BACKUP priority 200 unicast_src_ip <Self Primary IP Address> unicast_peer { <Peer Primary IP Address> } authentication { auth_type PASS auth_pass <Password> } track_script { chk_haproxy } notify_master /etc/keepalived/ip_failover12.sh notify_backup /etc/keepalived/ip_failback_sec1.sh }

Replace <Self Primary IP Address> with Primary Private IP address of node hap1. Replace <Peer IP Address> with Primary Private IP address of node hap2. Replace <Password> with some random passwords for both the vrrp instances.

- Public IP migration helper scripts: (On node hap1)