To troubleshoot the Router of the OpenShift, the logs are essential, especially the HAProxy logs on how the traffic is handled and forwarded. However, the HAProxy pumping the log to a rsyslog server by default, the Pod log of the Router is not able to see it.

Instead of setting up a rsyslog server outside of the cluster, this paper explores a “cloud-native” way of checking the HAproxy logs for debugging purposes.

Identify the container

The image of rsyslog/syslog_appliance_alpine seems like a good fit.

Customize the config file, rsyslog.conf , as below

global(processInternalMessages="on") module(load="imrelp")

module(load="imptcp")

module(load="imudp" TimeRequery="500") module(load="omstdout")

module(load="omelasticsearch") module(load="mmjsonparse")

module(load="mmutf8fix") input(type="imptcp" port="514")

input(type="imudp" port="514")

input(type="imrelp" port="1601") syslog.* :omstdout: include(file="/config/droprules.conf" mode="optional") # this permits the user to easily drop unwanted messages

action(name="main_utf8fix" type="mmutf8fix" replacementChar="?") # dump the haproxy log to console

local1.* :omstdout:

For debugging purposes, we emit the haproxy log to the stdout only.

Create a ConfigMap in the OpenShift cluster to host the config file.

oc new-project rsyslog

oc create cm cm-rsyslog-config --from-file=rsyslog.conf=rsyslog.conf

Create the deployment

Create the following deployment, apply it in the rsyslog namespace

apiVersion: apps/v1

kind: Deployment

metadata:

name: rsyslog

labels:

app: rsyslog

spec:

selector:

matchLabels:

app: rsyslog

template:

metadata:

labels:

app: rsyslog

spec:

securityContext:

runAsUser: 0

containers:

- name: rsyslog

image: rsyslog/syslog_appliance_alpine

ports:

- containerPort: 514

env:

- name: RSYSLOG_CONF

value: /myconfig/rsyslog.conf

volumeMounts:

- mountPath: /myconfig

name: rsyslog-config

- mountPath: /config

name: config

- mountPath: /work

name: work

- mountPath: /logs

name: logs volumes:

- name: rsyslog-config

configMap:

name: cm-rsyslog-config

- name: config

emptyDir: {}

- name: work

emptyDir: {}

- name: logs

emptyDir: {}

Mount the volume for /config, /work, /logs with emptyDir. Mount the config file from the configMap. Use the environment variable to point to the config.

As it will listen on port 514, we need to assign the pod to run as root. Set the security context in the pod. In addition, assign the anyuid SCC (Security Context Constraints) to the default service account.

oc adm policy add-scc-to-user anyuid -z default

The deployment pod is running.

Expose service

Expose the rsyslog service within the cluster by using the default ClusterIP. Apply the following yaml file,

apiVersion: v1

kind: Service

metadata:

name: rsyslog

spec:

selector:

app: rsyslog

ports:

- name: udp

protocol: UDP

port: 514

targetPort: 514

Now the rsyslog will be available in the cluster at the address of rsyslog.rsyslog.svc

View Router’s log

Update the router’s environment setting,

oc -n default set env dc/router ROUTER_SYSLOG_ADDRESS=rsyslog.rsyslog.svc ROUTER_LOG_LEVEL=debug

When the router pod is restarted, the HAproxy’s log is available by check the rsyslog pod stdout,

oc -n rsyslog log rsyslog-675c46ff6f-hnrww -f

A sample is shown below,