I — A Quick Word On D-Bus

Again, before jumping into architectural concerns and coding, let’s have a quick reminder about what is D-Bus and how it will help us achieving our goal. (if you’re a D-Bus expert, you can jump to section II).

D-Bus is an inter-process communication bus that, as its name underlines, allows multiple applications registered on it to communicate.

The applications using D-Bus are either servers or clients. A client connects to a D-Bus server and a server listens for incoming connections. Applications traditionally connect to a D-Bus server, they register to it and are given a well-known name to identify themselves.

From there, applications exchange messages and signals that may be captured by clients connected on the bus.

D-Bus purpose may be a bit hazy at first, but it comes as a very useful tool for your Linux system.

Applications such as the UPower service (responsible for monitoring your system power sources) could communicate with the thermald service (monitoring the overall temperature) to bring the power consumption down if some overheating problem were to be detected (without you even noticing it!)

Your system having 10 Google Chrome tabs open

So what’s the link between D-Bus and monitoring our systemd services?

Systemd is registered on D-Bus as the org.freedesktop.systemd1 service. Moreover, it sends some signals to the clients when the state of a systemd service changes over time.

This is what we are going to use for our whole system.

II — Capturing D-Bus signals

For this article, I am using a Xubuntu 18.04 machine with a standard Linux kernel. Our machine has to run the dbus-daemon and has to have the busctl utility installed.

One can check that this is the case by running :

(1) ps aux | grep dbus-daemon : results should enumerate at least two entries : one bus for the system and one bus for the session.

(2) busctl status : checks the bus status and returns the bus configuration.

a — Identifying useful D-Bus signals

As explained before, systemd service is registered on the bus and is sending signals when something related to systemd happens.

When a service starts, stops, or fails, systemd broadcasts signals on the bus to all available clients. As ton of messages are broadcasted by systemd when events occur, we are going to redirect the standard output to a file to analyze it.

sudo busctl monitor org.freedesktop.systemd1 > systemd.output will do the trick. Inspecting the file, we are presented with lots of messages, method calls, method returns and signals.

A systemd signal on the bus

See the ‘ActiveState’ line with the ‘deactivating’ value? That’s my InfluxDB service deactivating for a moment. We can even capture the timestamp related to the change!

The org.freedesktop.systemd service specifies 6 different states : active, reloading, inactive, failed, activating, deactivating. We are obviously particularly interested in the failed signal as it notifies a service failure on our system.

Now that we have a way to manually capture systemd signals on our system, let’s build our full automated system monitoring system.

Let the fun begin!

III — Architecture & Implementation

In order to monitor systemd services, we are going to use this architecture :