Send Yourself a Telegram when a Systemd-Unit Fails

For CurrySearch we needed a reliable way to tell if everything is running as expected. Along with collectd and central logging, notifications on unit failure help us ensure the reliability of the system and act swiftly in case of disaster.

You may like or dislike systemd but on way or the other, someday, you probably have to deal with it. This blogpost describes a way to get notifications through telegram when a systemd-unit fails.

Telegram is a modern messenger app with some unique features. Among others, it is possible to create bots that interact with users.

Based on an article on the archlinux wiki, which describes notifications via e-mail, we will show you how to make systemd units send their status to you through telegram.

You can find the results on GitHub.

When done everything right, you will be able to:

telegram "message" in any shell script: Your bot will send you “message” unit-status-telegram systemd-unit-name in any shell script: Your bot will send you the status message of systemd-unit Place OnFailure=unit-status-telegram@%n.service in the Unit section if any systemd-unit and be notified on its failure in telegram

Getting the ApiKey and Chat-Id

First you need a bot api-key and your chat id: Use @botfather to get an api-key. Next you can message your bot using its username and get the chat-id using from url:

https://api.telegram.org/bot$API_KEY/getUpdates

With the api-key and the chat-id we can now send us a telegram message from commandline:

curl -s -d "chat_id= $CHAT_ID &disable_web_page_preview=1&text=MESSAGE" https://api.telegram.org/bot $TELEGRAM_KEY /sendMessage

Talk to yourself

A bit of abstraction and cleanup results in the following bash-script.

telegram.sh:

#!/bin/bash source /etc/telegram/key.sh URL = "https://api.telegram.org/bot $TELEGRAM_KEY /sendMessage" curl -s -d "chat_id= $CHAT_ID &disable_web_page_preview=1&text= $1 " $URL > /dev/null

With key.sh beeing:

export TELEGREM_KEY = "API-KEY" export CHAT_ID =CHAT_ID

Move the telegram.sh into /usr/bin and set the x flag. Now you can simply run

telegram "Message"

to message yourself.

Systemd Unit Status

To message yourself with a systemd unit status use. unit-status-telegram.sh:

#!/bin/bash UNIT = $1 UNITSTATUS = $( systemctl status $UNIT ) ALERT = $( echo -e "\u26A0" ) telegram " $ALERT Unit failed $UNIT $ALERT Status: $UNITSTATUS "

Place the script into /usr/bin/, make sure it is executable and test it:

unit-status-telegram systemd-journald

Resulting in something like:

The systemd unit

Now we only need a systemd unit, that we can trigger on failure.

unit-status-telegram@.service:

[Unit] Description=Unit Status Telegram Service After=network.target [Service] Type=simple ExecStart=/usr/bin/unit-status-telegram %I

Place it under /etc/systemd/system/unit-status-telegram@.serivce.

Now we can finally add

OnFailure=unit-status-telegram@%n.service

to the [Unit] section of any service.