This article will be helpful for those who have Prometheus installed in their Kubernetes cluster, and willing to use custom business metrics that can be extracted from SQL database.

Pre-requisites

Kubernetes cluster

Prometheus running as a pod in your cluster

SQL Database

A short business case description

Components

Let’s describe a use case that we are going to implement. Say we have a source of data and we call it Publisher.

Another component is a Data ingestion service processing an important business logic and finally storing the result in any kind of Storage. Once the processing is finished it can have either Success or Failure result.

We also have a special Processing tracker (Tracker) service that is responsible for receiving the result from Data ingestion via HTTP channel. Tracker service is also connected to SQL database where the result will be persisted in.

The last but not least element is Prometheus that is currently only gathering Kubernetes cluster state.

Initial configuration

Task description

That’s great so far! Now a new requirement came to us. We have to be aware of those cases when data processing failed in Data ingestion. If that happened we must immediately notify our support team and the third new Subscription service that will perform a new subscription to the source of data.

Solution

At first glance, it looks simple. You need to add something to the infrastructure that can reach the database, execute a query and store the result somewhere. This result should be analysed and notification is performed.

Fortunately, there is a database scanner that is supported by a community that can scan your business information via SQL calls and store the result in Prometheus understandable format. Then, Prometheus itself is pinging this scanner and importing data.

Implementation phase

Prometheus exporters

The additional software (scanner) that provides us with Prometheus metrics format is called Prometheus exporter. In official documentation you can find a large list of available exporters. For our case, we are interested in postgresql exporter. Let’s first install it.

PostgreSQL exporter installation

We get back to our Kubernetes cluster and use helm in order to install PostgreSQL exporter. To speed up the things, you can use an existing helm chart. Your execution command will look like this:

helm upgrade --install --wait postgres-exporter \

-f ./helm/postgres-exporter/values.yaml \

stable/prometheus-postgres-exporter

The exporter will be trying to connect your database so we must fill the database configuration properties.

PostgreSQL exporter query definition

Once our exporter is able to connect to database we must define those queries that will be being executed each scraping time. You can mark each selected value in a query with one of three types of Prometheus metrics.

In our case, we would like to store the number of errors happened during the last 5 minutes in errors variable. So we mark errors as GAUGE metric (14–16). Since this point you can start searching for <query_name>_<GAUGE metric name> metric in your Prometheus database. The metric’s name will be pg_failing_event_processing_errors.

Moreover, you can include a couple of additional labels to your metric as LABELs type of the query as it is done with event name (17–19).

Connect Prometheus and PostgreSQL exporter

We assume you have already installed a Prometheus to the cluster. If not, you can find a perfect Helm chart and provision the cluster with it. Moreover, we will intensively be extending an input configuration file so you can wait until the end of the article.

Okay, now we need to add a new scraper that will be running on Prometheus side and extracting exported values from PostgreSQL exporter.

It looks pretty straightforward, right? Here is only a job name, frequency and targets of where to scrape the metrics. We just took a PostgreSQL exporter service that is installed automatically to the cluster with its helm chart and has 80 port by default (line 5).

Prometheus + PostgreSQL exporter

Alertmanager

Our system has to be provisioned with Alertmanager that will be responsible for sending notifications to Subscription service in case an alert has been triggered.

Alertmanager is delivered along with Prometheus chart. All you need is to provide an appropriate alertmanager.yml property:

Alertmanager configuration

You can find a comprehensive documentation about alerts and apply the configuration that fits to your purpose.

Alerts are Prometheus entities that contain PromQL expression. The expression is being evaluated from time to time and once expression became valid an appropriate alert is sent to Alertmanager.

Prometheus alert

Subscription service

You also need to develop a receiver for the alerts. In will be a Subscription service. We are going to introduce a /notifications endpoint for POST requests and message data model defined in documentation.

Validation

If you deploy the Prometheus now, put an alert with status ERROR to the database and wait for 5 minutes (you can reduce the time for testing purpose) then you will see that the alert has been detected an sent to Alertmanager that in turn will forward the message further to Subscription service.

Final Architecture

You are free to manage the incoming alerts in any way you wish. In our case we wanted Subscription service finding out a source of the event and notifying it about the error that occurred meaning we would like to receive that lost message once again.

Don’t forget, that Alertmanager is already able to sent messages to predefined systems. The approach that we are describing is a webhook based and intended to provide the most flexible way of alert message processing.

Conclusion

In this article, we have learned how to identify issues happening in the incoming messages processing and as a perform actions in order to not loose data.

We observed the existing implementations of the Prometheus exporters.

Next, we installed PostgreSQL exporter and defined a query that is being executed periodically. The result is grabbed by Prometheus and persisted internally.

We described how to configure an Alertmanager. After that, if special Prometheus entity Alert is triggered an approproate message will be generated and sent to the Alertmanager that will act as a proxy and forward the message to a webhook.