With the popularity of Python as a language on the rise and Kubernetes surge to prominence as the leading cloud native platform, you may soon find yourself at the intersection of both. Developing production grade applications on Kubernetes means including monitoring and metrics gathering with Prometheus. Prometheus is the monitoring and alerting component of Kubernetes highly leveraged by operations.

Python delivers a great library to integrate with Prometheus, prometheus_client. The prometheus_client library provides the basic metrics components like counters, gauges, summary, etc. and stores this data in the collection registry. Beyond this popular frameworks, like Flask & Django, have provided extensions (prometheus-flask-exporter, django-prometheus) that build upon the client to create useful predefined metrics out of the box.

While these modules are great for REST API monitoring, you may find yourself in need of worker processes to execute asynchronous tasks that can be started via a trigger whether that be REST, message via message queue, and/or scheduled heartbeat. The most popular python distributed task processing implementation is Celery.

Monitoring Celery with a Simple Library

Yet again open source provides a solution for monitoring Celery. If you have been using Celery in your stack for some time you may be familiar with Flower. Flower is a monitoring application for Celery that provides a Graphical User Interface (GUI) for looking a counts, trends, and diving deeper in to the tasks/messages responsible for the workers actions.

Monitoring Celery Tasks in Flower

Flower is a great tool for debugging but it doesn’t provide an alerting mechanism nor an integration with Prometheus to help tie this portion of your application in with your other components from an operational perspective. Luckily a celery exporter, celery-prometheus-exporter, exists that provides much of the features of Flower but allowing the desirable integration with Prometheus to show metrics in context of the rest of the application.

Deploying a Celery Exporter on OpenShift

Now that we have the library let’s get it up and running. The Github repository for the exporter does provide an example Dockerfile, so one could go that route for deployment on OpenShift; however, we will be working through an implementation that is utilizing source-to-image (s2i) for deployment. Leveraging s2i means you can develop locally with straight Python or on Kubernetes depending on your preferences. With two small steps we’ll enable an existing Celery application to be monitored with Prometheus. First step is to install the dependency on your existing application:

pipenv install celery-prometheus-exporter

This will enable us to reuse the repository that is currently being deployed via s2i and just alter the deployment command to launch the celery exporter at startup.

Next step is to create a deployment template YAML.

Template for Prometheus Celery Monitoring App

The image on the left is from the template that defines the monitoring application employed in my team’s OpenShift deployment. This template defines the BuildConfig, DeploymentConfig, ImageStream, Service, and Parameters. Two of the configuration parameters RABBITMQ_HOST and PROMETHEUS_DIR are defined in a separate template that defines the ConfigMap utilized by several of the deployed applications. The most critical portion of this deployment is the command specified in the DeploymentConfig:

celery-prometheus-exporter — enable-events — broker=amqp://${RABBITMQ_HOST}:5672// — verbose

The command that is executed launches HTTPD running on port 8888, which makes the metrics collection endpoint available to be “scraped” by Prometheus. The broker variable allows the application to connect to your Celery broker, RabbitMQ in our case, to monitor the activities of tasks using the data in the Celery broker.

You would deploy this application with the following steps using the OpenShift command line tools:

oc apply -f ${OPENSHIFT_TEMPLATE_PATH}

The above command will define the “masu-monitor-template” template in the project/namespace where the oc apply command was run. The next step depends on the ConfigMap mentioned earlier being previously created, but you could certainly extract the necessary configuration into a single template file if needed.

oc new-app — template ${OPENSHIFT_PROJECT}/masu-monitor-template \

— param-file=${MASU_MONITOR_ENV} \

— param NAMESPACE=${OPENSHIFT_PROJECT}

The above command references two variables OPENSHIFT_PROJECT and MASU_MONITOR_ENV. The OPENSHIFT_PROJECT variable is used to locate the template and locate shared resources, such as RabbitMQ, in that namespace. MASU_MONITOR_ENV is a variable to reference a parameter file that supplies default variables for other values within the template like resource requests and limits.

Prometheus Monitoring in Action

Once you have managed to deploy your application you can sit back and reap the benefits of your monitoring. With the above monitoring application deployed Prometheus just needs to be configured to scrape the available metrics from the operating endpoint. Assuming your Prometheus is not deployed in the same namespace as your Celery monitoring application you can use the local cluster reference to the monitoring application service as described below:

Prometheus Configuration to Scrape Metrics via Local Cluster Service Name

In the above example “masu-monitor” is the name of the DeploymentConfig. “myproject” is the project name from the default parameters file. The full line is a reference for the Service that was defined in the template for this referenced project in the local cluster.

Now in your Prometheus instance you should be able to see the following metrics:

Celery Metrics available in Prometheus

Celery Task Latency Example

Celery Task Latency Bucket Example

Celery Worker Monitoring Example

Celery Task Runtime Example

From here you can select the metrics and views that matter the most to your applications and create associated alerts to have production-level insight into your asynchronous task execution.

Synopsis

Hopefully this story has shown you in a few steps how you can add Prometheus monitoring for your Celery applications. We covered some of the available clients for integrating with Prometheus. How to deploy a Celery monitoring application in OpenShift, which utilizes the data already maintained by your Celery broker to gather metrics for Prometheus. Lastly, we looked at the configuration updates needed in Prometheus to scrape these metrics from the monitoring application along with example metrics from an operational application.