CoreOS and Prometheus: Building monitoring for the next generation of cluster infrastructure

• By Fabian Reinartz

Prometheus is an open source monitoring system, begun at SoundCloud in 2013. The project was made public in early 2015 and since then has seen a rapid increase in adoption by the community. Prometheus’s ability to monitor a massive number of targets without consuming a massive amount of resources quickly made it an integral part of the toolkit of organizations shifting to cloud native infrastructure.

Prometheus: An overview

At the core of the Prometheus monitoring system is the main server, which ingests samples from monitoring targets. A target is any application that exposes metrics according to the open specification understood by Prometheus. Since Prometheus pulls data, rather than expecting targets to actively push stats into the monitoring system, it supports a variety of service discovery integrations, like that with Kubernetes, to immediately adapt to changes in the set of targets.

The second core component is the Alertmanager, implementing the idea of time series based alerting. It intelligently removes duplicate alerts sent by Prometheus servers, groups the alerts into informative notifications, and dispatches them to a variety of integrations, like those with PagerDuty and Slack. It also handles silencing of selected alerts and advanced routing configurations for notifications.

There are several additional Prometheus components, such as client libraries for different programming languages, and a growing number of exporters. Exporters are small programs that provide Prometheus-compatible metrics from systems that are not natively instrumented.

Prometheus and CoreOS

Enabling GIFEE — Google Infrastructure for Everyone Else — is a primary mission at CoreOS, and open source is key to that goal. CoreOS is a leading developer of open source and open specification components for cloud native infrastructure, with projects like etcd, rkt, and CoreOS Linux itself released under the Apache 2 license. We also have a team dedicated to contributing to the CNCF’s Kubernetes cluster orchestration manager.

Prometheus was initially created to handle monitoring and alerting in modern microservice architectures. It steadily grew to fit the wider idea of cloud native infrastructure. Though it was not intentional in the original design, Prometheus and Kubernetes conveniently share the key concept of identifying entities by labels, making the semantics of monitoring Kubernetes clusters simple. As we discussed previously on this blog, Prometheus metrics formed the basis of our analysis of Kubernetes scheduler performance, and led directly to improvements in that code. Metrics are essential not just to keep systems running, but also to analyze and improve application behavior.

All things considered, Prometheus was an obvious choice for the next open source project CoreOS wanted to support and improve with internal developers committed to the code base.

The road ahead

First, we will work to enhance Prometheus’s integration with Kubernetes and Tectonic, the CoreOS Kubernetes distribution for the enterprise. We hope to raise awareness of the importance of full instrumentation at all layers of container cluster infrastructure, and work toward establishing best practices and standards for monitoring and metrics formats.

Get started!

If you are interested in trying out Prometheus, take a look at the getting started guide. The project always welcomes new contributors. Don’t know where to start? Check the GitHub issues in some of the Prometheus repositories or talk with us on #prometheus on Freenode.

Stay tuned for a follow-up on how to run Prometheus on Kubernetes to monitor services and cluster components. In the meantime, learn how to run Kubernetes on CoreOS Linux and try out Tectonic for free today.

CoreOS is hiring Prometheus developers

If you're interested in helping define the future of monitoring in cloud native infrastructure, we're hiring developers in Berlin.

Don’t forget, CoreOS Fest is in Berlin, May 9 and 10, and you can learn more by joining us there. Be sure to purchase your ticket.

San Francisco community members are welcome to join us at our CoreOS Fest satellite viewing event on May 9. Be sure to secure your ticket ahead of time.