February 10, 2015

Prometheus

Prometheus is a new open-source service monitoring system and time series database written in Go.

Check out the announcement and my article about monitoring Docker Containers with Prometheus if you don’t know what I’m talking about.

My Stack

I stopped running my own full blown server(s) a while ago. Nowadays I just have a old Raspberry Pi at home and two tiny DigitalOcean instances to host this blog and for general R&D stuff.

But I still want to monitor all this and, as showed earlier, Prometheus is the way to go. So I could now spin up another DigitalOcean instance and pay another $5/Mo, but given how cheap I am, I’d rather want to run it on my Raspberry Pi.

Cross-Compiling Go

First you need to build Go with support for your target OS and architecture. If you installed Go from sources, you can do this by running:

cd $GOROOT/src GOARCH=arm ./make.bash

With pure Go, cross compilation is trivial. This example:

package main import "fmt" func main() { fmt.Println("Hello World") }

can be cross-compiling for arm with GOARCH=arm go build test.go .

Now things get much more complicated once you use CGO, meaning Go code calling C functions.

The Prometheus server uses the Prometheus Go Client Library to provide metrics about itself. This client library uses prometheus/procfs which requires CGO to get process metrics from procfs. Cross-compiling this for Raspberry Pi is a pain. ARM != ARM, there are several variants and when I tried to cross-compiling Prometheus with CGO, it just lead to segfaults or invalid instructions. It should be possible to cross-compile it with the CGO dependency, it’s just painful and I quickly gave up.

Fortunately, we found an easy way to remove the dependency on procfs if CGO is disabled: https://github.com/prometheus/client_golang/commit/93d11c8e35ffcd969fd881efe1873e715a6ef93b. This removes some useful process level metrics about prometheus itself, but it makes cross compiling prometheus is as easy as in the example above:

cd $GOPATH/src/github.com/prometheus/prometheus go get -u # Update your dependencies GOARCH=arm go build -o prometheus.arm

Since cross-compilation by default disables CGO, this builds a statically linked prometheus binary ready to be run on a Raspberry Pi. If you’re running a newer Pi, you can set GOARM to the version of your ARM processor. See this for supported architectures.

Performance

I didn’t have time to benchmark it yet, but it seems to perform much better than I expected. Right now, it only scrapes itself, giving us ~250 time series. After running it for a few hours it already collected 137806 samples. Graphing a simple time series like process_goroutines for the past hour takes between 150-170ms. The probably most expensive operation you can do (and definitely shouldn’t do on a production system) is graphing all time series by executing {job=~".*"} this returns in about 30s on the Raspberry Pi.

Downloads

Update: There are now official images available: https://prometheus.io/download/

Checksums