Not all applications produce useful metrics, but some of them do produce logs.

There are a class of applications which have little to no useful metrics, however the information you'd like to have can be found in logs. One option to deal with this is the grok exporter. Let's say you were working with Apache, here are some example logs from a typical access.log :

x.x.x.x - - [20/Jan/2020:06:25:24 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1" x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16061 "-" "Go-http-client/1.1" x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET / HTTP/1.1" 200 16064 "-" "Go-http-client/1.1" x.x.x.x - - [20/Jan/2020:06:25:25 +0000] "GET /blog/rss HTTP/1.1" 301 3478 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)" x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET / HTTP/1.1" 200 16065 "-" "Go-http-client/1.1" x.x.x.x - - [20/Jan/2020:06:25:26 +0000] "GET /blog/feed HTTP/1.1" 200 3413 "-" "Tiny Tiny RSS/19.2 (adc2a51) (http://tt-rss.org/)" x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET /feed HTTP/1.1" 200 6496 "-" "Emacs Elfeed 3.2.0" x.x.x.x - - [20/Jan/2020:06:25:27 +0000] "GET / HTTP/1.1" 200 62316 "http://178.62.121.216" "Go-http-client/1.1"

To start off download, configure, and run the grok exporter:

wget https://github.com/fstab/grok_exporter/releases/download/v1.0.0.RC2/grok_exporter-1.0.0.RC2.linux-amd64.zip unzip grok_exporter-*.zip cd grok_exporter*amd64 cat << 'EOF' > config.yml global: config_version: 2 input: type: file path: access.log readall: true grok: patterns_dir: ./patterns metrics: - type: counter name: apache_http_response_codes_total help: HTTP requests to Apache match: '%{COMBINEDAPACHELOG}' labels: method: '{{.verb}}' path: '{{.request}}' code: '{{.response}}' server: port: 9144 EOF ./grok_exporter -config config.yml

If you visit http://localhost:9144/metrics you'll see the metric:

# HELP apache_http_response_codes_total HTTP requests to Apache # TYPE apache_http_response_codes_total counter apache_http_response_codes_total{code="200",method="GET",path="/"} 5 apache_http_response_codes_total{code="200",method="GET",path="/blog/feed"} 1 apache_http_response_codes_total{code="200",method="GET",path="/feed"} 1 apache_http_response_codes_total{code="301",method="GET",path="/blog/rss"} 1

For the sake of demonstration readall has been enabled, which will have the exporter read the log files from the start rather than the usual desired behaviour of only processing newly added log lines.

To back up a bit, Grok is a way that you can parse log lings in Logstash (Logstash is the L in the ELK stack). Accordingly patterns have been written for many common applications, including Apache. The Grok exporter allows you to build on the existing corpus of patterns, plus add your own, to extract metrics from logs. COMMMONAPACHELOG from above for example is defined as

COMMONAPACHELOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

which itself relies on many other patterns. So you should be glad you don't have to write that yourself from scratch. The label fields provide you with Go templating (as seen in Prometheus alerting and notification templating) to access these values.

Beyond counting log lines, the Grok exporter can allow you to observe event sizes such as how many bytes are in responses:

- type : summary name : apache_http_response_bytes help : Size of HTTP responses match : '%{COMMONAPACHELOG} ' value : ' {{.bytes}} '

Or gauges such as when the last request was:

- type : gauge name : apache_http_last_request_seconds help : Timestamp of the last HTTP request match : '%{COMMONAPACHELOG} ' value: '{{timestamp "02/Jan/2006:15:04:05 -0700" .timestamp}}'

This is using the grok exporter's timestamp function, which is basically Go's time.Parse. There's also a divide function, in case you need to convert milliseconds to seconds.

This is a small taste of what you can do with the Grok exporter. As you can see parsing logs does require some configuration, so you should only use an exporter like this if you've no way to get the application itself to produce good metrics.

Want to get metrics into Prometheus? Contact us.