In the previous post we looked at testing rules. You can also test alerts.

Let's take a simple example of an alert with some templating:

wget https://github.com/prometheus/prometheus/releases/download/v2.5.0/prometheus-2.5.0.linux-amd64.tar.gz tar -xzf prometheus-*.tar.gz cd prometheus-* cat >rules.yml <<'EOF' groups: - name: example rules: - alert: MyAlert expr: avg without(instance)(up) < 0.75 for: 2m labels: severity: page annotations: description: 'Only {{$value}} of {{$labels.job}} job is up' EOF

To test this as before we create a test.yml , with the following content:

rule_files: - rules.yml evaluation_interval: 1m tests: - interval: 1m input_series: - series: 'up{job="node",instance="foo"}' values: '1+0x10' - series: 'up{job="node",instance="bar"}' values: '1+0x5 0+0x5' - series: 'up{job="prometheus",instance="foo"}' values: '1+0x10' alert_rule_test: - alertname: MyAlert eval_time: 7m - alertname: MyAlert eval_time: 8m exp_alerts: - exp_labels: severity: page job: node exp_annotations: description: 'Only 0.5 of node job is up'

Once again you can then run these tests with:

./promtool test rules test.yml

which should return:

Unit Testing: test.yml SUCCESS

This is the same input data as in the previous post, but here we're testing alerts instead. The first tests that no alerts are firing for MyAlert at 7 minutes in:

alert_rule_test: - alertname: MyAlert eval_time: 7m

The second tests that a single alert is firing a 8 minutes in, and that it has all the right labels and annotations:

- alertname: MyAlert eval_time: 8m exp_alerts: - exp_labels: severity: page job: node exp_annotations: description: 'Only 0.5 of node job is up'

If you expected more alerts to be firing with the same alertname, you would list all of them under exp_alerts .

And that's all there is to it. You can test rules and alerts together, and there are more advanced features like specifying what order rule groups should be evaluated in to help simulate rules across federation hierarchies.

Wondering what are good alerts to have? Contact us.