Measure all the things!

In modern IT systems, you have to measure everything. Being an engineer you have to know several system metrics starting from very low level like CPU usage of the host, network utilization or memory usage, up to processing time or database latency. And sooner or later the business will start to ask questions: how many users we have? How many times the product’s page was displayed?

Setting up a reliable metrics system is probably a topic for another article. In short — at SoftwareMill we often use Prometheus/Grafana/Alertmanager stack. In this article, I’d like to focus on only one thing — managing the Grafana dashboards.

Custom dashboards

There are a lot of ready to use, powerful and shiny official and community maintained dashboards on Grafana site. But it’s not uncommon you need some changes in those dashboards. You’ll also need custom dashboards to visualize business metrics specific to your applications. There are several problems with this:

Backup and restore. Creating a complex dashboard can be a very time-consuming task, and you definitely don’t want to lose your work! If you have multiple environments (like staging and production) which are completely separated (definitely a good practice!) you’ll probably want to have the same dashboards on each environment. Keeping multiple dashboards in multiple environments in sync can be painful. It’s a good idea to store dashboards in a system which can keep track of changes and store multiple versions.

Grafana Dashboards are stored internally in the database — SQLite, MySQL or PostgreSQL. We tried to synchronize databases between multiple Grafana instances, but we quickly found it very inconvenient.

It’s possible to export and import dashboard using Grafana API, but using it manually or creating a custom tool for this was out of the question.

The right tool for the job

Here enters — on the white horse — a great tool which solved our problems: wizzy.

Generally, the idea is quite simple: define connections to one or multiple Grafana instances on local machine and import/export data sources definitions and dashboards using the Grafana API. Additionally — all data is stored in text files (JSON) so it can be easily integrated with version control system like Git.

According to the wizzy documentation there are several use cases:

Copy, move Grafana dashboards from one installation of Grafana to another. For example, from your dev installtion to prod. Store Grafana dashboards, rows, panels, datasources, orgs and even template variables, etc. in Git. Copy, move, remove, edit rows and panels either within a dashboard or between two dashboards. Upload/Store/Download Grafana dashboards from and to AWS S3 respectively. Search/Download Grafana.net community dashboards in a single command. Generate 8 second GIFs for any Grafana dashboard with last 24 hours of data for your presentations.

Using wizzy

Installation

To install and use wizzy you need nodejs and npm installed:

node -v

npm -v

If you have npm — installing wizzy is simple:

npm install -g wizzy

Initialization

Create a directory and initialize it as a wizzy “database”:

mkdir grafana

cd grafana

wizzy init

✔ conf directory created.

✔ conf file created.

✔ wizzy successfully initialized.

It will create conf directory with wizzy.json file inside it.

If you want to store your data in git you have to initialise the git repository:

git init

and set remote url:

git remote add origin git@ github.com:softwaremill/devops-grafana.git

To check the status of current directory:

wizzy status

✔ conf directory exists.

✔ conf file exists.

✔ .git directory exists.

✔ wizzy setup complete.

Configuration

Wizzy has a concept of contexts — you can set current context to particular Grafana (environment). To add multiple environments:



wizzy set grafana envs staging url \

roject.com

✔ grafana:envs:staging:url updated successfully.

✔ conf file saved. wizzy add grafana stagingwizzy set grafana envs staging url \ https://grafana.staging.p roject.com✔ grafana:envs:staging:url updated successfully.✔ conf file saved. wizzy set grafana envs staging username admin

✔ grafana:envs:staging:username updated successfully.

✔ conf file saved. wizzy set grafana envs staging password 5ecurePassw0rd

✔ grafana:envs:staging:password updated successfully.

✔ conf file saved.

And the same for the production environment:



wizzy set grafana envs prod url

✔ grafana:envs:prod:url updated successfully.

✔ conf file saved. wizzy add grafana prodwizzy set grafana envs prod url https://grafana.prod.p roject.com✔ grafana:envs:prod:url updated successfully.✔ conf file saved. wizzy set grafana envs prod username admin

✔ grafana:envs:prod:username updated successfully.

✔ conf file saved. wizzy set grafana envs prod password m0re5ecurePazzw0rd

✔ grafana:envs:prod:password updated successfully.

✔ conf file saved.

Operations

If you want to switch to staging:

wizzy set context grafana staging

To list all dashboard on staging:

wizzy list dashboards

It will output a nice table with two columns: dashboard name and slug — kind of a dashboard id.

To import all dashboards from staging to local database:

wizzy import dashboard

Importing 9 dashboards:

✔ Dashboard jmx-overview imported successfully.

✔ Dashboard jvm-jmx-overview imported successfully.

✔ Dashboard kafka-consumer-metrics imported successfully.

✔ Dashboard kafka-producer-metrics imported successfully.

✔ Dashboard kafka-server-metrics imported successfully.

✔ Dashboard kafka-topic-metrics imported successfully.

✔ Dashboard mysql-overview imported successfully.

✔ Dashboard node-exporter-full imported successfully.

✔ Dashboard prometheus-redis imported successfully.

✔ 9 dashboards imported successfully.

If you want to import a particular dashboard:

wizzy import dashboard <dashboard-slug>

Dashboard are stored in dashboards directory.

How to export dashboard to production environment?

You probably got it already, it’s so easy as:

wizzy set context grafana prod

wizzy export dashboards

And voila! All dashboards from staging are already on production. Pretty easy, isn’t it? :)

Securing Grafana credentials

If you check the content of the conf/wizzy.json file you’ll find credentials to Grafana instances stored in plain text. As you remember — we want to use git to store the Grafana configuration, but storing also the credentials can be a security risk. We decided to secure this data using git-crypt.

Installation of git crypt is out of the scope — you’ll find a good tutorials using your favourite search engine ;).

You have to initialise git-crypt in current directory:

git-crypt init

(you’ve already got the pattern, haven’t you? ;) )

Create a .gitattributes file — the syntax is similar to .gitignore :

conf/wizzy.json filter=git-crypt diff=git-crypt

You can check a status of encryption:

git-crypt status

not encrypted: .gitattributes

encrypted: conf/wizzy.json

not encrypted: dashboards/jmx-overview.json

not encrypted: dashboards/jvm-jmx-overview.json

not encrypted: dashboards/kafka-consumer-metrics.json

not encrypted: dashboards/kafka-producer-metrics.json

not encrypted: dashboards/kafka-server-metrics.json

not encrypted: dashboards/kafka-topic-metrics.json

not encrypted: dashboards/mysql-overview.json

not encrypted: dashboards/node-exporter-full.json

not encrypted: dashboards/prometheus-redis.json



When a file is marked as encrypted — it will be encrypted on-the-fly if you commit and push it to remote server.

You can use git-crypt in two modes: GPG mode and symmetric mode. We found GPG mode more convenient — sharing common encryption key in secure way can be difficult.

To add a user’s public key:

git-crypt add-gpg-user USER_ID

Please note — this command will add a file in .git-crypt directory and create a commit.

When a new user wants to work with this repository you have to add his public key to repository using command described above.

Then he has to clone this repository and unlock it:

git-crypt unlock

Summary

Managing multiple Grafana instances can be challenging if you don’t use a proper tool for it. Wizzy hardened by git-crypt seems to do the job. Let’s check problems we had with custom Grafana dashboard:

Backup and restore. Creating a complex dashboard can be a very time-consuming task, and you definitely don’t want to lose your work!

Checked! Git does it very well.

If you have multiple environments (like staging and production) which are completely separated (definitely a good practice!) you’ll probably want to have the same dashboards on each environment. Keeping multiple dashboards in multiple environments in sync can be painful.

Checked! wizzy is the right tool for it.

It’s good idea to store dashboards in system which can keep track of changes and store multiple versions.

Checked! — again, using git solves this problem.

So — we solved all our problems, and we did it in a secure way!