Overview

Having relevant information about the state of the infrastructure and other systems in a microservices architecture is crucial for ensuring stability, thus keeping the end-user satisfied. Metrics concerning health and performance of the deployments let the team react faster while having a negligible effect on the system.

Systems applying the microservice architecture pattern are harder to monitor, as Kubernetes pods are mortal, therefore a fixed scraping config is off the mark. That's where the service discovery comes in.

This tutorial walks you through building a microservice monitoring solution from scratch with dynamic service discovery inside a Kubernetes cluster using Prometheus and Grafana.

Most monitoring platforms utilize time-series databases, which are intended for storing time-based information, such as application metrics. Some products only manage the database part (e.g. InfluxDB), while Prometheus offers visualization alongside the metrics management. However, it's common to extend its visualization capabilities with Grafana.

So without further ado, let's get started!

Prerequisite

The tutorial assumes that you have a Kubernetes cluster set up and ready-to-go.

Also, it's handy to have a Spring Boot microservice laying around to monitor.

Publishing metrics from Spring Boot applications

Spring Boot's Actuator provides a production-ready solution for shipping sensible default application metrics without any hassle.

Dependencies

Our first step should be adding the required dependencies.

<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency>

Configuring Actuator

Actuator exposes some additional information out of the box. To disable these endpoints and include those which are needed in our monitoring setup, add this to your application.yml :

management: endpoints: enabled-by-default: false web: base-path: /status exposure: include: prometheus, health

In a production environment, it is advised to disable these endpoints, as they might expose details about the system's underlying implementation via various HTTP endpoints. Remapping the base path is optional.

(Optional) Publish custom application name

Some Grafana dashboards provide grouping capabilities if you publish the application name alongside the default metrics.

import io.micrometer.core.instrument.MeterRegistry; import org.springframework.beans.factory.annotation.Value; import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class MeterRegistryCustomizerConfiguration { @Value("${spring.application.name}") private String applicationName; @Bean MeterRegistryCustomizer<MeterRegistry> metricsCommonTags(){ return registry -> registry.config().commonTags("application", applicationName); } }

Your application now exposes two more additional endpoints: /status/health and /status/prometheus to let you monitor your application.

Collecting the information

Prometheus is an open-source system for monitoring and alerting. We'll use it to collect application metrics.

Prometheus configuration

Prometheus is designed to be configured via command-line flags and a configuration file. Providing this configuration file as a ConfigMap is a simple but reliable solution for setting up Prometheus according to our needs. The previously mentioned service discovery's essence comes from relabeling the existing pod scrape endpoints to the ones exposed by Spring Boot Actuator. This is done by simple regex expressions.

kind: ConfigMap apiVersion: v1 metadata: name: prometheus-config namespace: my-namespace labels: k8s-app: metrics-service data: prometheus.yml: |- scrape_configs: - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name

Note the following annotations: prometheus.io/scrape , prometheus.io/path , prometheus.io/port . We'll wire them in later.

Setting up a service account for Prometheus

The service discovery relies on metrics provided by Kubernetes. To access these, we'll need a service account.

apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: prometheus-kube rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - extensions resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: prometheus-kube roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus-kube subjects: - kind: ServiceAccount name: default namespace: my-namespace

An example Prometheus deployment

Prometheus should be accessible inside the cluster in order to use it as Grafana's datasource. There is no need to expose it outside the cluster though, therefore a ClusterIP is enough.

kind: Service apiVersion: v1 metadata: name: prometheus-service namespace: my-namespace labels: k8s-app: prometheus-service spec: ports: - protocol: TCP port: 9090 targetPort: 9090 selector: k8s-app: prometheus-service type: ClusterIP --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: prometheus-service namespace: my-namespace labels: k8s-app: prometheus-service spec: replicas: 1 selector: matchLabels: k8s-app: prometheus-service template: metadata: name: prometheus-service labels: k8s-app: prometheus-service spec: volumes: - name: prometheus-config-volume configMap: name: prometheus-config containers: - name: prometheus image: prom/prometheus imagePullPolicy: Always ports: - containerPort: 9090 volumeMounts: - name: prometheus-config-volume mountPath: /etc/prometheus args: ["--config.file=/etc/prometheus/prometheus.yml"] restartPolicy: Always dnsPolicy: ClusterFirst

It is advised to set a data retention policy, with the --storage.tsdb.retention=32d container flag.

Updating your deployments

The expressions showed previously utilize the annotations defined in your Kubernetes deployments. The regex based relabeling allows the actual pod scrape endpoint to be configured via the following annotations:

prometheus.io/scrape : Set the value to true to enable scraping of pods belonging to the deployment prometheus.io/path : If the metrics path is not /metrics override this. In our example, it's /status/prometheus . prometheus.io/port : Scrape the pod on the indicated port instead of the default of 9102 .

spec: template: metadata: annotations: prometheus.io/path: '/status/prometheus' prometheus.io/port: '8080' prometheus.io/scrape: 'true'

Adding Grafana to the cluster

To be able to access your Grafana dashboard outside the cluster, you should assign a NodePort to it.

kind: Service apiVersion: v1 metadata: name: grafana-service namespace: my-namespace labels: k8s-app: grafana-service spec: ports: - protocol: TCP port: 3000 targetPort: 3000 nodePort: 3000 selector: k8s-app: grafana-service type: NodePort --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: grafana-service namespace: my-namespace labels: k8s-app: grafana-service spec: replicas: 1 selector: matchLabels: k8s-app: grafana-service template: metadata: name: grafana-service labels: k8s-app: grafana-service spec: volumes: containers: - name: grafana image: grafana/grafana imagePullPolicy: Always ports: - containerPort: 3000 env: - name: GF_SECURITY_ADMIN_PASSWORD value: D4vidH4ss3lh0ff1sC00L - name: GF_SECURITY_ADMIN_USER value: admin - name: GF_USERS_ALLOW_SIGN_UP value: "false" restartPolicy: Always dnsPolicy: ClusterFirst

Connecting Grafana to Prometheus

After you've applied the previous deployments, Grafana should be accessible at http://your-node.example.com:3000/login with the credentials we defined previously.

The next step is to configure Grafana to use Prometheus as a datasource. In the configuration wizard add the following datasource:

Picking a Grafana dashboard from the store

Grafana has a store, where premade dashboards are available. We recommend this one for Spring Boot applications.

Conclusion

In this tutorial, we explored how to scrape service discovery based application metrics from Spring Boot using Prometheus and Grafana inside a Kubernetes cluster. Congratulations! You have just set up a microservice monitoring infrastructure!