Thanks to the work of Joel Takvorian and others in the Kiali team, Kiali is now able to show custom telemetry dashboards for application metrics. Those dashboards are available for Applications and Workloads.

Screenshot of a workload with custom metrics (“Quarkus metrics”)

This feature is available in Kiali version 0.15. To use it, you need a runtime that is able to expose telemetry in the Prometheus text format. Before we get started, let’s have a look at how this all works.

The following diagram shows the deployed applications in yellow, the Prometheus database in green and Kiali related resources in turquoise. The lines between the items are described below the diagram.

Architecture of the dashboarding feature

When installing Kiali, a new Custom Resource Definition (CRD), monitoringdashboard.monitoring.kiali.io, is installed in the system. Along with this a number of dashboard-definition custom resources are also deployed for popular runtimes like Quarkus, Thorntail, Vert.X or Node.js(*1). Those definitions are shown in the white-turquoise boxes.

It is also easily possible to create such custom resources for your own project — we’ll come to that later. Kiali is using those custom resources as templates to populate the actual dashboard tabs in the UI (magenta lines).

In the next step you need to add additional information to the Kubernetes Deployment object of the instrumented application:

apiVersion: extensions/v1beta1

kind: Deployment

[...]

spec:

template:

metadata:

annotations:

kiali.io/runtimes: quarkus,my-app

prometheus.io/port: "9080"

prometheus.io/scheme: http

prometheus.io/scrape: "true"

The kiali.io/runtimes annotation(*2) defines the dashboards to be shown in the Kiali console, represented by the brown lines in the diagram.

The prometheus.io/* ones instruct Prometheus that data is to be fetched (scrape), by what means (scheme) and on what port (port). If you don’t specify a path, Prometheus will scrape from the /metrics endpoint on your application.

Prometheus documentation holds the full list of options.

Scraping is indicated in the diagram by the solid green lines.

Now let’s have a look on how to write a custom dashboard definition. For this I have defined a counter in my application and am exposing it via MicroProfile Metrics.

$ curl -HAccept:text/plain localhost:8080/metrics/application

# TYPE application:ratings_endpoint_get_rating counter

application:ratings_endpoint_get_rating 2.0

With this information we can now create our custom resource. Create a new file myApp.yaml and add the following content (or better: what matches your app):

apiVersion: "monitoring.kiali.io/v1alpha1"

kind: MonitoringDashboard

metadata:

name: my-app

spec:

title: App Rating

charts:

- name: "Ratings calls"

unit: ""

spans: 6

metricName: "application:ratings_endpoint_get_rating"

dataType: "raw"

- name: "Ratings rate"

unit: ""

spans: 6

metricName: "application:ratings_endpoint_get_rating"

dataType: "rate"

To make this dashboard definition available to all namespaces, deploy it via kubectl apply -f myApp.yaml -n istio-system. If you deploy it to a namespace other than istio-system, it will only be available for deployments in that namespace.

After the next screen refresh you will now see two tabs for the two dashboards:

Our new dashboard ‘App Rating’ is now visible

The above definition has the name of the dashboard, as it appears on the deployment in metadata.name. Spec.title shows the title of the tab and then below the individual charts are shown. Span determines the width of a chart (the full width would be 12 spans as Bootstrap is using), name and unit are free-text. Obviously ‘metricName’ provides the name of the metric. In our example I am using it once. In the first case it shows the raw data (i.e. how often that counter has been bumped). The second case shows the rate of change (i.e. the one divided by time).

The source file kiali_monitoring_types.go shows the other available options, but it is probably best to have a look at the provided example dashboard definitions.

Traffic from unknown?

If you run the above in a plain Istio environment, it is possible that as soon as you mark your application for scraping, you will all of a sudden see traffic coming from unknown into them. This is because the default Istio rules are not filtering it out. I have updated my article on that subject which also shows how to re-configure a rule in the Istio configuration.

As an alternative you can also use the new ‘hide’ feature in Kiali 0.15 to hide traffic from unknown in the UI.

Conclusion

The new dashboard feature lets you see metrics of the actual applications that matter to you. You can easily view this data in Kiali, but since the feature makes the data available in Prometheus, it is also possible to view the data using the Prometheus UI or other tools like Grafana.

As the feature does not depend on Istio at all (it only uses the Prometheus that comes with it), I can imagine that it could be easily put into a separate module to be pulled in by other tools.

Footnotes

*)1 In 0.15 you have to install them by hand, while in 0.16 onwards the installer will take care. To install the definitions go to https://github.com/kiali/kiali/tree/v0.15.0/deploy/dashboards and install the files via kubectl apply -f *

*2) In the version I am using, there must be no whitespace after the comma to separate more than one definition. This requirement will be relaxed in the future.