Part 4: Start scaling

Prometheus-Adapter is an API extension for k8s that uses user-defined Prometheus queries to populate k8s resource & custom metrics APIs.

We’re going to install it into our cluster and add a rule that tracks the rate of requests, per pod. The rules are defined as YAML that we’re going to add to the Prometheus Adapter’s Helm chart.

We need to define rules that populate the custom.metrics.k8s.io group.

Here’s a quote from the rules config docs:

Each rule can be broken down into roughly four parts: Discovery, which specifies how the adapter should find all Prometheus metrics for this rule Association, which specifies how the adapter should determine which Kubernetes resources a particular metric is associated with. Naming, which specifies how the adapter should expose the metric in the custom metrics API. Querying, which specifies how a request for a particular metric on one or more Kubernetes objects should be turned into a query to Prometheus.

Let’s start off by adding the rules key to our config file:

rules: []

We only need to discover one metric: demo_app_button_clicks_total. We add a not null match on the pod label to ensure it’s associated with one.

rules:

- seriesQuery: 'demo_app_button_clicks_total{pod!=""}'

Association is quite simple: Prometheus labels all directly map to Kubernetes resources (job, namespace, pod etc), so we can use a template to map resources:

rules:

- seriesQuery: 'demo_app_button_clicks_total{pod!=""}'

resources: {template: "<<.Resource>>"}

The total sum of demo button clicks is not useful: we need to make it scale per pod. This is as easy as adding a simple naming definition to our rule:

rules:

- seriesQuery: 'demo_app_button_clicks_total{pod!=""}'

resources: {template: "<<.Resource>>"}

name:

matches: "^(.*)_total"

as: "${1}_per_second"

This only leaves us to define the query:

seriesQuery: 'demo_app_button_clicks_total{pod!=""}'

resources: { template: "<<.Resource>>" }

name:

matches: "^(.*)_total"

as: "${1}_per_second"

metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)"

This is a paramaterised Prometheus query. Let’s break down what it does.

Series is any series matched by seriesQuery. There is only one series for this rule, so we could replace <<.Series>> with demo_app_button_clicks_total. <<.LabelMatchers>> is where the adapter will place label queries (e.g. “pod=’deployment-name-abcd-1234'”. <<.GroupBy>> represents what resource you want to scale on: just pods in our case.

One shortcoming of Prometheus-Adapter is that configuration is via a static file deployed with the application, rather than a ConfigMap or some other API-based object.

We’re going to use another Helm Chart to deploy this, and write our rule using a helm values file.

Our helm values file

helm install stable/prometheus-adapter -n prom-adapter -f helm-values/prometheus-adapter-values.yml

Wait a few minutes, then run this query:

jq is available from all reputable package managers.

kubectl get --raw="/apis/custom.metrics.k8s.io/v1beta1" | jq

Your output should look like this:

{

"kind": "APIResourceList",

"apiVersion": "v1",

"groupVersion": "custom.metrics.k8s.io/v1beta1",

"resources": [

{

"name": "pods/demo_app_button_clicks_per_second",

"singularName": "",

"namespaced": true,

"kind": "MetricValueList",

"verbs": [

"get"

]

},

{

"name": "services/demo_app_button_clicks_per_second",

"singularName": "",

"namespaced": true,

"kind": "MetricValueList",

"verbs": [

"get"

]

},

{

"name": "jobs.batch/demo_app_button_clicks_per_second",

"singularName": "",

"namespaced": true,

"kind": "MetricValueList",

"verbs": [

"get"

]

},

{

"name": "namespaces/demo_app_button_clicks_per_second",

"singularName": "",

"namespaced": false,

"kind": "MetricValueList",

"verbs": [

"get"

]

}

]

}

Next, let’s get the metric for our pod:

kubectl get --raw="/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/demo_app_button_clicks_per_second?pod=$(kubectl get po -l app=prometheus-demo-app -o name)" | jq

The output should look similar to this:

{

"kind": "MetricValueList",

"apiVersion": "custom.metrics.k8s.io/v1beta1",

"metadata": {

"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/demo_app_button_clicks_per_second"

},

"items": [

{

"describedObject": {

"kind": "Pod",

"namespace": "default",

"name": "prometheus-demo-app-86cc784f46-c5m66",

"apiVersion": "/v1"

},

"metricName": "demo_app_button_clicks_per_second",

"timestamp": "2019-09-20T18:06:55Z",

"value": "0"

}

]

}

Note the zero for value: as we have not started clicking the button, the normalised rate is 0 per second.

Navigate to the webpage of the demo app and hammer the button, then re-run the above command. You should see the rate increase.

Tip: if you don’t have ingress set up, run this command to get the web-page port-forwarded to your machine: