Image quoted from Amazon.com

In the last paper, I explored how to create custom metrics of the application and present the result in the Grafana dashboard. In this paper, I will achieve the Horizontal Pod Autoscale (HPA) goal based on a selected custom metric.

The steps are summarised as below

Update and install the Prometheus-Adapter helm chart for custom metrics to be consumed by HPA. Define and create the custom metric Create the HPA object Testing with the scale out.

Let’s start.

1. Prometheus Adapter

The Prometheus Adapter extends Kubernetes by implementing the custom metrics API, which enables the HorizontalPodAutoscaler controller to retrieve metrics using the “custom.metrics.k8s.io” API. By defining our own metrics through the adapter’s configuration, we are able to let HPA scale based on our custom metrics.

My Kubernetes environment is IBM Cloud Private (ICP) 3.1.1. Although there is a Prometheus Adapter installed in this version, the configuration to create custom metrics is not exposed out. We will install the adapter based on the latest helm chart.

By doing the re-installation, some of the duplicated Kubernets object will be overwritten. You need to make sure those default pod metrics is not in use if the cluster is for other purpose other than exploration.

As explored in this paper, the Prometheus in ICP requires the Client Certification Authentication, we need to update the helm chart to allow the configuration for Client Certification Authentication.

Clone the repository and introduce new variables in the values.yaml file.



url:

port: 9090

clientAuthentication:

use: true

ca: |-

# CA

cert: |-

#cert of client

key: |-

# key for the cert prometheus:url: http://prometheus.default.svc port: 9090

Update the deployment yaml file with the extra command arguments to allow Prometheus authentication.

#...

- /adapter

- --secure-port=6443

{{- if .Values.tls.enable }}

- --tls-cert-file=/var/run/serving-cert/tls.crt

- --tls-private-key-file=/var/run/serving-cert/tls.key

{{- end }}

- --cert-dir=/tmp/cert

- --logtostderr=true

- --prometheus-url={{ .Values.prometheus.url }}:{{ .Values.prometheus.port }}

- --metrics-relist-interval={{ .Values.metricsRelistInterval }}

- --v={{ .Values.logLevel }}

- --config=/etc/adapter/config.yaml

{{- if .Values.prometheus.clientAuthentication.use }}

- --prometheus-auth-config=/etc/prometheus/prometheus.kubecfg.yaml

{{- end }} #...

It is not obvious for the user that the “prometheus-auth-config” is utilizing the kubecfg file format to talk to the Prometheus server. (There is a discussion going on to address this). Here I create a new configMap file in the chart as,

{{- if .Values.prometheus.clientAuthentication.use -}}

apiVersion: v1

kind: ConfigMap

metadata:

name: {{ template "k8s-prometheus-adapter.prometheus.kubecfg" . }}

labels:

app: {{ template "k8s-prometheus-adapter.name" . }}

chart: {{ template "k8s-prometheus-adapter.chart" . }}

release: {{ .Release.Name }}

heritage: {{ .Release.Service }}

data:

prometheus.kubecfg.yaml: |-

apiVersion: v1

kind: Config

preferences: {}

clusters:

- cluster:

# insecure-skip-tls-verify: true

certificate-authority-data: {{ b64enc .Values.prometheus.clientAuthentication.ca }}

server: {{ .Values.prometheus.url }}

name: prometheus-server

users:

- name: with-cert

user:

client-certificate-data: {{ b64enc .Values.prometheus.clientAuthentication.cert }}

client-key-data: {{ b64enc .Values.prometheus.clientAuthentication.key }}

contexts:

- context:

cluster: prometheus-server

user: with-cert

name: promethues-server-connection

current-context: promethues-server-connection

{{- end -}}

In the configMap, the cluster server points to the URL defined by prometheus.url variable. It also defines the CA cert, client cert, and client key.

The CA cert is the CA that used by Prometheus. Checking the Kubernetes object definition, it is the secret of “cluster-ca-cert”. Retrieve the CA cert and its private key,

# Save the following output as ca.pem

kubectl -n kube-system get secret cluster-ca-cert -o jsonpath="{ .data.tls\.crt }" | base64 -di # Save the following output as ca.key

kubectl -n kube-system get secret cluster-ca-cert -o jsonpath="{ .data.tls\.key }" | base64 -di

Create the client cert and key using this CA with the cfssl tools. (refer to the paper for details)

cfssl gencert -ca=ca.pem -ca-key=ca.key -config=ca-config.json -profile=client -hostname="" clientRequest.json | cfssljson -bare prometheus-client

Update the deployment file in the chart to mount this configMap as a file for Prometheus to consume.

volumeMounts:

- mountPath: /etc/adapter/

name: config

readOnly: true

- mountPath: /tmp

name: tmp

{{- if .Values.tls.enable }}

- mountPath: /var/run/serving-cert

name: volume-serving-cert

readOnly: true

{{- end }}

{{- if .Values.prometheus.clientAuthentication.use }}

- mountPath: /etc/prometheus

name: prometheus-client-authentication

readOnly: true

{{- end }} # ... {{- if .Values.prometheus.clientAuthentication.use }}

- name: prometheus-client-authentication

configMap:

name: {{ template "k8s-prometheus-adapter.prometheus.kubecfg" . }}

{{- end }}

Create a vars.yaml file as below,



url:

port: 9090

clientAuthentication:

use: true

ca: |-

-----BEGIN CERTIFICATE-----

MIIFmzCCA4OgAwIBAgIJAMVsH80dT0MRMA0GCSqGSIb3DQEBCwUAMGMxCzAJBgNV

...

-----END CERTIFICATE----- prometheus:url: https://monitoring-prometheus.kube-system port: 9090clientAuthentication:use: trueca: |------BEGIN CERTIFICATE-----MIIFmzCCA4OgAwIBAgIJAMVsH80dT0MRMA0GCSqGSIb3DQEBCwUAMGMxCzAJBgNV...-----END CERTIFICATE----- cert: |-

-----BEGIN CERTIFICATE-----

MIIEnjCCAoagAwIBAgIULumvDA5hcKj4JmGMsH8SN2CtN90wDQYJKoZIhvcNAQEN

...

-----END CERTIFICATE----- key: |-

-----BEGIN RSA PRIVATE KEY-----

MIIEpAIBAAKCAQEAqJJw+u2NHy3CEQwZgE76ZzkleBmls+nMGY5qHdCI0XEF2oLH

...

-----END RSA PRIVATE KEY----- replicas: 1

rbac:

create: true serviceAccount:

create: true

name: rules:

default: true

custom: [] tls:

enable: false nodeSelector:

management: "true"

tolerations:

- key: dedicated

value: infra

effect: "NoSchedule" logLevel: 10

The Prometheus URL is “https” and the server name is the service name of Prometheus in the cluster since the adapter pod will be running inside the same cluster.

Copy the certs in the yaml file.

I will deploy the pod on the dedicated management node and that's why nodeSelectot and tolerations are applied.

Run the helm command to install the chart. The “prometheus-adapter.chart.updated” is the directory where the updated chart resides.

helm install prometheus-adapter.chart.updated --tls --values var.yaml

Validate the pod is running. Now the custom metrics API is available. Validate it by running the command of

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq .

You should see a list of default metrics from the Adapter.

2. Create Custom Metrics

For simplicity purpose, I will remove all the default custom metric, only create one metric from the sample application in my last paper.

Prepare the following configMap file, replace the {{ .chart }} and {{ .release }} with the content from the default configMap file created by helm.

apiVersion: v1

kind: ConfigMap

metadata:

labels:

app: prometheus-adapter

chart: {{ .chart }}

heritage: Tiller

release: {{ .release}}

name: {{ .release }}-prometheus-adapter

namespace: default

data:

config.yaml: |

rules:

- seriesQuery: '{__name__= "myapp_client_connected"}'

seriesFilters: []

resources:

overrides:

k8s_namespace:

resource: namespace

k8s_pod_name:

resource: pod

name:

matches: "myapp_client_connected"

as: ""

metricsQuery: <<.Series>>{<<.LabelMatchers>>,container_name!="POD"}

Select the metric from Prometheus named as “myapp_client_connected”. Assign the namespace with the label of “k8s_namespace”, pod with the label of “k8s_pod_name” (refer to the last paper of the Prometheus relabel_config to find out how the label is defined)

Use the same metric name “myapp_client_connected” without any regex substitution. Last is the metricQuery which use the PromQL type of query to get the value of the metric.

Replace the old configMap with this new configMap. Delete the old pod to spawn a new pod to let the configuration changes be in effect. Now we can test the custom metrics API.

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq

{

"kind": "APIResourceList",

"apiVersion": "v1",

"groupVersion": "custom.metrics.k8s.io/v1beta1",

"resources": [

{

"name": "namespaces/myapp_client_connected",

"singularName": "",

"namespaced": false,

"kind": "MetricValueList",

"verbs": [

"get"

]

},

{

"name": "pods/myapp_client_connected",

"singularName": "",

"namespaced": true,

"kind": "MetricValueList",

"verbs": [

"get"

]

}

]

}

and further to list the value by calling

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/myapp_client_connected | jq

{

"kind": "MetricValueList",

"apiVersion": "custom.metrics.k8s.io/v1beta1",

"metadata": {

"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/myapp_client_connected"

},

"items": [

{

"describedObject": {

"kind": "Pod",

"namespace": "default",

"name": "hpa-sim-5cd9d7c47b-nd6c7",

"apiVersion": "/v1"

},

"metricName": "myapp_client_connected",

"timestamp": "2019-01-04T17:16:11Z",

"value": "0"

},

{

"describedObject": {

"kind": "Pod",

"namespace": "default",

"name": "hpa-sim-5cd9d7c47b-nfc9h",

"apiVersion": "/v1"

},

"metricName": "myapp_client_connected",

"timestamp": "2019-01-04T17:16:11Z",

"value": "0"

},

{

"describedObject": {

"kind": "Pod",

"namespace": "default",

"name": "hpa-sim-5cd9d7c47b-wlqqp",

"apiVersion": "/v1"

},

"metricName": "myapp_client_connected",

"timestamp": "2019-01-04T17:16:11Z",

"value": "0"

}

]

}

Notice all the three pods’ metric are returned.

3. Create the HPA for the Deployment

Recall in the last paper, I have Deployment “hpa-sim”. With the custom metric ready we can auto scale it based on the metric defined. Here I simply choose the number of active sessions. The HPA is defined as below,

apiVersion: autoscaling/v2beta1

kind: HorizontalPodAutoscaler

metadata:

name: hpa-sim

namespace: default

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

name: hpa-sim

minReplicas: 1

maxReplicas: 10

metrics:

- type: Pods

pods:

metricName: myapp_client_connected

targetAverageValue: 20

We will scale the Deployment named as “hpa-sim” from 1 to 10 replicas. The metrics will be based on the custom metrics of type Pods. The metric name is “myapp_client_connected” and the measurement is based on an average value across all relevant pods. (In fact, only average value target is supported). Set the targetAverageValue as 20. So when the average concurrent sessions of the HTTP handler are more than 20, HPA will be triggered.

Apply the config, monitor the HPA starts to collect metric value,

k get hpa hpa-sim

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE

hpa-sim Deployment/hpa-sim 0/20 1 10 1 3h

4. Pumping the Load and Monitoring the Scaling

Let's test it out with some Golang function using the magefile tasks.



var wg sync.WaitGroup

for index := 0; index < config.LoadCount; index++ {

wg.Add(1)

go func() {

url := fmt.Sprintf("

_, err := http.Get(url)

if err != nil {

log.Printf("Request Failed")

}

wg.Done()

}()

} func (Load) T01_PumpUniformLoad() {var wg sync.WaitGroupfor index := 0; index < config.LoadCount; index++ {wg.Add(1)go func() {url := fmt.Sprintf(" http://192.168.64.244:30543/service?cost=%f ", config.FixedLatency)_, err := http.Get(url)if err != nil {log.Printf("Request Failed")wg.Done()}() wg.Wait()

log.Printf("ALL request are returned.")

}

The default LoadConfig is set as 100, and FixedLatency as 120 from .env file. Run the Mage task, trigger a 100 concurrent connection lasting for 120 seconds. Open another shell, run the same task before the first 2 minutes load finishing to keep the load.

Monitor with the Grafana dashboard created in the last paper,

We can clearly see the pods are scaled up from one single pod to multiple pods and scale down back to one. Running the command kubectl get hpa hpa-sim -w , following shows how the pods grow during the peak and reduce after the peak.

Conclusion

With 4 papers, I have examined the Prometheus, Custom Metrics using client API, Custom Metrics for Grafana, and Custom Metric for HPA and getting some thorough understandings of these topics. Hopefully, you enjoy the readings and learn something together with me.