Image from epicurious.com

This paper is a continuous exploration of enabling HTTPS for the app without https implemented. (The first paper can be reached here.) Here we will enable client certificate authentication for a non-https app using the sidecar pattern.

When client certificate authentication is turned on, the client HTTPS connection must submit with a valid cert that signed by the CA. Otherwise, the connection will be rejected.

In the last part of the paper, we examine the Prometheus in IBM Cloud Private which is using the same https sidecar pattern.

Steps to setup client certificate authentication

First, we enable the client certificate authentication by adding the following lines in the nginx.conf file as what we have in the first paper.

...

server {

listen 443 ssl;

server_name localhost; ssl_certificate /app/cert/hello-server.pem;

ssl_certificate_key /app/cert/hello-server-key.pem;



ssl_client_certificate /app/cert/hello-server-ca.pem;

ssl_verify_client on; ssl_protocols TLSv1.2;

...

When ssl_verify_client is set on, the ssl_client_certificate need to be set as the CA cert that is used to sign the server and client cert. If a client doesn’t use a cert signed by this CA, the https connection will be rejected.

Secondly, we create the K8s secret with all the certs required,

kubectl create secret generic hello-sidecar-nginx-certs --from-file=hello-server-cert=./hello-server.pem --from-file=hello-server-key=./hello-server-key.pem --from-file=hello-server-ca-cert=./myca.pem

Then update the K8s deployment file, to mount the CA to the Nginx container.

...

spec:

containers:

- name: hello

image: zhiminwen/hello:v1

imagePullPolicy: IfNotPresent

env:

- name: LISTENING_PORT

value: "8080"

- name: tls-sidecar

image: nginx

imagePullPolicy: IfNotPresent

volumeMounts:

- name: secret-volume

mountPath: /app/cert

- name: config-volume

mountPath: /etc/nginx/nginx.conf

subPath: nginx.conf

volumes:

- name: secret-volume

secret:

secretName: hello-sidecar-nginx-certs

items:

- key: hello-server-cert

path: hello-server.pem

- key: hello-server-key

path: hello-server-key.pem

- key: hello-server-ca-cert

path: hello-server-ca.pem

- name: config-volume

configMap:

name: hello-sidecar-nginx-conf

Apply the updated yaml file. Now the application is HTTPS client certificate authentication enabled.

Positive and Negative Test

Test without the valid cert.



<html>

<head><title>400 No required SSL certificate was sent</title></head>

<body>

<center><h1>400 Bad Request</h1></center>

<center>No required SSL certificate was sent</center>

<hr><center>nginx/1.15.7</center>

</body>

</html> curl -k https://192.168.64.244:31463/date 400 No required SSL certificate was sent 400 Bad Request No required SSL certificate was sent nginx/1.15.7

To test with the valid cert, let’s generate the cert signed by the CA first.

Create a json file as below, save it as “clientRequest.json”

{

"CN": "client-for-hello-server",

"hosts": [

""

],

"key": {

"algo": "rsa",

"size": 2048

}

}

Generate the client cert with the client profile (as defined in the first paper).

cd certs cfssl gencert -ca=myca.pem -ca-key=myca-key.pem -config=ca-config.json -profile=client -hostname="127.0.0.1" clientRequest.json | cfssljson -bare hello-client

You will have

hello-client.pem, the public key

hello-client-key.pem, the private key

Test with these keys,



time now: 03:36:36 curl -k --cert certs\hello-client.pem --key certs\hello-client-key.pem https://192.168.64.244:31463/date time now: 03:36:36

At this point, we implemented the HTTPS with client cert authentication for the non-https application using the K8s sidecar pattern.

ICP Prometheus

By default, Prometheus doesn’t provide any HTTP/TLS capability. IBM Cloud Private, ICP, uses the sidecar technique to enable the HTTPS/TLS and client certificate authentication.

Attached is part of the result of kubectl -n kube-system describe pods monitoring-prometheus-74c6d846d7-plb2n

The sidecar container, router, in the pods

router:

Container ID: docker://04b9191d822ddb7f14e063e264afaeb2299d6d846777d230b17ec7404f92bade

Image: devcluster.icp:8500/ibmcom/icp-management-ingress:2.2.2

Image ID: docker-pullable://devcluster.icp:8500/ibmcom/icp-management-ingress@sha256:c6e8be6e465e69b0d9e045a78a42ec22ef9e79d1886dfce691b9e7f1e9738d6a

Port: 8080/TCP

Host Port: 0/TCP

Command:

/opt/ibm/router/entry/entrypoint.sh

State: Running

Started: Wed, 28 Nov 2018 15:16:22 +0800

Ready: True

Restart Count: 0

Environment: <none>

Mounts:

/opt/ibm/router/caCerts from monitoring-ca-certs (rw)

/opt/ibm/router/certs from monitoring-certs (rw)

/opt/ibm/router/conf from router-config (rw)

/opt/ibm/router/entry from router-entry (rw)

/opt/ibm/router/lua-scripts from lua-scripts-config-config (rw)

/var/run/secrets/kubernetes.io/serviceaccount from default-token-tvb2h (ro)

The nginx.conf can be found by running kubectl -n kube-system describe cm monitoring-prometheus-router-nginx-config

The server block excerpt is listed as below,

server {

listen 8443 ssl default_server;

ssl_certificate server.crt;

ssl_certificate_key server.key;

ssl_client_certificate /opt/ibm/router/caCerts/tls.crt;

ssl_verify_client on;

ssl_protocols TLSv1.2;

# Modulo ChaCha20 cipher.

ssl_ciphers EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:!EECDH+3DES:!RSA+3DES:!MD5;

ssl_prefer_server_ciphers on; ...

Because of the ssl_verify_client setting, the client which needs to contact to Prometheus must, therefore, use the certificate signed by the CA cert.

The location block, which redirects the traffic to the Prometheus, is shown below,

location /federate {

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;

} proxy_pass http://127.0.0.1:9090/prometheus/federate location /api/v1/series {

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;

content_by_lua 'rewrite.write_release_response()';

}

rewrite_by_lua 'rewrite.rewrite_query()';

proxy_pass if ($arg_match[] = "helm_release_info") {content_by_lua 'rewrite.write_release_response()';rewrite_by_lua 'rewrite.rewrite_query()';proxy_pass http://127.0.0.1:9090/prometheus/api/v1/series } location / {

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Host $http_host;rewrite_by_lua 'rewrite.rewrite_query()';

} proxy_pass http://127.0.0.1:9090/prometheus/

The API call to Prometheus will be redirected to the Prometheus container in the same pod.

Conclusion

These two papers explore the sidecar pattern for HTTPS and client certificate authentication. With the knowledge, it is useful to understand the ICP Prometheus and to further extend Prometheus functionality.