In mid-August 2016, fromAtoB switched from running on a few hand-managed bare-metal servers to Google Cloud Platform (GCP), using saltstack , packer , and terraform to programmatically define and manage our infrastructure. After this migration, it was relatively straightforward to setup and expose our internal services such as kibana , grafana , and prometheus to the internet at large with a small set of salt states that managed oauth2_proxy , nginx , and lego on individual machines running the services managed by systemd.

Then, mid-September, we migrated our Ruby on Rails code to run within Kubernetes on Google Container Engine (GKE). This was a big change for us on almost all parts of the stack, as deployments no longer worked with Capistrano , but via kubectl , and our CI setup had to change dramatically to allow for docker images to be built and pushed during CI. However, everything went rather smoothly in the end.

Next, we wanted to migrate our internal services to run within Kubernetes, too - but we did not have an easy solution to managing Ingress. The Rails application ran as a NodePort service connected to a terraform managed GCP HTTP Load Balancer, which had all of our main site SSL certificates. Marrying this setup to Let’s Encrypt would not have been very easy, as the GCP HTTP Load Balancer does not support TLS-SNI (at the time of this article).

TLS-SNI and Google Cloud Platform Woes

On GCP, the HTTP load balancers do not support TLS-SNI, which means you need a new frontend IP address per SSL certificate you have. For internal services, this is a pain, as you cannot point a wildcard DNS entry to a single IP, like *.fromatob.com and then have everything just work. Luckily, we realized that using a TCP load balancer with the Nginx IngressController would work just as well, and support TLS-SNI no problem.

Setting this up was straightforward, by creating a Kubernetes DaemonSet that runs the IngressController on every node, and then pointing a TCP Load Balancer+HealthCheck to each GKE instance we run.

apiVersion : extensions/v1beta1 kind : DaemonSet metadata : name : nginx namespace : nginx-ingress spec : template : metadata : labels : app : nginx spec : terminationGracePeriodSeconds : 60 containers : - image : gcr.io/google_containers/nginx-ingress-controller: 0.8.3 name : nginx imagePullPolicy : Always env : - name : POD_NAME valueFrom : fieldRef : fieldPath : metadata.name - name : POD_NAMESPACE valueFrom : fieldRef : fieldPath : metadata.namespace readinessProbe : httpGet : path : /healthz port : 10254 scheme : HTTP livenessProbe : httpGet : path : /healthz port : 10254 scheme : HTTP initialDelaySeconds : 30 timeoutSeconds : 5 ports : - containerPort : 80 hostPort : 80 - containerPort : 443 hostPort : 443 args : - /nginx-ingress-controller - --default-backend-service=nginx-ingress/default-http-backend - --nginx-configmap=nginx-ingress/nginx-ingress-controller

Adding Oauth 2 Authentication

It’s relatively important to expose your internal dashboards and services to the outside world with authentication, and oauth2 proxy makes this super simple. We like to run it inside the same Pod that manages our service deployment - for Kibana this means our deployment looks like

apiVersion : extensions/v1beta1 kind : Deployment metadata : name : kibana namespace : default spec : replicas : 1 revisionHistoryLimit : 2 template : metadata : labels : app : kibana spec : containers : - image : kibana: 5.0.1 imagePullPolicy : Always name : kibana env : - name : ELASTICSEARCH_URL value : "http://elasticsearch:9200" resources : limits : cpu : 200m memory : 200Mi requests : cpu : 50m memory : 100Mi ports : - containerPort : 5601 - name : oauth2-proxy image : a5huynh/oauth2_proxy args : - "-upstream=http://localhost:5601/" - "-provider=github" - "-cookie-secure=true" - "-cookie-expire=168h0m" - "-cookie-refresh=60m" - "-cookie-secret=SECRET COOKIE" - "-cookie-domain=kibana.fromatob.com" - "-http-address=0.0.0.0:4180" - "-redirect-url=https://kibana.fromatob.com/oauth2/callback" - "-github-org=fromAtoB" - "-email-domain=*" - "-client-id=github oauth ID" - "-client-secret=github oauth secret" ports : - containerPort : 4180

and the service for the deployment is just as straightforward -

apiVersion : v1 kind : Service metadata : name : kibana namespace : default spec : ports : - port : 80 targetPort : 4180 protocol : TCP selector : app : kibana

Adding Let’s Encrypt

Fortunately for us, integrating Let’s Encrypt with Kubernetes via the Nginx Ingress Controller is easy, thanks to the fantastic kube-lego which automatically provisions SSL certificates for Kubernetes Ingress Resources with the addition of a few simple annotations.

After setting up the appropriate service and deployment for Kibana, simply creating an Ingress resource results in the Nginx being set up and a Let’s Encrypt certificate provisioned for the domain.

apiVersion : extensions/v1beta1 kind : Ingress metadata : name : kibana namespace : default annotations : kubernetes.io/tls-acme : "true" kubernetes.io/ingress.class : "nginx" spec : tls : - hosts : - kibana.fromatob.com secretName : kibana-tls rules : - host : kibana.fromatob.com http : paths : - path : / backend : serviceName : kibana servicePort : 80

Conclusion

With the right setup, it’s super easy to expose protected, HTTPS resources from Kubernetes, if you just want to copy our setup, the manifests we use in production are available on GitHub .