When explaining Kubernetes to people new in the space I noticed that the concept of services is often not well understood. To help you better understand what services are and how you can troubleshoot them, we will have a look at a concrete setup and discuss the inner workings of services in this post.

In a nutshell, Kubernetes services are an abstraction for pods, providing a stable, virtual IP (VIP) address. As pods may come and go, for example in the process of a rolling upgrade, services allow clients to reliably connect to the containers running in the pods, using the VIP. The virtual in VIP means it's not an actual IP address connected to a network interface but its purpose is purely to forward traffic to one or more pods. Keeping the mapping between the VIP and the pods up-to-date is the job of kube-proxy, a process that runs on every node, which queries the API server to learn about new services in the cluster.

Enough theory, let's have a look at a concrete example: in the following I'm using a simple containerized HTTP service that exposes endpoint0/ at port 9876 , returning a JSON payload echoing the server it is serving from along with some other infos.

Initially, let's deploy a pod under the supervision of a ReplicationController along with a Service, resulting in:

$ kubectl get pods

NAME READY STATUS RESTARTS AGE

sise-bkvhs 1/1 Running 0 9m $ kubectl describe pod sise-bkvhs

Name: sise-bkvhs

Namespace: namingthings

Security Policy: restricted

Node: 192.168.99.100/192.168.99.100

Start Time: Tue, 18 Apr 2017 09:45:22 +0100

Labels: app=sise

Status: Running

IP: 172.17.0.2

Controllers: ReplicationController/sise

...



You can now, from within the cluster, access the pod directly via its assigned IP 172.17.0.2 :

$ curl 172.17.0.2:9876/endpoint0

{"host": "172.17.0.2:9876", "version": "0.4.0", "result": "all is well"}



This is however, as mentioned above, not advisable since the IPs assigned to pods may change. Enter services:

$ kubectl get svc

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

simpleservice 172.30.40.155 80/TCP 1m $ kubectl describe svc simpleservice

Name: simpleservice

Namespace: namingthings

Labels: name=simpleservice

Selector: app=sise

Type: ClusterIP

IP: 172.30.40.155

Port: 80/TCP

Endpoints: 172.17.0.2:9876

Session Affinity: None

No events.



From within the cluster we can now access simpleservice like so:

$ curl 172.30.40.155:80/endpoint0

{"host": "172.30.40.155", "version": "0.4.0", "result": "all is well"}



So far so go. But what makes the VIP 172.30.40.155 forward the traffic to the pod? The answer is: IPTables. Think of IPTables as a long list of rules that tells the Linux kernel what to do with a certain IP package.

Looking at the rules that concern our service (executed on a cluster node) yields:

$ sudo iptables-save | grep simpleservice

-A KUBE-SEP-ASIN52LB5SMYF6KR -s 172.17.0.2/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ

-A KUBE-SEP-ASIN52LB5SMYF6KR -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.2:9876

-A KUBE-SERVICES -d 172.30.40.155/32 -p tcp -m comment --comment "namingthings/simpleservice: cluster IP" -m tcp --dport 80 -j KUBE-SVC-IKIIGXZ2IBFIBYI6

-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -j KUBE-SEP-ASIN52LB5SMYF6KR



Above you can see the four rules that kube-proxy has thankfully added to the routing table, essentially stating that TCP traffic to 172.30.40.155:80 should be forwarded to 172.17.0.2:9876 , which is our pod.

Let's now add a second pod by scaling up the ReplicationController supervising it:

$ kubectl scale --replicas=2 rc/sise

replicationcontroller "sise" scaled

$ kubectl get pods

NAME READY STATUS RESTARTS AGE

sise-bkvhs 1/1 Running 0 11m

sise-vncz4 1/1 Running 0 8s



When we now check the relevant parts of the routing table again we notice the addition of a bunch of rules:

$ sudo iptables-save | grep simpleservice

-A KUBE-SEP-ASIN52LB5SMYF6KR -s 172.17.0.2/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ

-A KUBE-SEP-ASIN52LB5SMYF6KR -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.2:9876

-A KUBE-SEP-RP53IYKEFRDLQANZ -s 172.17.0.4/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ

-A KUBE-SEP-RP53IYKEFRDLQANZ -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.4:9876

-A KUBE-SERVICES -d 172.30.40.155/32 -p tcp -m comment --comment "namingthings/simpleservice: cluster IP" -m tcp --dport 80 -j KUBE-SVC-IKIIGXZ2IBFIBYI6

-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ASIN52LB5SMYF6KR

-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -j KUBE-SEP-RP53IYKEFRDLQANZ



In above routing table listing we see rules for the newly created pod serving at 172.17.0.4:9876 as well as an additional rule -A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ASIN52LB5SMYF6KR which causes the traffic to the service being equally split between our two pods by invoking the statistics module of IPTables.

Summing up, this is our final setup:

Notice in above figure the different IP ranges for the pods, the service VIP as well as the node IP ( 192.168.99.100 in our example).

If you'd like to try out the above steps yourself, you can simply use the same setup I've used, that is, Minishift running locally on your machine. Ah, and before I forget it: kubectl on my machine is actually a symlink to oc , the OpenShift CLI client:

$ stat kubectl ; ls -l kubectl

File: "kubectl"

Size: 2 FileType: Symbolic Link

Mode: (0755/lrwxr-xr-x) Uid: ( 501/mhausenblas) Gid: ( 20/ staff)

Device: 1,4 Inode: 3685078 Links: 1

Access: Fri Apr 14 10:21:59 2017

Modify: Fri Apr 14 10:21:59 2017

Change: Fri Apr 14 10:21:59 2017

lrwxr-xr-x 1 mhausenblas staff 2 14 Apr 10:21 kubectl -> oc



Should you be interested in digging deeper into the topic of Kubernetes services and how to overcome scalability issues with large-scale service deployments—due to the inherent limitations of IPTables—check out the following resources:

I hope this post sheds some light on the topic of services in Kubernetes, what they do and how they work. Let me know what you think and which other Kubernetes objects we should have a look at in future posts.