As a Developer, it is always useful to be able to debug an application with its own IDE.

When your application only works with the Kubernetes API, you can simply launch your application in the IDE and connect it to the remote Kubernetes API.

But When your application needs to connect to other systems which are only available inside the Kubernetes cluster, this solution does not work anymore.

Build the application

The application I want to debug is a Cassandra Operator which is based on the CoreOS Operator SDK

The Operator used the script build.sh to build the Go application, I added an input parameter DEBUG to the script so that it can build a debug version of the application, with the addition of specific gcflags, and then suffix the binary with -debug.

Note that we also added the dlv binary (which is the Go debugger) to our target binaries.

#!/usr/bin/env bash



set -o errexit

set -o nounset

set -o pipefail



if ! which go > /dev/null; then

echo "golang needs to be installed"

exit 1

fi



BIN_DIR="$(pwd)/tmp/_output/bin"

mkdir -p ${BIN_DIR}

PROJECT_NAME="cassandra-operator"

REPO_PATH="gitlab.si.francetelecom.fr/kubernetes/cassandra-operator"

BUILD_PATH="${REPO_PATH}/cmd/${PROJECT_NAME}"



if [ $# -gt 0 ] && [ "$1" = "DEBUG" ] ; then

echo "building "${PROJECT_NAME}" In DEBUG Mode..."

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -gcflags "-N -l" -o ${BIN_DIR}/${PROJECT_NAME}-debug $BUILD_PATH

cp /usr/local/bin/dlv ${BIN_DIR}

else

echo "building "${PROJECT_NAME}"..."

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ${BIN_DIR}/${PROJECT_NAME} $BUILD_PATH

fi

Build the Docker Image

The Operator SDK generate a Dockerfile to build the image for our operator, and a script (docker_build.sh) used to build it, on which we also add the DEBUG parameter :

#!/usr/bin/env bash



if ! which docker > /dev/null; then

echo "docker needs to be installed"

exit 1

fi



: ${IMAGE:?"Need to set IMAGE, e.g. gcr.io/<repo>/<your>-operator"}



if [ $# -gt 0 ] && [ "$1" = "DEBUG" ] ; then

echo "building container ${IMAGE} in DEBUG Mode..."

docker build -t "${IMAGE}" -f tmp/build/Dockerfile-debug .

else

echo "building container ${IMAGE}..."

docker build -t "${IMAGE}" -f tmp/build/Dockerfile .

fi

And we add a new Dockerfile-debug with only those Modifications at the end :

....

ADD tmp/_output/bin/cassandra-operator-debug /usr/local/bin

ADD tmp/_output/bin/dlv /usr/local/bin



EXPOSE 40000



ENTRYPOINT ["/usr/local/bin/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "exec", "/usr/local/bin/cassandra-operator-debug"]

We add the debug version of the application and the delve debugger inside the docker image. We also changed the entrypoint telling the image to start the debugger which will then execute the operator in debug mode.

Delve exposes the port 40000, on which we will configure our IDE to communicate With.

Excerpt of the Makefile used to build the whole operator in debug mode :

docker-build-debug: docker-get-deps

echo "Generate CRD Client"

tmp/codegen/update-generated.sh

echo "Build Go Application In DEBUG Mode"

docker run --rm -v $(PWD):$(WORKDIR):rw $(REPOSITORY)/dev:$(VERSION) /bin/bash -c './tmp/build/build.sh DEBUG'

echo "Build Docker Image With DEBUG Enabled"

IMAGE=$(REPOSITORY):$(VERSION)-debug ./tmp/build/docker_build.sh DEBUG

Deploy the Application

Once you have compiled the operator in debug mode and recreate the Docker Image, we also need a small change in the deployment in order to expose the port 40000 and to add the SYS_PTRACE capability to the operator Pod in the cluster. I used a Helm chart in order to deploy the application which manages a debug value to customize the deployment :

... {{- if .Values.debug }}

image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}-debug"

{{- else}}

image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

{{- end }}

imagePullPolicy: "{{ .Values.image.pullPolicy }}"

resources:

{{ toYaml .Values.resources | indent 10 }}

env:

- name: WATCH_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

ports:

- containerPort: 9710

name: metrics

protocol: TCP

{{- if .Values.debug }}

- containerPort: 40000

name: debug

protocol: TCP

securityContext:

capabilities:

add:

SYS_PTRACE

{{- end }}

To deploy the operator in debug mode I just need to surcharge the debug value :

helm install ./helm/cassandra-operator --name cassandra-operator-debug --set debug=true

This will create all necessary Kubernetes objects for the operator. If you have a TCP ingress you can configure a specific route to reach the port 40000 of your pod. Since I don’t have one, I will create a port-forward to be able to reach this port from my local machine where my IDE sits:

kubectl port-forward <cassandra-operator-debug-pod-name> 40000:40000

Configure the IDE

I used Goland to debug my Go programs, but it will work similarly with others IDE

We need to create a Remote Debug and we point it to localhost:40000 since I have activated the port forward.

From that point, we are able to set Breakpoint in our local IDE, and it will communicate with the delve debugger deployed in the Kubernetes cluster to allows to debug our application :

Finally

The application will wait for your IDE to connect to it before starting execution of code, that way you are able to start debugging from start.

The Pod will stop in the state completed each time you stop debugging on IDE side (or loose connection) and may be automatically restarted if you’re using a Kubernetes deployment.

You need to know that :