I recently wrote about our experience taking a Kubenetes Operator to Production — and a large part of that, of course, is testing. This article will dive into the testing story a little deeper.

I will refer to our Prescaled CronJob operator throughout — found here: https://github.com/microsoft/k8s-cronjob-prescaler.

Getting Started

There are some really good articles and the Ginkgo docs to help you get bootstrapped and running. I’d suggest you read the following 2 guides first.

We’ll focus on testing Operators with Ginkgo.

Writing your Tests

Since our tests are basically sending objects to our operator in Kubernetes, and checking what we got back was correct, we need a Kubernetes manager & client. In our BeforeSuite method:

k8sManager, err = ctrl.NewManager(cfg, ctrl.Options{

Scheme: scheme.Scheme,

MetricsBindAddress: metricsAddr,

}) k8sManager.Start(ctrl.SetupSignalHandler()) k8sClient = k8sManager.GetClient()

Now let’s create a resource and fire it at k8s. The code below shows a Describe / Context / It block, where we’re sending *some* object to Kubernetes. This could be a pod, a deployment, or one of your own CRDs:

var _ = Describe("PrescaledCronJob Controller", func() {

Context("Cronjob Autogeneration", func() {

It("Should create cronjob correctly", func() {

toCreate := generatePSCSpec()

Expect(k8sClient.Create(ctx, &toCreate)).Should(Succeed()) ...

Now let’s get the object back, and check what it looks like. For this, we’ll use the Eventually() block from the Gomega matcher library — this is a handy helper function that will keep trying to do something until the matcher succeeds, or times out:

Eventually(func() bool {

err := k8sClient.Get(ctx, types.NamespacedName{Name:toCreate.Name, Namespace: namespace}, fetched)

return err == nil

}, timeout, interval).Should(BeTrue())

Now we’ll check it looks how it’s supposed to look:

Expect(fetchedAutogenCron.Name).To(Equal(autogenName))

// ... many more Expects...

Table Driven Tests

We wanted to test lots of similar scenarios, with different parameters. This was a great chance to use Ginkgo’s table driven tests. These are different from many other data-driven test frameworks, in that each row of data passed to the test autogenerates a new test at runtime.

Import the table extension:

. "github.com/onsi/ginkgo/extensions/table"

Create a DescribeTable block, including the function to run each time, and rows of data to be passed to it:

DescribeTable(“Integration test configurations”,

func(testCase testCase) { // your test logic here...

Expect(result).To(Equal(testCase.shouldPass))

Expect(err != nil).To(Equal(testCase.shouldError))

}, // add a line per test case here… Entry(“LONG TEST: 1 minute, 1 warmup”, testCase{minsApart: 1, warmUpMins: 1, shouldPass: true, shouldError: false}), Entry(“LONG TEST: 3 minutes, 1 warmup”, testCase{minsApart: 3, warmUpMins: 2, shouldPass: true, shouldError: false}), Entry(“LONG TEST: 4 minutes, 1 warmup”, testCase{minsApart: 4, warmUpMins: 2, shouldPass: true, shouldError: false}),

As you might notice, we created a basic testCase struct to pass around rather than lots of individual params.

Test Environment

You have 2 primary options for an environment to test with — either use a real cluster, or a local API endpoint. If your operator does not interact or depend on other parts of K8s, then use the inbuilt API endpoint. For our case, we needed to check that the default CronJob operator had received and created an autogenerated CronJob object, so we needed to use a real cluster.

Fortunately with the envtest package (“sigs.k8s.io/controller-runtime/pkg/envtest”), it’s really simple:

testEnv in suite_test.go

It’s also possible to apply customisations to a test environment before running each test:

CRDDirectoryPaths allow you to apply yaml at test time

This is a nice way to configure your test environment and make sure required resources are there before you run your tests — *however* — in our experience it was simpler to apply yaml elsewhere (such as the makefile), since we could decide when or if we wanted to apply it.

Testing against Kind

Kind was crucial for integration testing, as you can run a full cluster locally — even in a build pipeline. Since the testEnv is now using a ‘real’ cluster — it will just fire commands against whatever kubectl is pointing at. For us, that’s kind.

In the makefile:

deploy-kind: kind-start kind-load-img deploy-cluster manifests install-crds kustomize-deployment

The above make command will start Kind then deploy all our customisations and operator.

To run the tests we then run:

kind-tests: ginkgo --skip="LONG TEST:" --nodes 6 --race --randomizeAllSpecs --cover --trace --progress --coverprofile ../controllers.coverprofile ./controllers -kubectl delete prescaledcronjobs --all -n psc-system

The above make command does a few things:

Skips any test described as “LONG TEST:*”. This is a pretty low-fi way of selecting which tests to run locally, on every build, vs maybe longer running integration tests.

Parallelises the testing across 6 go routines ( --nodes 6 ). This was a great way of speeding up overall test execution.

). This was a great way of speeding up overall test execution. Runs the coverage tool

Tries to delete the resources we made during the tests. The - character in the makefile says “just carry on if this line throws an error”.

Testing against a real cluster

One of the best things about this whole approach is that it’s trivial to now run your tests against a ‘real’ cluster. Ours was in Azure. All you need to do is set your kubectx to be pointing at your real cluster instead of Kind, deploy your customisations, and run the tests again.

deploy-cluster: manifests install-crds kustomize-deployment $> make deploy-cluster

$> make kind-tests

With this reuse of tests, it’s easy to see how you might want to run tests against Kind in a build, and maybe run the integration test suite against your real cluster during a release process.

Outputting Test Results

Since many build agents (we used Azure Devops) don’t ‘understand’ go test results, there’s a handy extension you can use to make Ginkgo output the test results in JUnit format, which many build agents understand. It the entry point to your tests:

junitReporter := reporters.NewJUnitReporter(fmt.Sprintf("../TEST-ginkgo-junit_%d.xml", config.GinkgoConfig.ParallelNode)) RunSpecsWithDefaultAndCustomReporters(t, "Controller Suite", []Reporter{envtest.NewlineReporter{}, junitReporter})

This will create an xml file per ‘node’ (go routine) in your test. But build agents should pick them all up to report results: