Lately I’ve been writing some Chef code. One of the best things about Chef is custom resources: https://docs.chef.io/custom_resources.html

Let’s see an example on how to create a Kafka topic using Chef and how to make it idempotent.

Before writing any Chef code it is important to understand how to manage a topic (grouping of messages of a similar type).

From the Kafka install directory, first check if the topic already exists:

bin/kafka-topics.sh

--zookeeper localhost:2181 \

--describe \

--topic <name>

If it doesn't exist, create it:

bin/kafka-topics.sh

--zookeeper localhost:2181 \

--create \

--topic <name> \

--replication-factor <value> \

--partitions <value>

If the topic exists, to increase the number of partitions (way to parallelize the consumption of messages from a topic):

bin/kafka-topics.sh

--zookeeper localhost:2181 \

--alter \

--topic <name> \

--partitions <value>

If the topic exists, to increase the number of replicas (number of copies of the partitions for data redundancy):

bin/kafka-reassign-partitions.sh \

--zookeeper localhost:2181 \

--reassignment-json-file increase-replication-factor.json \

--execute

Where the file increase-replication-factor.json follows the format:

{

"version": 1,

"partitions": [

{

"topic":"foo",

"partition": 0,

"replicas":[1, 2, 3]

}

]

}

The basic custom resource structure:

Using the resource (cookbook example):

With this code creating topics is possible. However, it isn’t checking if the topic already exists or if the settings are valid.