With the business data (vegetables) and the transaction-scoped metadata being stored in the database, it’s time to set up the Debezium Postgres connector and stream the data changes from the vegetable and transaction_context_data tables into corresponding Kafka topics. Again refer to the example README file for the details of deploying the connector .

The dbserver1.inventory.vegetable topic should contain change events for created, updated and deleted vegetable records, whereas the dbserver1.inventory.transaction_context_data topic should only contain create messages for each inserted metadata record.

In order to manage the growth of involved topics, the retention policy for each topic should be well-defined. For instance for the actual audit log topic with the enriched change events, a time based retention policy might be suitable, keeping each log event for as long as needed as per your requirements. The transaction metadata topic on the other hand can be fairly short-lived, as its entries are not needed any longer, once all corresponding data change events have been processed. It may be a good idea to set up some monitoring of the end-to-end lag in order to make sure the log enricher stream application keeps up with the incoming messages and doesn’t fall behind that far so it is at risk of transaction messages being discarded before processing the corresponding change events.

Now, if we look at messages from the two topics, we can see that they can be correlated based on the transaction id. It is part of the source structure of vegetable change events, and it is the message key of transaction metadata events:

Once we’ve found the corresponding transaction event for a given vegetable change event, the client_date , usecase and user_name attributes from the former can be added to the latter:

This kind of message transformation is a perfect use case for Kafka Streams , a Java API for implementing stream processing applications on top of Kafka topics, providing operators that let you filter, transform, aggregate and join Kafka messages.

As runtime environment for our stream processing application we’re going to use Quarkus , which is "a Kubernetes Native Java stack tailored for GraalVM & OpenJDK HotSpot, crafted from the best of breed Java libraries and standards".

The extension also comes with "live development" support, which automatically reloads the stream processing application while you’re working on it, allowing for very fast turnaround cycles during development.

Amongst many others, Quarkus comes with an extension for Kafka Streams , which allows to build stream processing applications running on the JVM and as native code compiled ahead-of-time. It takes care of the lifecycle of the streaming topology, so you don’t have to deal with details like registering JVM shutdown hooks, awaiting the creation of all input topics and more.

But unfortunately, there is a timing issue: as the messages are consumed from multiple topics, it may happen that at the point in time when an element from the vegetables stream is processed, the corresponding transaction metadata message isn’t available yet. So depending on whether we’d be using an inner join or a left join, we’d in this case either skip change events or propagate them without having enriched them with the transaction metadata. Both outcomes are not desirable.

If a KStream - KStream join isn’t feasible, what else could be done? A join between a KStream and GlobalKTable looks promising, too. It doesn’t have the co-partitioning requirements of stream-to-stream joins, as all partitions of the GlobalKTable are present on all nodes of a distributed Kafka Streams application. This seems like an acceptable trade-off, because the messages from the transaction metadata topic can be discarded rather quickly and the size of the corresponding table should be within reasonable bounds. So we could have a KStream sourced from the vegetables topic and a GlobalKTable based on the transaction metadata topic.

Now, in order to join the two streams, the message key must be the same on both sides. This means the vegetables topic must be re-keyed by transaction id (we cannot re-key the transaction metadata topic, as there’s no information about concerned vegetables contained in the metadata events; and even if that were the case, one transaction might impact multiple vegetable records). By doing so, we’d loose the original ordering guarantees, though. One vegetable record might be modified in two subsequent transactions, and its change events may end up in different partitions of the re-keyed topic, which may cause a consumer to receive the second change event before the first one.

Another problem arises in regards to ordering guarantees of the change events. By default, Debezium will use a table’s primary key as the message key for the corresponding Kafka messages. This means that all messages for the same vegetable record will have the same key and thus will go into the same partition of the vegetables Kafka topic. This in turn guarantees that a consumer of these events sees all the messages pertaining to the same vegetable record in the exact same order as they were created.

When thinking about the actual implementation of the enrichment logic, a stream-to-stream join might appear as a suitable solution. By creating KStream s for the two topics, we may try and implement the joining functionality. One challenge though is how to define a suitable joining window , as there is no timing guarantees between messages on the two topics, and we must not miss any event.

Customized Joins With Buffering

The combination of KStream and GlobalKTable still hints into the right direction. Only that instead of relying on the built-in join operators we’ll have to implement a custom joining logic. The basic idea is to buffer messages arriving on the vegetable KStream until the corresponding transaction metadata message is available from the GlobalKTable s state store. This can be achieved by creating a custom transformer which implements the required buffering logic and is applied to the vegetable KStream .

Let’s begin with the streaming topology itself. Thanks to the Quarkus Kafka Streams extension, a CDI producer method returning the Topology object is all that’s needed for that:

@ApplicationScoped public class TopologyProducer { static final String STREAM_BUFFER_NAME = "stream-buffer-state-store"; static final String STORE_NAME = "transaction-meta-data"; @ConfigProperty(name = "audit.context.data.topic") String txContextDataTopic; @ConfigProperty(name = "audit.vegetables.topic") String vegetablesTopic; @ConfigProperty(name = "audit.vegetables.enriched.topic") String vegetablesEnrichedTopic; @Produces public Topology buildTopology() { StreamsBuilder builder = new StreamsBuilder(); StoreBuilder<KeyValueStore<Long, JsonObject>> streamBufferStateStore = Stores .keyValueStoreBuilder( Stores.persistentKeyValueStore(STREAM_BUFFER_NAME), new Serdes.LongSerde(), new JsonObjectSerde() ) .withCachingDisabled(); builder.addStateStore(streamBufferStateStore); (1) builder.globalTable(txContextDataTopic, Materialized.as(STORE_NAME)); (2) builder.<JsonObject, JsonObject>stream(vegetablesTopic) (3) .filter((id, changeEvent) -> changeEvent != null) .filter((id, changeEvent) -> !changeEvent.getString("op").equals("r")) .transform(() -> new ChangeEventEnricher(), STREAM_BUFFER_NAME) .to(vegetablesEnrichedTopic); return builder.build(); } }

1 State store which will serve as the buffer for change events that cannot be processed yet 2 GlobalKTable based on the transaction metadata topic 3 KStream based on the vegetables topic; on this stream, any incoming tombstone markers are filtered, the reasoning being that the retention policy for an audit trail topic typically should be time-based than based on log compaction; similarly, snapshot events are filtered, assuming they are not relevant for an audit trail and there wouldn’t be any corresponding metadata provided by the application for the snapshot transaction initiated by the Debezium connector Any other messages are enriched with the corresponding transaction metadata via a custom Transformer (see below) and finally are written to an output topic

The topic names are injected using the MicroProfile Config API, with the values being provided in Quarkus application.properties configuration file. Besides the topic names, this file also has the information about the Kafka bootstrap server, default serdes any more:

audit.context.data.topic=dbserver1.inventory.transaction_context_data audit.vegetables.topic=dbserver1.inventory.vegetable audit.vegetables.enriched.topic=dbserver1.inventory.vegetable.enriched # may be overridden with env vars quarkus.kafka-streams.bootstrap-servers=localhost:9092 quarkus.kafka-streams.application-id=auditlog-enricher quarkus.kafka-streams.topics=${audit.context.data.topic},${audit.vegetables.topic} # pass-through kafka-streams.cache.max.bytes.buffering=10240 kafka-streams.commit.interval.ms=1000 kafka-streams.metadata.max.age.ms=500 kafka-streams.auto.offset.reset=earliest kafka-streams.metrics.recording.level=DEBUG kafka-streams.default.key.serde=io.debezium.demos.auditing.enricher.JsonObjectSerde kafka-streams.default.value.serde=io.debezium.demos.auditing.enricher.JsonObjectSerde kafka-streams.processing.guarantee=exactly_once

In the next step let’s take a look at the ChangeEventEnricher class, our custom transformer. The implemention is based on the assumption that change events are serialized as JSON, but of course it could be done equally well using other formats such as Avro or Protocol Buffers.

This is a bit of code, but hopefully its decomposition into multiple smaller methods makes it comprehensible:

class ChangeEventEnricher implements Transformer <JsonObject, JsonObject, KeyValue<JsonObject, JsonObject>> { private static final Long BUFFER_OFFSETS_KEY = -1L; private static final Logger LOG = LoggerFactory.getLogger(ChangeEventEnricher.class); private ProcessorContext context; private KeyValueStore<JsonObject, JsonObject> txMetaDataStore; private KeyValueStore<Long, JsonObject> streamBuffer; (5) @Override @SuppressWarnings("unchecked") public void init(ProcessorContext context) { this.context = context; streamBuffer = (KeyValueStore<Long, JsonObject>) context.getStateStore( TopologyProducer.STREAM_BUFFER_NAME ); txMetaDataStore = (KeyValueStore<JsonObject, JsonObject>) context.getStateStore( TopologyProducer.STORE_NAME ); context.schedule( Duration.ofSeconds(1), PunctuationType.WALL_CLOCK_TIME, ts -> enrichAndEmitBufferedEvents() ); (4) } @Override public KeyValue<JsonObject, JsonObject> transform(JsonObject key, JsonObject value) { boolean enrichedAllBufferedEvents = enrichAndEmitBufferedEvents(); (3) if (!enrichedAllBufferedEvents) { bufferChangeEvent(key, value); return null; } KeyValue<JsonObject, JsonObject> enriched = enrichWithTxMetaData(key, value); (1) if (enriched == null) { (2) bufferChangeEvent(key, value); } return enriched; } /** * Enriches the buffered change event(s) with the metadata from the associated * transactions and forwards them. * * @return {@code true}, if all buffered events were enriched and forwarded, * {@code false} otherwise. */ private boolean enrichAndEmitBufferedEvents() { (3) Optional<BufferOffsets> seq = bufferOffsets(); if (!seq.isPresent()) { return true; } BufferOffsets sequence = seq.get(); boolean enrichedAllBuffered = true; for(long i = sequence.getFirstValue(); i < sequence.getNextValue(); i++) { JsonObject buffered = streamBuffer.get(i); LOG.info("Processing buffered change event for key {}", buffered.getJsonObject("key")); KeyValue<JsonObject, JsonObject> enriched = enrichWithTxMetaData( buffered.getJsonObject("key"), buffered.getJsonObject("changeEvent")); if (enriched == null) { enrichedAllBuffered = false; break; } context.forward(enriched.key, enriched.value); streamBuffer.delete(i); sequence.incrementFirstValue(); } if (sequence.isModified()) { streamBuffer.put(BUFFER_OFFSETS_KEY, sequence.toJson()); } return enrichedAllBuffered; } /** * Adds the given change event to the stream-side buffer. */ private void bufferChangeEvent(JsonObject key, JsonObject changeEvent) { (2) LOG.info("Buffering change event for key {}", key); BufferOffsets sequence = bufferOffsets().orElseGet(BufferOffsets::initial); JsonObject wrapper = Json.createObjectBuilder() .add("key", key) .add("changeEvent", changeEvent) .build(); streamBuffer.putAll(Arrays.asList( KeyValue.pair(sequence.getNextValueAndIncrement(), wrapper), KeyValue.pair(BUFFER_OFFSETS_KEY, sequence.toJson()) )); } /** * Enriches the given change event with the metadata from the associated * transaction. * * @return The enriched change event or {@code null} if no metadata for the * associated transaction was found. */ private KeyValue<JsonObject, JsonObject> enrichWithTxMetaData(JsonObject key, JsonObject changeEvent) { (1) JsonObject txId = Json.createObjectBuilder() .add("transaction_id", changeEvent.get("source").asJsonObject() .getJsonNumber("txId").longValue()) .build(); JsonObject metaData = txMetaDataStore.get(txId); if (metaData != null) { LOG.info("Enriched change event for key {}", key); metaData = Json.createObjectBuilder(metaData.get("after").asJsonObject()) .remove("transaction_id") .build(); return KeyValue.pair( key, Json.createObjectBuilder(changeEvent) .add("audit", metaData) .build() ); } LOG.warn("No metadata found for transaction {}", txId); return null; } private Optional<BufferOffsets> bufferOffsets() { JsonObject bufferOffsets = streamBuffer.get(BUFFER_OFFSETS_KEY); if (bufferOffsets == null) { return Optional.empty(); } else { return Optional.of(BufferOffsets.fromJson(bufferOffsets)); } } @Override public void close() { } }

1 When a vegetables change event arrives, look up the corresponding metadata in the state store of the transaction topic’s GlobalKTable , using the transaction id from the source block of the change event as the key; if the metadata could be found, add the metadata to change event (under the audit field) and return that enriched event 2 If the metadata could not be found, add the incoming event into the buffer of change events and return 3 Before actually getting to the incoming event, all buffered events are processed; this is required to make sure that the original change events is retained; only if all could be enriched, the incoming event will be processed, too 4 In order to emit buffered events also if no new change event is coming in, a punctuation is scheduled that periodically processes the buffer 5 A buffer for vegetable events whose corresponding metadata hasn’t arrived yet

The key piece is the buffer for unprocessable change events. To maintain the order of events, the buffer must be processed in order of insertion, beginning with the event inserted first (think of a FIFO queue). As there’s no guaranteed traversing order when getting all the entries from a KeyValueStore , this is implemented by using the values of a strictly increasing sequence as the keys. A special entry in the key value store is used to store the information about the current "oldest" index in the buffer and the next sequence value.

One could also think of alternative implementations for such buffer, e.g. based on a Kafka topic or a custom KeyValueStore implementation that ensures iteration order from oldest to newest entry. Ultimately, it could also be useful if Kafka Streams came with built-in means of retrying a stream element that cannot be joined yet; this would avoid any custom buffering implementation.

If Things Go Wrong For a reliable and consistent processing logic it’s vital to think about the behavior in case of failures, e.g. if the stream application crashes after adding an element to the buffer but before updating the sequence value. The key to this is the exactly_once value of the processing.guarantee property given in application.properties. This ensures a transactionally consistent processing; e.g. in the aforementioned scenario, after a restart the original change event would be handled again, and the buffer state would look exactly like it did before the event was processed for the first time. Consumers of the enriched vegetable events should apply an isolation level of read_committed ; otherwise they may see uncommitted and thus duplicate messages in case of an application crash after a buffered event was forwarded but before it was removed from the buffer.

With the custom transformer logic in place, we can build the Quarkus project and run the stream processing application. You should see messages like this in the dbserver1.inventory.vegetable.enriched topic:

{"id":10} { "before": { "id": 10, "description": "Yummy!", "name": "Tomato" }, "after": { "id": 10, "description": "Tasty!", "name": "Tomato" }, "source": { "version": "0.10.0-SNAPSHOT", "connector": "postgresql", "name": "dbserver1", "ts_ms": 1569700445392, "snapshot": "false", "db": "vegetablesdb", "schema": "inventory", "table": "vegetable", "txId": 610, "lsn": 34204240, "xmin": null }, "op": "u", "ts_ms": 1569700445537, "audit": { "client_date": 1566461551000000, "usecase": "UPDATE VEGETABLE", "user_name": "farmermargaret" } }

Of course, the buffer processing logic may be adjusted as per your specific requirements; for instance instead of indefinitely waiting for corresponding transaction metadata, we may also decide that it makes more sense to propagate change events unenriched after some waiting time or to raise an exception indicating the missing metadata.