Release Notes - Kafka - Version 2.5.0

Below is a summary of the JIRA issues addressed in the 2.5.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-6049] - Kafka Streams: Add Cogroup in the DSL

[KAFKA-6144] - Allow serving interactive queries from in-sync Standbys

[KAFKA-7251] - Add support for TLS 1.3

[KAFKA-8843] - Zookeeper migration tool support for TLS

[KAFKA-9352] - unbalanced assignment of topic-partition to tasks

[KAFKA-9445] - Allow fetching a key from a single partition rather than iterating over all the stores on an instance

[KAFKA-9487] - Followup : KAFKA-9445(Allow fetching a key from a single partition); addressing code review comments

Improvement

[KAFKA-6607] - Kafka Streams lag not zero when input topic transactional

[KAFKA-6614] - kafka-streams to configure internal topics message.timestamp.type=CreateTime

[KAFKA-6819] - Refactor build-in StreamsMetrics internal implementations

[KAFKA-7052] - ExtractField SMT throws NPE - needs clearer error message

[KAFKA-7317] - Use collections subscription for main consumer to reduce metadata

[KAFKA-7538] - Improve locking model used to update ISRs and HW

[KAFKA-7639] - Read one request at a time from socket to reduce broker memory usage

[KAFKA-7658] - Add KStream#toTable to the Streams DSL

[KAFKA-7689] - Add Commit/List Offsets Operations to AdminClient

[KAFKA-8179] - Incremental Rebalance Protocol for Kafka Consumer

[KAFKA-8455] - Add VoidSerde to Serdes

[KAFKA-8482] - alterReplicaLogDirs should be better documented

[KAFKA-8503] - Implement default.api.timeout.ms for AdminClient

[KAFKA-8507] - Support --bootstrap-server in all command line tools

[KAFKA-8710] - InitProducerId changes for KIP-360

[KAFKA-8805] - Bump producer epoch following recoverable errors

[KAFKA-8821] - Avoid pattern subscription to allow for stricter ACL settings

[KAFKA-8834] - Distinguish URPs caused by reassignment plus other metrics

[KAFKA-8855] - Collect and Expose Client's Name and Version in the Brokers

[KAFKA-8904] - Reduce metadata lookups when producing to a large number of topics

[KAFKA-8953] - Consider renaming `UsePreviousTimeOnInvalidTimestamp` timestamp extractor

[KAFKA-8960] - Move Task determineCommitId in gradle.build to Project Level

[KAFKA-8964] - Refactor Stream-Thread-level Metrics

[KAFKA-8968] - Refactor Task-level Metrics

[KAFKA-8980] - Refactor State-Store-level Metrics

[KAFKA-8991] - Enable scalac optimizer

[KAFKA-9011] - Add KStream#flatTransform and KStream#flatTransformValues to Scala API

[KAFKA-9016] - Warn when log dir stopped serving replicas

[KAFKA-9039] - Optimize replica fetching CPU utilization with large number of partitions

[KAFKA-9040] - Implement --all option for describing configs

[KAFKA-9086] - Refactor Processor Node Streams Metrics

[KAFKA-9091] - KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

[KAFKA-9098] - Name Repartition Filter, Source, and Sink Processors

[KAFKA-9102] - Increase default zk session timeout and max lag

[KAFKA-9106] - metrics exposed via JMX shoud be configurable

[KAFKA-9110] - Improve efficiency of disk reads when TLS is enabled

[KAFKA-9115] - Add Roll-up Throughput Sensor for Processing on Source Nodes

[KAFKA-9152] - Improve Sensor Retrieval

[KAFKA-9183] - SaslSslAdminClientIntegrationTest takes too long to execute

[KAFKA-9202] - serde in ConsoleConsumer with access to headers

[KAFKA-9226] - Section on deletion of segment files is out of date

[KAFKA-9277] - move all group state transition rules into their states

[KAFKA-9316] - ConsoleProducer help info not expose default properties

[KAFKA-9324] - Drop support for Scala 2.11 (KIP-531)

[KAFKA-9337] - Simplifying standalone mm2-connect config

[KAFKA-9360] - emitting checkpoint and heartbeat set to false will not disable the activity in their SourceTask

[KAFKA-9375] - Add thread names to kafka connect threads

[KAFKA-9384] - Loop improvements

[KAFKA-9405] - Use Map API computeIfAbsent Where Applicable

[KAFKA-9408] - Use StandardCharsets UTF-8 instead of UTF-8 Name

[KAFKA-9420] - Bump APIs to enable flexible versions

[KAFKA-9422] - Track the set of topics a connector is using

[KAFKA-9426] - OffsetsForLeaderEpochClient Use Switch Statement

[KAFKA-9437] - KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[KAFKA-9460] - Enable TLSv1.2 by default and disable all others protocol versions

[KAFKA-9474] - Kafka RPC protocol should support type 'double'

[KAFKA-9477] - Doc: Add RoundRobinAssignor as an option to consumer configs

[KAFKA-9483] - Add Scala KStream#toTable to the Streams DSL

[KAFKA-9499] - Improve deletion process by batching more aggressively

[KAFKA-9515] - Upgrade ZooKeeper to 3.5.7

[KAFKA-9567] - Docs and system tests for ZooKeeper 3.5.7 and KIP-515

[KAFKA-9575] - "Notable changes in 2.5.0" doesn't mention ZooKeeper 3.5.7

[KAFKA-9586] - Fix errored json filename in ops documentation

[KAFKA-9758] - Add documentations for KIP-523 and KIP-527

[KAFKA-9810] - Document Connect Root REST API on /

Bug

[KAFKA-3061] - Get rid of Guava dependency

[KAFKA-4203] - Java producer default max message size does not align with broker default

[KAFKA-5868] - Kafka Consumer Rebalancing takes too long

[KAFKA-6266] - Kafka 1.0.0 : Repeated occurrence of WARN Resetting first dirty offset of __consumer_offsets-xx to log start offset 203569 since the checkpointed offset 120955 is invalid. (kafka.log.LogCleanerManager$)

[KAFKA-7925] - Constant 100% cpu usage by all kafka brokers

[KAFKA-8025] - Flaky Test RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest#shouldForwardAllDbOptionsCalls

[KAFKA-8030] - Flaky Test TopicCommandWithAdminClientTest#testDescribeUnderMinIsrPartitionsMixed

[KAFKA-8081] - Flaky Test TopicCommandWithAdminClientTest#testDescribeUnderMinIsrPartitions

[KAFKA-8162] - IBM JDK Class not found error when handling SASL authentication exception

[KAFKA-8211] - Flaky Test: ResetConsumerGroupOffsetTest.testResetOffsetsExportImportPlan

[KAFKA-8245] - Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups

[KAFKA-8340] - ServiceLoader fails when used from isolated plugin path directory

[KAFKA-8700] - Flaky Test QueryableStateIntegrationTest#queryOnRebalance

[KAFKA-8705] - NullPointerException was thrown by topology optimization when two MergeNodes have common KeyChaingingNode

[KAFKA-8764] - LogCleanerManager endless loop while compacting/cleaning segments

[KAFKA-8786] - Deprecated Gradle features making it incompatible with Gradle 6.0.

[KAFKA-8819] - Plugin path for converters not working as intended

[KAFKA-8842] - Reading/Writing confused in Connect QuickStart Guide

[KAFKA-8928] - Logged producer config does not always match actual config values

[KAFKA-8940] - Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance

[KAFKA-8945] - Incorrect null check in the constructor for ConnectorHealth and AbstractState

[KAFKA-9002] - Flaky Test org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenCreated

[KAFKA-9014] - AssertionError thrown by SourceRecordWriteCounter when SourceTask.poll returns an empty list

[KAFKA-9024] - org.apache.kafka.connect.transforms.ValueToKey throws NPE

[KAFKA-9025] - ZkSecurityMigrator not working with zookeeper chroot

[KAFKA-9051] - Source task source offset reads can block graceful shutdown

[KAFKA-9056] - Selector outbound byte metric does not count partial sends

[KAFKA-9068] - Fix incorrect JavaDocs for `Stores.xxxSessionStore(...)`

[KAFKA-9074] - Connect's Values class does not parse time or timestamp values from string literals

[KAFKA-9077] - System Test Failure: StreamsSimpleBenchmarkTest

[KAFKA-9083] - Various parsing issues in Values class

[KAFKA-9131] - failed producer metadata updates result in the unrelated error message

[KAFKA-9143] - DistributedHerder misleadingly log error on connector task reconfiguration

[KAFKA-9159] - Consumer.endOffsets Throw TimeoutException: Failed to get offsets by times in 30000ms after a leader change

[KAFKA-9169] - Standby Tasks point ask for incorrect offsets on resuming post suspension

[KAFKA-9184] - Redundant task creation and periodic rebalances after zombie worker rejoins the group

[KAFKA-9192] - NullPointerException if field in schema not present in value

[KAFKA-9204] - ReplaceField transformation fails when encountering tombstone event

[KAFKA-9218] - MirrorMaker 2 can fail to create topics

[KAFKA-9219] - NullPointerException when polling metrics from Kafka Connect

[KAFKA-9233] - Kafka consumer throws undocumented IllegalStateException

[KAFKA-9241] - SASL Clients are not forced to re-authenticate if they don't leverage SaslAuthenticateRequest

[KAFKA-9255] - MessageSet v1 protocol wrong specification

[KAFKA-9258] - Connect ConnectorStatusMetricsGroup sometimes NPE

[KAFKA-9267] - ZkSecurityMigrator should not create /controller node

[KAFKA-9287] - Transaction completion may block unnecessarily after abortable error

[KAFKA-9297] - CreateTopic API do not work with older version of the request/response

[KAFKA-9305] - Add version 2.4 to streams system tests

[KAFKA-9329] - KafkaController::replicasAreValid should return error

[KAFKA-9330] - Calling AdminClient.close in the AdminClient completion callback causes deadlock

[KAFKA-9338] - Incremental fetch sessions do not maintain or use leader epoch for fencing purposes

[KAFKA-9355] - RocksDB statistics are removed from JMX when EOS enabled and empty local state dir

[KAFKA-9364] - Fix misleading consumer logs on throttling

[KAFKA-9379] - Flaky Test TopicCommandWithAdminClientTest.testCreateAlterTopicWithRackAware

[KAFKA-9386] - Flaky test AclAuthorizerTest.testHighConcurrencyDeletionOfResourceAcls

[KAFKA-9390] - Non-key joining of KTable not compatible with confluent avro serdes

[KAFKA-9457] - Flaky test org.apache.kafka.common.network.SelectorTest.testGracefulClose

[KAFKA-9462] - Correct exception message in DistributedHerder

[KAFKA-9480] - Value for Task-level Metric process-rate is Constant Zero

[KAFKA-9491] - Fast election during reassignment can lead to replica fetcher failures

[KAFKA-9492] - ProducerResponse with record-level errors throw NPE with older client version

[KAFKA-9500] - Foreign-Key Join creates an invalid topology

[KAFKA-9503] - TopologyTestDriver processes intermediate results in the wrong order

[KAFKA-9505] - InternalTopicManager may falls into infinite loop with partially created topics

[KAFKA-9507] - AdminClient should check for missing committed offsets

[KAFKA-9512] - Flaky Test LagFetchIntegrationTest.shouldFetchLagsDuringRestoration

[KAFKA-9517] - KTable Joins Without Materialized Argument Yield Results That Further Joins NPE On

[KAFKA-9535] - Metadata not updated when consumer encounters FENCED_LEADER_EPOCH

[KAFKA-9556] - KIP-558 cannot be fully disabled and when enabled topic reset not working on connector deletion

[KAFKA-9558] - getListOffsetsCalls doesn't update node in case of leader change

[KAFKA-9563] - Fix Kafka connect consumer and producer override documentation

[KAFKA-9577] - Client encountering SASL_HANDSHAKE protocol version errors on 2.5 / trunk

[KAFKA-9599] - create unique sensor to record group rebalance

[KAFKA-9601] - Workers log raw connector configs, including values

[KAFKA-9634] - ConfigProvider does not document thread safety

[KAFKA-9658] - Removing default user quota doesn't take effect until broker restart

[KAFKA-9661] - Config synonyms are no longer included in kafka-configs --describe output

[KAFKA-9662] - Throttling system test fails when messages are produced before consumer starts up

[KAFKA-9668] - Iterating over KafkaStreams.getAllMetadata() results in ConcurrentModificationException

[KAFKA-9700] - Negative estimatedCompressionRatio leads to misjudgment about if there is no room

[KAFKA-9701] - Consumer could catch InconsistentGroupProtocolException during rebalance

[KAFKA-9706] - Flatten transformation fails when encountering tombstone event

[KAFKA-9707] - InsertField.Key transformation should apply to tombstone records

[KAFKA-9712] - Reflections library 0.9.12 introduced in 2.5 causes regression scanning for plugins on plugin_path

[KAFKA-9741] - ConsumerCoordinator must update ConsumerGroupMetadata before calling onPartitionsRevoked()

[KAFKA-9749] - TransactionMarkerRequestCompletionHandler should treat storage exceptions as retriable

[KAFKA-9750] - Flaky test kafka.server.ReplicaManagerTest.testFencedErrorCausedByBecomeLeader

[KAFKA-9752] - Consumer rebalance can be stuck after new member timeout with old JoinGroup version

[KAFKA-9770] - Caching State Store does not Close Underlying State Store When Exception is Thrown During Flushing

[KAFKA-9801] - Static member could get empty assignment unexpectedly

[KAFKA-9807] - Race condition updating high watermark allows reads above LSO

[KAFKA-9815] - Consumer may never re-join if inconsistent metadata is received once

Task

[KAFKA-7737] - Consolidate InitProducerId API

[KAFKA-8963] - Benchmark and optimize incremental fetch session handler

[KAFKA-9343] - Add ps command for Kafka and zookeeper process on z/OS.

[KAFKA-9606] - Document Metrics Changes from KIP-444

Test

[KAFKA-9181] - Flaky test kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe

[KAFKA-9188] - Flaky Test SslAdminClientIntegrationTest.testSynchronousAuthorizerAclUpdatesBlockRequestThreads

[KAFKA-9319] - Run some system tests using TLSv1.3

[KAFKA-9334] - Add more unit tests for Materialized class

[KAFKA-9509] - Fix flaky test MirrorConnectorsIntegrationTest.testReplication

[KAFKA-9523] - Reduce flakiness of BranchedMultiLevelRepartitionConnectedTopologyTest

Sub-task