| Kafka 4 Adds Queue Semantics Support |
| Written by Kay Ewbank |
| Thursday, 17 April 2025 |
|
Kafka 4.0 has been released, with major changes. This is the first version to operate entirely in KRaft mode by default. It also adds a new consumer group protocol designed to dramatically improve rebalance performance, and includes early access to Queues for Kafka, enabling Kafka to support traditional queue semantics directly. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications. It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.
Apache Kafka 4.0 is a significant milestone, marking the first major release to operate entirely without Apache ZooKeeper. Instead, it runs in KRaft mode by default, simplifying deployment and management and enhancing scalability. Kafka 4.0 also introduces what the team says is a powerful new consumer group protocol designed to dramatically improve rebalance performance. They say this optimization significantly reduces downtime and latency, enhancing the reliability and responsiveness of consumer groups, especially in large-scale deployments. The new version also includes early access to Queues for Kafka. This is not the traditional concept of a queue, but instead introduces the concept of a share group as a way of enabling cooperative consumption using Kafka topics. A share group is roughly equivalent to a "durable shared subscription" in existing systems. The Kafka Improvement Proposal about queues says this is queues done in a Kafka way, with no maximum queue depth and the ability to reset to a specific time for point-in-time recovery. The plan is that this will widen the uses for which Kafka can be a good messaging platform, particularly for those requiring point-to-point messaging patterns. There are a number of other improvements, including an extension for cluster operators to collect client metrics directly from brokers using a plugin. The previous extension only gathers metrics from Kafka clients (admin, consumer, and producer), and doesn't cover application-level metrics for embedded clients, such as Kafka Streams. This addition introduces a way for applications that use Kafka clients to include their own metrics alongside the existing client metrics. Kafka 4.0 is available now.
More InformationRelated ArticlesKafka 3.9 Adds Dynamic KRaft Quorums Kafka Adds KRaft-Based Authorizer To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin. 
Comments
or email your comment to: comments@i-programmer.info |
