Kafka 3.0 Released With KRaft
Written by Kay Ewbank   
Friday, 24 September 2021

Apache has released Kafka 3.0 with new features, breaking API changes, and improvements to KRaft—Apache Kafka’s built-in consensus mechanism that will replace Apache ZooKeeper.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.


It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

One major improvement to Apache 3.0 is the more mature version of KRaft, the internal Raft implementation that allows you to run Kafka brokers without Apache ZooKeeper. KRaft is not yet recommended for production, but the team says they have made many improvements to the KRaft metadata and APIs. A new major feature in 3.0 is the ability for KRaft Controllers and KRaft Brokers to generate, replicate, and load snapshots for the metadata topic partition that is used by the Kafka Cluster to store and replicate metadata information about the cluster like Broker configuration, topic partition assignment, and leadership,

Kafka Connect is another area to have been improved. Connect is a Kafka component that acts as a data hub for data integration between databases, key-value stores, search indexes, and file systems. The new version handles task restarts more smoothly. Until now, if you wanted to restart the whole connector, you had to make individual calls to restart the Connector instance and the Task instances. In 3.0, KIP-745 gives the ability to the users to restart either all or only the failed of a connector’s Connector and Task instances with a single call.

Kafka Streams has improvements in timestamp-based synchronization. The way Streams tasks fetch records has been enhanced, and this is supported by a new method in the Kafka Consumer API to deal locally with the consumer lag of a specific partition.

Another change of note is to MirrorMaker.This is a stand-alone tool for copying data between two Apache Kafka clusters, and it now has more flexible configuration options.

Kafka 3 is available now. 



More Information

Kafka Website

Related Articles

Kafka Replaces Zookeeper With Quorum

Apache Kafka 2.7 Updates Broker

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams



To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.


Code.Org Distributing $1 Million In CS Leaders Prizes

This year's Hour of Code will take place as usual during December's Computer Science Education Week. Meanwhile there's a deadline of November 21st for applying for a CS Leaders Prize of $10,000 to&nbs [ ... ]

Apple Looks Set To Lose In Epic Case

The most important legal case to programmers at the moment is Apple v Epic Games. A year ago Epic lost and Apple won the right to impose its 30% app tax on programmers and users alike. Now there are s [ ... ]

More News





or email your comment to: comments@i-programmer.info