Kafka 3.9 has been released. The team says this is a major release and the final in the 3.x line. It This will also be the final major release to feature the deprecated Apache ZooKeeper mode. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

a major improvement to this version is support for dynamic KRaft quorums, which Kafka relies on to store its metadata log when in KRaft mode. Until now, controller quorums were specified by a static configuration that needed to be set on all brokers and controllers. This made it difficult to make changes without downtime to the hostnames of controllers, or to add or remove them. KRaft Controller Membership Changes makes quorum membership dynamic. The Kafka team says this feature has been eagerly awaited by the community, and that while there is a main limitation in 3.9 in a lack of support for converting over static metadata quorums to dynamic ones, this ability is coming soon.

As this is the last version to support ZooKeeper mode, it isn't surprising that it also offers improved ZooKeeper migration. Colin McCabe, who currently works for Confluent to improve the performance, scalability, and "general awesomeness" of Apache Kafka, said that Kafka 3.9 is the final and best iteration of the current ZooKeeper migration feature, and the developers have removed the reasons that kept some users tied to ZooKeeper:

"Kafka 4.0 will be fully saying goodbye to ZooKeeper. There will be no support for running in ZK mode, or migrating from ZK mode."

This means that administrators that are still using the deprecated ZooKeeper mode who need to upgrade to 4.0 and beyond, will need to do an interim upgrade to version 3.9 before completing the move to 4.0 or beyond.

Tiered storage is now production ready in this release. Tiered storage allows Kafka to seamlessly offload data to pluggable external storage systems, such as cloud object stores, and this version also comes with a mechanism for dynamically disabling tiered storage on a per-topic basis, and the ability to set quotas on tiered storage upload and download rates.

Elsewhere, Kafka Connect handles nullable values better and you can set a REST endpoint that can be used to determine if Kafka Connect workers are healthy.

Kafka 3.9 is available now. 

More Information

Kafka Website

Related Articles

Kafka Adds KRaft Migration

Kafka Adds KRaft-Based Authorizer  

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info