Apache Kafka has been updated with improvements including the ability to to migrate Kafka clusters from ZK to KRaft mode with no downtime. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

The main improvement to Kafka 3.4 is the ability to migrate Kafka clusters from ZooKeeper to KRaft mode. The migration copies the cluster metadata from ZooKeeper to the KRaft metadata log. During the migration, brokers are restarted in KRaft mode one at a time allowing the whole migration process to happen without cluster downtime. The Kafka developers say this is an early access feature and they're looking for feedback.

The next improvement is an expansion of the metadata each group member passes to the group leader so that it's possible to check whether a partition is still owned and thus valid.

Other improvements add the option to disable JMX Reporter, and configure Kafka to disable JMX reporter for environments where it’s not being used. Producer IDs have also received attention, so you can now set a separate configuration for producer ID expiry, and can clean up producer IDs more efficiently to avoid excess memory usage.

Rack awareness for partitions has been added to this release, with new rack-aware partition assignment for Kafka Consumers, and for consumer rebalancing.

Kafka Connect has also received attention, with a new custom Kafka Admin interface for MirrorMaker2 that means users can run MirrorMaker2 with custom implementations for the Kafka resource manager. 

Kafka 3.4 is available now. 

More Information

Kafka Website

Related Articles

Kafka Adds KRaft-Based Authorizer  

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info