Apache Kafka, the big data and datastreams platform, has been updated in a new version that the developers say moves closer to replacing ZooKeeper in Kafka with a self-managed metadata quorum.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

Currently, Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller.  The developers are working to remove this dependency to enable them to manage metadata in a more scalable and robust way, enabling support for more partitions.  It will also simplify the deployment and configuration of Kafka.

The latest work on this large task involves a new AlterIsr API to replace the notification znode so the controller has the exclusive ability to update Leader and ISR state. Leaders will use this API to request an ISR change from the controller rather than directly mutating the underlying state.

The work on replacing ZooKeeper is far from over. There are currently seven tasks in active development to provide support for more partitions per cluster, simpler operation, and tighter security.

 The new release, Kafka 2.7, also provides the addition of the Core Raft Implementation. This is part of a larger set of work to improve the architecture of Kafka to be based on a replicated metadata log which is maintained by a self-managed quorum. Kafka relies heavily on log replication, and the developers want to use the ideas behind  is at the heart of Kafka, and it is also at the heart of consensus protocols such as Raft. The latest work adds a separate "raft" module containing the core consensus protocol. 

Other improvements adds metrics to Kafka Streams to report properties of RocksDB, and support for sliding windows aggregations in the DSL.

More Information

Kafka Website

Related Articles

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

Kafka Webview Released

Comparing Kafka To RabbitMQ

Apache Kafka Adds New Streams API

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info