Apache Kafka, the big data and data streams platform, has been updated in a new version that adds new metrics and improves security.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications. It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

Work has continued in this released to the removal of direct access to Apache ZooKeeper from Kafka's admin tools. This will improve security, decouple the server-side metadata format from the client, and is a necessary first step towards storing Kafka metadata in Kafka.

The work carried out on this means that dynamic configs will no longer require ZooKeeper access. The new release also ships with a more up-to-date version of ZooKeeper that adds ZooKeeper TLS support. This is used when deploying a secure Kafka cluster, remove direct ZooKeeper access from the Kafka Administrative tools.

Another noteworthy change simplifies the API for applications that read from and write to Kafka transactionally. Until now, you had to use separate producer instances for each input partition, but that's no longer the case, so it's easier to build EOS applications that consume large numbers of partitions. The developers say this is foundational for a similar improvement in Kafka Streams in the next release.

Reliability has been improved for the idempotent/transactional producer, overcoming a problem with producer state retention on the broker. Previously, when the log was truncated to enforce retention or truncated from a call to delete records, the broker dropped producer state, which led to errors. With this improvement, the broker instead retains producer state until expiration.

More Information

Kafka Website

Related Articles

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

Kafka Webview Released

Comparing Kafka To RabbitMQ

Apache Kafka Adds New Streams API

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info