Apache Kafka has been updated with a new way to run Apache Kafka with self-managed metadata. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

The main improvement to Kafka is the marking of KRaft mode as production ready, though only for new clusters. This gives a way to run Apache Kafka with self-managed metadata, so eliminating the need to run an Apache Zookeeper cluster alongside every Kafka cluster. The new KRaft mode improves partition scalability and resiliency while simplifying deployments of Apache Kafka.

There are some features that are currently supported by Apache ZooKeeper mode that are not yet supported by KRaft mode. These include configuring SCRAM users via the administrative API, supporting JBOD configurations with multiple storage directories, and modifying certain dynamic configurations on the standalone KRaft controller. The plan is that these features will be added over the next few months.

Other additions include transactional EOS for source connectors. Exactly Once Semantics (EOS) is a framework that allows stream processing applications such as Kafka Streams to process data through Kafka without loss or duplication. The second improvement (which is also part of the work supporting EOS for source connectors) is support for atomic commit of source connector records and offsets, meaning that their offsets are committed if and only if all records up to that point have been ack'd by the producer.

The other new feature is the ability for the admin API to use batched offset fetch requests.

Kafka 3.3 is available now. 

More Information

Kafka Website

Related Articles

Kafka Adds KRaft-Based Authorizer  

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

Apache Kafka 2.7 Updates Broker

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info