Apache Kafka, the distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications, has been updated. Improvements include extending SASL/OAUTHBEARER to add support for Open ID Connect (OIDC).

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

The new support for Open ID Connect (OIDC) aims to extend earlier improvements. The OAuth Authentication via SASL/OAUTHBEARER addition introduced a framework that allowed for integration with OAuth-compliant providers. With this framework in place, Kafka clients can pass a JWT access token to a broker when initializing the connection as a means of authentication, so Kafka can use these standards for authorization and authentication. The improvement provides a concrete implementation of the interfaces to allow Kafka to connect to an Open ID identity provider for authentication and token retrieval.

Elsewhere, this release adds support for infinite endpoints for range queries in Kafka Streams KV stores. This extends the semantics of the existing range and reverseRange interfaces in the ReadOnlyKeyValueStore to support unbounded ranges. Specifically, the interfaces now support the use of null values as a way to represent unbounded ranges.

The new version also introduces two new metrics that are exposed by both the ZooKeeper and KRaft controller. ActiveBrokerCount and FencedBrokerCount expose the number of active brokers in the cluster known by the controller and the number of fenced brokers known by the controller.

RocksDB support has been improved in two areas. Firstly, there's no longer any need to set WAL-related configs for RocksDB instances within Streams as Streams disables the write-ahead log (WAL) provided by RocksDB since it replicates the data in changelog topics. Secondly, the RocksDBStoreTest and RocksDBKeyValueStoreTest classes have been consolidated.

Support has also been added for Java 17.

Kafka 3.1 is available now. 

More Information

Kafka Website

Related Articles

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

Apache Kafka 2.7 Updates Broker

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info