Apache has released version 0.3 of Iceberg-rust, an official Rust implementation of the Iceberg high-performance format for huge analytic tables.

Iceberg was originally developed by Netflix, was made an opensource Apache incubator project in 2018, and graduated to be a top level project in 2020.

Iceberg is widely used and is supported by nearly all query engines. Users can write SQL to query data from an Iceberg table stored at object storage services without ingesting it into a database first.

Iceberg Rust 0.3 is built and tested with stable rust, and will keep a rolling MSRV(minimum supported rust version).  This release is being used by Databend, an open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake; and iceberg-catalog, a Rust implementation of the Iceberg REST Catalog specification.

This release is described by the developers as including most of the functionality users need to read and write an Iceberg table, though currently only the REST catalog has been implemented. Other catalogs are under active development.

The main changes to this version start with the addition of Sync and Send to the Catalog trait. Send and Sync are fundamental to Rust's concurrency handling, and are used to work out whether a type can be sent to or shared with other threads. 

This release has also implemented OAuth for the catalog's REST client. OAuth is an open-standard authorization protocol that applications can use to provide secure designated access.

There's also new Parquet writer and reader capabilities with support for data projection, and memory catalog and memory file IO support. A SQL Catalog has been established and there's now a subproject for this.

Support has been added for Google Cloud Storage (GCS) and AWS session tokens to ensure Iceberg-rust can be used with GCS and AWS sessions.

Iceberg-rust's concurrency support has been enhanced with the implementation of concurrent table scans and data file fetching, and there are enhanced predicate builders and expression evaluators. The developers have also added support for timestamp columns in row filters.

The contributors say that plans for the next release include the integration of native support for DataFusion, and the ability to connect other query engines with Iceberg, such as Databend.

Iceberg-rust 0.30 is available now.

More Information

Apache Iceberg

Related Articles

Snowflake Support For Apache Iceberg Goes GA

Apache Iceberg Improves Spark Support

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info