Apache Iceberg 0.14 has been released with improvements to support for Spark and a common REST catalog client that uses change-based commits to resolve commit conflicts on the server side.

Iceberg is a high-performance format for huge analytic tables of titanic proportions that was originally developed by Netflix. Iceberg was made an opensource Apache incubator project in 2018, and graduated to be a top level project in 2020.

Netflix created Iceberg to provide a way of working with very large datasets and ensure the format would work as reliably and predictably as SQL. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.

SQL commands can be used in Iceberg to merge new data, update existing rows, and perform targeted deletes. Iceberg can eagerly rewrite data files for read performance, or it can use delete deltas for faster updates.

Schema evolution just works, columns can be renamed and reordered, and actions such as adding a column won't bring back "zombie" data. Iceberg supports "hidden partitioning". In other words, it handles the task of producing partition values for rows in a table and skips unnecessary partitions and files automatically. No extra filters are needed for fast queries, and table layout can be updated as data or queries change.

Time-travel enables reproducible queries that use exactly the same table snapshot, or lets users easily examine changes. Version rollback allows users to quickly correct problems by resetting tables to a good state. 

The improvements to the latest version start with several performance improvements for scan planning and Spark queries.  Specifically, Parquet vectorized reads are enabled by default, and ScanBuilder now has SupportsReportStatistics. Spark tables have also been updated to avoid expensive (and inaccurate) size estimations. Support has been added for Spark 3.3, including AS OF syntax for SQL time travel queries. There's also merge-on-read support for MERGE and UPDATE queries in Spark 3.2 or later,

Other improvements include a new common REST catalog client that uses change-based commits to resolve commit conflicts on the service side; and new interfaces for consuming data incrementally (both append and changelog scans). The developers have also added a spec and implementation for Puffin, which is a format for large stats and index blobs, like Theta sketches or bloom filters.

Iceberg 0.14 is available now.

More Information

Apache Iceberg

Related Articles

.NET For Apache Spark Updated

Spark BI Gets Fine Grain Security

Spark Announcements

Apache Hive Adds Support For Set Operations  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info