Ibis 8.0 has been released with stream processing backends. The new release includes Apache Flink as a streaming backend, and RisingWave, a streaming database backend. There's also a new batch backend with Exasol, bringing the total number of backends Ibis supports to 20. 

The Ibis team says that most geospatial operations are now supported in the DuckDB backend, making Ibis a local option for geospatial analytics.

Ibis is a portable Python dataframe library. It provides support for scaling data transformation code in Python and can be used to work with data in multiple data platforms. Ibis has been downloaded more than 10 million times.

Ibis is an open source project and is independently governed. While four out of five members of the steering committee are employed by Voltron Data, it is not owned by Voltron Data. The fifth member of the steering committee is at Alphabet working on Google BigQuery.

Ibis has the ability to work with multiple dataframes, and provides a common API for data manipulation in Python, and compiles that API into the backend's native language. The Ibis team acknowledges that portability with Ibis isn't perfect, but say that commonalities across backends and SQL dialects combined with years of engineering effort produce a full-featured and robust framework for data manipulation in Python.

The new support for stream processing opens up the option of using Ibis for working with high-throughput, low-latency data processing with time semantics for applications such as fraud detection, real-time analytics, and IoT. The inclusion of Apache Flink means Ibis has the most popular open-source stream processing framework, while RisingWave is an open-source Postgres-compatible streaming database with a cloud offering.

With the release of version 8.0, Ibis now supports 20 different query engines, accommodating a wide range of data processing needs from small-scale queries with DuckDB to large, distributed preprocessing/ETL jobs with engines like BigQuery, Spark, Theseus, and more.

The updated list of supported backends includes Google BigQuery, ClickHouse, Dask, DataFusion, Druid, DuckDB, Exasol, Flink, Impala, Microsoft SQL Server, MySQL, Oracle, pandas, Polars, PostgreSQL, PySpark, RisingWave, Snowflake, SQLite and Trino.

Ibis 8 is available now.

More Information

Ibis Website

Related Articles

Google Builds Data Lake On BigQuery

Apache Flink ML 2.0 Released

Apache Flink 1.9 Adds New Query Engine

Apache Flink 1.5.0 Adds Support For Broadcast State

Flink Gets Event-time Streaming

Google BigQuery Updated

Pandas Reaches 1.0

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info