Apache Arrow 5 Improves Asynchronous Scanner
Written by Kay Ewbank   
Monday, 16 August 2021

Apache Arrow 5 has been released, alongside Apache Arrow Rust 5. Both versions have a number of improvements, including a better asynchronous scanner for the Dataset layer. This is the first release where the Rust projects have moved to separate repositories outside the main Arrow monorepo.

Apache Arrow is a development platform for in-memory analytics. It has technologies that enable big data systems to process and move data fast..It is language independent, can be used for flat and hierarchical data, and the data store is organized for efficient analytic operations. It also provides computational libraries. Languages currently supported are C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust.

arrow

The improvements to the Dataset layer start with the asynchronous scanner introduced in Arrow 4.  This has been improved with truly asynchronous readers implemented for CSV, Parquet, and IPC file formats and file-level parallelism added.

The compute layer has lots of new scalar functions, including 30 new scalar arithmetic and math functions, a collection of scalar bitwise functions, 21 scalar string functions, 16 scalar temporal functions and a group of 'other' scalar functions such as case_when, coalesce, if_else, and make_struct.

The Flight support has been improved in Arrow's Go implementation, and now supports custom metadata and middleware.

Java improvements include Improved support for extension types using a complex storage type, e.g. struct, map or union.

Python support has been extended, with the ability to scan files asynchronously in Datasets. The developers say this should provide better performance in environments where I/O can be slow, such as with remote sources.

The developers working on the R support in Arrow say they've more than doubled the number of functions you can call on Arrow Datasets inside dplyr::filter(), mutate(), and arrange(), including many more string, datetime, and math functions.The support for the Arrow C interface has been deepened. This allows integration with other projects, such as DuckDB.

Apache Arrow 5 is available now.

arrow 

More Information

Apache Arrow Website

Arrow On GitHub

Related Articles

Apache Arrow 4 Adds New C++ Compute Functions

Apache Arrow Improves C++ Support

Apache Arrow 2 Improves C++ and Rust Support

Apache Arrow Reaches 1.0

Apache Arrow Flight Released

Apache Arrow Adds DataFusion Rust-Native Engine

Apache Arrow Adds Streaming Binary Format

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Apache Drill 1.19 Milestone Release Adds Cassandra Connector
09/09/2021

Apache Drill has been updated in what the developers are calling its biggest release ever. Version 1.19 adds new connectors for Apache Cassandra, Elasticsearch, and Splunk, along with Avro support for [ ... ]



Intel Ditches RealSense
23/08/2021

Intel has taken the decision to "wind down" its RealSense vision division. As well as signalling the end of its range of the high-tech cameras and sensors, this also means the end of the road for Inte [ ... ]


More News

square

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 16 August 2021 )