Apache Arrow 14 has been released with new view data types for columnar formats, and a wide range of other improvements.

Apache Arrow is a development platform for in-memory analytics. It has technologies that enable big data systems to process and move data quickly. Arrow is language independent, can be used for flat and hierarchical data, and the data store is organized for efficient analytic operations. It also provides computational libraries. Languages currently supported are C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust.

The improvements to the new release start with new data types for viewing data in Arrow's columnar format. The developers say the improvements were motivated by recent innovations in DuckDB and Meta's Velox engine. The extra view data types start with 16-byte StringView and BinaryView that provide better buffer reuse and faster "false" string comparisons. There are also new ListView and LargeListView types for more performant "out-of-order" building and processing of lists and better buffer reuse.

A new compute function has been added that can be used to calculate a cumulative mean on numeric data, and several other functions have been improved including rounding, divide, take and filter, with new support for duration inputs. 

A new RPC method was added to allow polling for completion in long-running queries as an alternative to the existing blocking GetFlightInfo call, and an experimental asynchronous GetFlightInfo call was added to the client-side API in C++ and Python. The CMake configuration was also fixed to correctly require linking to Arrow Flight RPC when using Arrow Flight SQL

Elsewhere, in Go, the underlying generated Protobuf code is now exposed for easier low-level integrations with Flight, and in Java, utilities were added to help implement basic Flight SQL services for unit testing.

Apache Arrow 14 is available now.  

More Information

Apache Arrow Website

Related Articles

Apache Arrow 5 Improves Asynchronous Scanner

Apache Arrow 4 Adds New C++ Compute Functions

Apache Arrow Improves C++ Support

Apache Arrow 2 Improves C++ and Rust Support

Apache Arrow Reaches 1.0

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info