Google has announced the public preview of BigQuery metastore, a fully managed, unified metadata service that works with multiple engines and supports consistent data governance.

Google describes BigQuery metastore as a highly scalable runtime metadata service that works with multiple engines including BigQuery, Apache Spark, Apache Hive and Apache Flink, and supports the open Apache Iceberg table format.

Google BigQuery is a distributed, serverless SQL engine that provides a way to query petabytes of data. It has built in machine learning, is serverless, and supported by Google Cloud.

The new service is designed so analytics engines can work with and query one copy of the data with a single schema, whether the data is stored in BigQuery storage tables, BigQuery tables for Apache Iceberg, or BigLake external tables.

Google says the BigQuery metastore will serve as a critical component for customers looking to migrate and modernize from legacy data lakes to a modern lakehouse architecture. It is closely integrated with BigQuery, and provides built-in security and governance for user interactions with data.

Google says the key benefits of BigQuery metastore are its cross-engine interoperability, support for open formats and catalogs, built-in governance, and the fact it is fully managed at BigQuery scale.

The cross-engine interoperability means it provides a unified view of all metadata for all data sources in the lakehouse, which enables query processing and DML for data stored in open and proprietary formats across object stores, BigQuery storage, and across analytics runtimes.

The built-in governance comes from the fact it is with the governance capabilities provided in BigQuery, such as automated cataloging and universal search, business metadata, data profiling, data quality, fine-grained access controls, data masking, sharing, data lineage and audit logging.

Being a serverless, fully managed service, BigQuery metastore has integrations with engines including BigQuery, Spark, Hive and Flink.

Google BigQuery Metastore is available in preview now.

More Information

Google BigQuery

Related Articles

Google BigQuery Updated

BigQuery Updated and Repriced

Google BigQuery Updated

BigQuery Now Open to All

Google BigQuery Service

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info