Google BigQuery Updates: Nested/Repeated records & JSON support, Higher Import Quotas, and more

[Michael Manoochehri]

Hello BigQuery Developers:

Last week, the Google BigQuery engineering team released several new features that expanded support for data formats, increased data loading performance, and more. Here is a summary of our latest features:

Support for Nested/Repeated Fields and JSON import/export
Developers can now load and query data that contains nested and repeated fields. In addition, BigQuery now supports newline-delimited JSON as both an import and an export format. JSON format supports the same standard BigQuery datatypes as CSV, along with the "record," datatype, which indicates a nested JSON object.

To load newline-delimited JSON object data into BigQuery, include the sourceFormat parameter in your load request configuration with the value "NEWLINE_DELIMITED_JSON." Similarly, to extract data from BigQuery in JSON format, set the destinationFormat parameter to "NEWLINE_DELIMITED_JSON."

Read more about importing and exporting data with BigQuery.

Increased Load Job Quota Limits
We have increased the quota limit for a single load job to 1TB. Individual JSON and CSV files without newlines in string fields may be as large as 100GB. Other import quotas, including those for compressed file formats, are listed on our quota policy page.

We've also removed BigQuery's previous 2 imports per minute rate limit. It's now possible to submit all your ingestion jobs and BigQuery will handle data load job queuing as necessary.

Google App Engine Datastore Admin backup to BigQuery: Trusted Tester Program
We've recently launched a new feature which enables you to import data from the experimental Google App Engine Datastore backup tool directly into BigQuery for analysis. We are initially opening access to this feature to a small group of testers. To apply for access, please sign up.

New Best Practices Documentation
We've just published two new best practices guides for getting the most out of BigQuery - a data Ingestion Cookbook and Query Cookbook, with recipes and best practices for loading your data, and developing queries for timestamps, geospatial data, and other advanced use cases.

Get Help Developing with Google BigQuery
We support developers using the Google BigQuery API on Stack Overflow. Google engineers monitor and answer question with the tag google-bigquery. Please use this tag when asking questions.

Thanks!
BigQuery Team