Google BigQuery Updates: Nested/Repeated records & JSON support, Higher Import Quotas, and more
876 views
Skip to first unread message
Michael Manoochehri
unread,
Oct 8, 2012, 5:08:09 PM10/8/12
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to bigquery...@googlegroups.com
Hello BigQuery Developers:
Last week, the Google BigQuery engineering team released several new features that expanded support for data formats, increased data loading performance, and more. Here is a summary of our latest features:
Support for Nested/Repeated Fields and JSON import/export Developers can now load and query data that contains nested and repeated fields. In addition, BigQuery now supports newline-delimited JSON as both an import and an export format. JSON format supports the same standard BigQuery datatypes as CSV, along with the "record," datatype, which indicates a nested JSON object.
To load newline-delimited JSON object data into BigQuery, include the sourceFormat parameter in your load request configuration with the value "NEWLINE_DELIMITED_JSON." Similarly, to extract data from BigQuery in JSON format, set the destinationFormat parameter to "NEWLINE_DELIMITED_JSON."
Increased Load Job Quota Limits We have increased the quota limit for a single load job to 1TB. Individual JSON and CSV files without newlines in string fields may be as large as 100GB. Other import quotas, including those for compressed file formats, are listed on our quota policy page.
We've also removed BigQuery's previous 2 imports per minute rate limit. It's now possible to submit all your ingestion jobs and BigQuery will handle data load job queuing as necessary.
Google App Engine Datastore Admin backup to BigQuery: Trusted Tester Program We've recently launched a new feature which enables you to import data from the experimental Google App Engine Datastore backup tool directly into BigQuery for analysis. We are initially opening access to this feature to a small group of testers. To apply for access, please sign up.
New Best Practices Documentation We've just published two new best practices guides for getting the most out of BigQuery - a data Ingestion Cookbook and Query Cookbook, with recipes and best practices for loading your data, and developing queries for timestamps, geospatial data, and other advanced use cases.
Get Help Developing with Google BigQuery We support developers using the Google BigQuery API on Stack Overflow. Google engineers monitor and answer question with the tag google-bigquery. Please use this tag when asking questions.