Google BigQuery Updates: Increased Result Set Size, Google API Console, GZIP Compression, New PHP Client (Feb 16th, 2012)

[Michael Manoochehri]

Hi Google BigQuery Developers:

Thanks to your feedback as Limited Preview users, we've been able to make continual improvements to Google BigQuery. We have the following updates to share this week:

1. We've increased the maximum response size of your query jobs by an order of magnitude!

One of our most commonly requested features has been for an increase to the maximum size of your query job result sets. We've recently deployed an update that increases the limit of your query response sizes to over 10 times the previous limit. Now, you should be able to retrieve query result sets of at least 64Mb. Technical note: since the result set limit actually refers to the query result size compressed on our end, in many cases the actual result set limit will be even greater!

2. The BigQuery API is now available as a service in the Google API Console

BigQuery is now available as a service in the Google API Console. Any of your current Google API projects that contain BigQuery tables will have this service automatically activated soon (see screenshot below for what this looks like). You'll be asked to confirm the BigQuery terms of service the next time you visit the project in the API Console.

3. New Google PHP Client Library for BigQuery

A good number of BigQuery Limited Preview partners mentioned that they were interested in developing BigQuery applications using PHP. Thanks to Chirag and other maintainers of the Google API Client Library for PHP, we've added a BigQuery-specific PHP client (check out the contrib directory). Download the latest release here. We'll soon be updating our BigQuery documentation with code snippets for each API method in various languages, including Java, Python, and PHP. Feel free to share code snippets in this forum as well!

4. Update on support for gzip compressed files

I've been testing BigQuery's new ability to ingest gzip-compressed CSV files, and I've found that it works very well. I want to share an actual example: I have a test dataset totaling about 350 Gb (in various 1 Gb to 2 Gb sized CSV files). Originally, I loaded these files into BigQuery in 4 separate batches. However, with the gzip feature, I was able to merge the original files together into larger individual ones (about 10 Gb to 15 Gb each), and then compress these larger files to about 20% or 30% of their original size... meaning I could actually ingest the entire 350 Gb dataset into BigQuery in a single batch load job. If you've also tested this feature, we would love to hear more about your experience!

Thanks!

Michael Manoochehri
Google BigQuery Team