Database Error: Invalid Credentials

[Hector Rovira]

I've tried several times, and methods, to create a table from data in
my google storage account. The latest attempt I made followed the
babynames example step-by-step here: http://code.google.com/apis/bigquery/docs/getting-started.html

I've tried this from Mac OS X and Linux with fresh installs of Python
(2.7), gsutil and bq. I've also tried with two separate accounts that
both are part of the same google storage project. One has 2-step
verification and authorization for apps. The other one is just a
google account (already transitioned from google apps).

Both of these accounts have recently been added to the BigQuery v2
testers.

I've stepped through the "bq login --user_email=..." process, and it
seems okay.

I'm getting this error on "bq create" regardless of dataset:
Traceback (most recent call last):
File "/local/python/2.7.2/bin/bq", line 8, in <module>
load_entry_point('bigquery-shell==1.1.0', 'console_scripts', 'bq')
()
File "/local/python/2.7.2/py-lib/bigquery_shell-1.1.0-py2.7.egg/
bq.py", line 296, in run_main
appcommands.Run()
File "/local/python/2.7.2/py-lib/google_apputils-0.2.2-py2.7.egg/
google/apputils/appcommands.py", line 759, in Run
return app.run()
File "/local/python/2.7.2/py-lib/google_apputils-0.2.2-py2.7.egg/
google/apputils/app.py", line 208, in run
return _actual_start()
File "/local/python/2.7.2/py-lib/google_apputils-0.2.2-py2.7.egg/
google/apputils/app.py", line 236, in _actual_start
really_start()
File "/local/python/2.7.2/py-lib/google_apputils-0.2.2-py2.7.egg/
google/apputils/appcommands.py", line 745, in _CommandsStart
sys.exit(command.CommandRun(GetCommandArgv()))
File "/local/python/2.7.2/py-lib/google_apputils-0.2.2-py2.7.egg/
google/apputils/appcommands.py", line 275, in CommandRun
ret = self.Run(argv)
File "/local/python/2.7.2/py-lib/google_apputils-0.2.2-py2.7.egg/
google/apputils/appcommands.py", line 386, in Run
return self._cmd_func(argv)
File "/local/python/2.7.2/py-lib/bigquery_shell-1.1.0-py2.7.egg/
bq.py", line 131, in _Create
_JsonPrinter(_BigQueryClient().CreateTableFromFile(table,
schema_path))
File "/local/python/2.7.2/py-lib/bigquery_shell-1.1.0-py2.7.egg/
bq_client.py", line 183, in CreateTableFromFile
return self.CreateTableFromJson(table_name, schema_json)
File "/local/python/2.7.2/py-lib/bigquery_shell-1.1.0-py2.7.egg/
bq_client.py", line 210, in CreateTableFromJson
params=dict(name=table_name, fields=schema)))
File "/local/python/2.7.2/py-lib/bigquery_shell-1.1.0-py2.7.egg/
bigquery_db_api.py", line 479, in Call
raise DatabaseError('Database error: %s' % message)
bigquery_db_api.DatabaseError: Database error: Invalid Credentials

[Michael Sheldon]

Hello Hector,

There are two issues here.

• First, I suspect your accounts have been signed up for app-specific passwords and you are using your standard password rather than the app-specific password. If this is the case, here's the URL to use to create an app-specific password that will work with bq:

https://www.google.com/accounts/IssuedAuthSubTokens

If that doesn't fix it, let me know, and we can dig in further.

• Second, you mention bigquery v2, but it looks like you are following the bigquery v1 documentation. (We haven't removed that since both versions are currently supported. We're working on some documentation updates that will hopefully make this more clear.)

Please look at https://code.google.com/apis/bigquery/docs/v2/index.html and notice the left side has two sets of documentation links: the bigquery v1 links are on the top, and the bigquery v2 links are on the bottom. Here's the reference link for the new bigquery web ui:  https://code.google.com/apis/bigquery/docs/v2/browser_tool.html. 

Cheers,

--Michael Sheldon

[Hector Rovira]

I did setup the Auth token for my account and verified that it worked (wrong password got a 403) on the "bq" tool.  Looking at the v2 docs helped, thanks!  Turns out the bq tool does not support v2 yet.

Command-line interface - BigQuery V2 will expose a command-line tool similar to the bq tool; however, this tool is not yet available.

I tried loading the example dataset (babynames) through the UI, and that worked today.  Not sure why it didn't yesterday.  Is there any lag between the time you upload to Google Storage and when it is available for upload into BigQuery?

I also suspect that loading is failing because my dataset is too big (2.8GB).  But I'm not getting any error message in the job output, only that it failed to load.  I'll split the data and try loading it again.

[Michael Sheldon]

Hello Hector,

I'm glad things are working better now. Some answers to your questions inline...

--Michael Sheldon

On Wed, Oct 12, 2011 at 1:23 PM, Hector Rovira <hrovi...@gmail.com> wrote:

I did setup the Auth token for my account and verified that it worked (wrong password got a 403) on the "bq" tool.  Looking at the v2 docs helped, thanks!  Turns out the bq tool does not support v2 yet.

Command-line interface - BigQuery V2 will expose a command-line tool similar to the bq tool; however, this tool is not yet available.

I tried loading the example dataset (babynames) through the UI, and that worked today.  Not sure why it didn't yesterday.  Is there any lag between the time you upload to Google Storage and when it is available for upload into BigQuery?

The Google Storage storage model is a consistent one: one you successfully upload a file, it should be available for read immediately.

I also suspect that loading is failing because my dataset is too big (2.8GB).

BigQuery v1 had a 1GB per file import quota limit. We doubled this to 2GB per file. It looks like you are running into this limit.

 But I'm not getting any error message in the job output, only that it failed to load.  I'll split the data and try loading it again.

Thanks for the notice on this. We'll investigate: you should have received an error on "source too large"...

[Hector Rovira]

I split my data into 3 files (<1GB each) and the upload worked (one file, 20 mins).  I still need to figure out how to append data using the REST API.  It would be nice to have the bq tool for this.

The "source too large" error did not show on the UI.  Here's job ID, let me know if you need more info:  job-a173d6b1-ad7e-4560-93b1-803bbb387771

Thanks for your help!

[Hector Rovira]

Hi Michael,

Just following up on our conversation.  We have started implementing a provisional BigQueryV2 client using OAuth2 and the REST API.  This python utility currently only adds functionality not available through the Web UI (e.g. append).  Code is available here:  http://code.google.com/p/simplebq2/

I successfully submitted an append operation to a table, the only weirdness was that when I submitted the job through the REST API it did not show up in the list of pending jobs in the Web UI.

Also, my colleague Adam found this bug in the example code: 

https://code.google.com/apis/bigquery/docs/v2/developers_guide.html#aclsandrights

    http = httplib2.Http()

    http = credentials.authorize(http)

    service = build("bigquery", "v2beta1")

should be:  

    service=build("bigquery", "v2beta1", http)

Let me know if you have any comments or questions,

Hector

[Craig Citro]

Hi Hector,

We've also been implementing a python client for BigQuery v2:
http://code.google.com/p/google-bigquery-tools/
I think it'll handle your use case (note that the 'load' command will
append by default), but let us know if there are any features you'd
like to see in bq.

> Also, my colleague Adam found this bug in the example code:
> https://code.google.com/apis/bigquery/docs/v2/developers_guide.html#aclsandrights
>
>     http = httplib2.Http()
>     http = credentials.authorize(http)
>     service = build("bigquery", "v2beta1")
> should be:
>     service=build("bigquery", "v2beta1", http)
>

Thanks -- I'll make sure we get this updated!

--

-cc

[Hector Rovira]

that was definitely timely!  I'll give it a whirl and let you know.