Error 401 when attempting to bulk download with appcfg.py

[solsTiCe d'Hiver]

hi.

Like I used to do it several years ago, I am attempting to bulk download the entities of the datastore of my app with appcfg.py (from latest python SDK)

I am using this little script:

#!/bin/bash
date=`date +%Y%m%d`
appid=myappid
url="https://$appid.appspot.com/_ah/remote_api"
GOOGLE_APPLICATION_CREDENTIALS=myapp.json appcfg.py download_data -v -e myemail@gmail.com --application=s~$appid --url=$url --filename=allentities-$date.sql3
rm -f bulkloader-*

because this command does not ask for an oauth token, I have previously run the command

appcfg.py list_versions

just to get the oauth2 token

And as said in https://developers.google.com/accounts/docs/application-default-credentials I have generated a json file with the google credentials and use it with GOOGLE_APPLICATION_CREDENTIALS variable as shown above.

However when I run the script I get a long list of

2016-04-17 19:21:29,316 INFO client.py:546 Attempting refresh to obtain initial access_token
2016-04-17 19:21:29,378 INFO client.py:804 Refreshing access_token
2016-04-17 19:21:30,901 INFO client.py:578 Refreshing due to a 401 (attempt 1/2)
2016-04-17 19:21:30,988 INFO client.py:804 Refreshing access_token

2016-04-17 19:21:31,570 INFO client.py:578 Refreshing due to a 401 (attempt 2/2) 

2016-04-17 19:21:31,659 INFO client.py:804 Refreshing access_token 

2016-04-17 19:21:32,436 INFO client.py:578 Refreshing due to a 401 (attempt 1/2) 

2016-04-17 19:21:32,526 INFO client.py:804 Refreshing access_token 

2016-04-17 19:21:33,162 INFO client.py:578 Refreshing due to a 401 (attempt 2/2) 

[...]

and this ends with 

Error 401: --- begin server output ---
You must be logged in as an administrator to access this.
--- end server output ---

I have the same 401 error in the log of my app.

What could be wrong ?

I am the owner of the app and my email is listed as such in the logins

I don't want to enable billing and use the Google Cloud Platform to export entities from the datastore with "Admin Datastore" (by the way clicking the button "Open Admin Datastore" does nothing, one need to use a right click and open in new tab to get something)

[Nick (Cloud Platform Support)]

Hey solsTiCe d'Hiver,

This appears to follow the symptoms of an issue some other users have observed.

Other than this, I can think of some potentially fruitful questions to ask:

* which oauth2 token are you using now, is it based on the default credentials of a service account? Does that service account have OWNER permission on your app?

* if not, how are you exactly generating the credentials json file?

* does deployment work if you remove 's~' from the app id in the bash script? 

With this information, we'll be much closer to diagnosing the exact nature of this issue, and determining what solutions to provide. 

Best wishes,

Nick
Cloud Platform Community Support

[solsTiCe d'Hiver]

hi nick,

- the oauth2 token is associated with an account which is owner of the app.
- to generate the credentials file I followed the instructions in the url I gave. It seems both the oath2 token and the credentials is needed ? or the oath2 is ignored ?
-I have not s~ in my bash script.

I am trying to do that on the real server not the local dev server.

[Nick (Cloud Platform Support)]

Hey solsTiCe,

Unfortunately it's not quite clear to me which instructions you're following. Also, it seems strange that you'd need both credentials and a token. Could you be a little more specific about exactly what steps you're following to build "myapp.json"? 

Additionally, I can see in your bash script that you use "application=s~$appid", could you attempt a test without the '~s'?

Thanks for clarifying,

Nick
Cloud Platform Community Support

[solsTiCe d'Hiver]

hi. sorry

So to make thing a little clearer I hope:

I follow the standard procedure using appcfg.py using an oauth 2 token. that's when I got that error message

2016-07-01 17:55:25,147 INFO client.py:981 Timeout attempting to reach GCE metadata service. 
Traceback (most recent call last):
  File "/home/solstice/Applications/google_appengine/appcfg.py", line 133, in <module>
    run_file(__file__, globals())
  File "/home/solstice/Applications/google_appengine/appcfg.py", line 129, in run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py", line 5495, in <module>
    main(sys.argv)
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py", line 5486, in main
    result = AppCfgApp(argv).Run()
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py", line 2964, in Run
    self.action(self)
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py", line 5142, in __call__
    return method()
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py", line 4924, in PerformDownload
    run_fn(args)
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/appcfg.py", line 4829, in RunBulkloader
    sys.exit(bulkloader.Run(arg_dict, self._GetOAuth2Parameters()))
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py", line 4349, in Run
    return _PerformBulkload(arg_dict, oauth2_parameters)
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py", line 4242, in _PerformBulkload
    return_code = app.Run()
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py", line 3408, in Run
    oauth2_parameters=self.oauth2_parameters)
  File "/home/solstice/Applications/google_appengine/google/appengine/tools/bulkloader.py", line 1280, in __init__
    rpc_server_factory=throttled_rpc_server_factory)
  File "/home/solstice/Applications/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 747, in ConfigureRemoteApiForOAuth
    credentials = client.GoogleCredentials.get_application_default()
  File "/home/solstice/Applications/google_appengine/lib/oauth2client/oauth2client/client.py", line 1211, in get_application_default
    return GoogleCredentials._get_implicit_credentials()
  File "/home/solstice/Applications/google_appengine/lib/oauth2client/oauth2client/client.py", line 1201, in _get_implicit_credentials
    raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

So I don't understand why the oauth token does not work and why I am asked to use another method aka the GOOGLE_APPLICATION_CREDENTIALS file. Anyway...

As prompted, I go to https://developers.google.com/accounts/docs/application-default-credentials and follow the procedure from the "How the Application Default Credentials work" paragraph.

Then, I use the .json as second auth method as asked as the first oauth does not work. Therefore, I don't get the exception but get a 401 auth error from the site.

and finally, I have tried without the s~ and it's still the same 401 error.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/p9Jx-b5ckx0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/a17043c9-1219-4274-9db4-b77938e6e9a2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nick (Cloud Platform Support)]

Hey solsTiCe,

Could you perhaps try adding the relevant service account as an App Engine Administrator and Project Owner in your Developers' Console "IAM" section?

2016-04-19 16:40 GMT+02:00 'Nick (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com>:

Hey solsTiCe,

Unfortunately it's not quite clear to me which instructions you're following. Also, it seems strange that you'd need both credentials and a token. Could you be a little more specific about exactly what steps you're following to build "myapp.json"? 

Additionally, I can see in your bash script that you use "application=s~$appid", could you attempt a test without the '~s'?

Thanks for clarifying,

Nick
Cloud Platform Community Support

On Tuesday, April 19, 2016 at 6:31:54 AM UTC-4, solsTiCe d'Hiver wrote:

hi nick,

- the oauth2 token is associated with an account which is owner of the app.
- to generate the credentials file I followed the instructions in the url I gave. It seems both the oath2 token and the credentials is needed ? or the oath2 is ignored ?
-I have not s~ in my bash script.

I am trying to do that on the real server not the local dev server.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/p9Jx-b5ckx0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

[Bemmu]

I had the same problem and just got it to work.

As the OP mentioned, I could not manage to get it to work following the steps suggested by the appcfg.py error message (oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.)

Getting the key by those instructions did not work (I tried the app engine key when prompted to select whether I wanted compute engine or app engine key, maybe I chose wrong).

I wrote the solution up as a blogspot with the relevant screenshot here: Backing up data from App Engine with appcfg.py

Anyway, steps I followed to get it to work:

1. Go to IAM
   
2. Click the three balls on the right side of "Compute Engine default service account"
3. "Create key"
4. JSON
5. Save the key compute-key.json

Then create a script called "backup":

GOOGLE_APPLICATION_CREDENTIALS=compute-key.json appcfg.py download_data . --filename=data_store_backup

Instead of . you can also put the full path to the directory containing app.yaml if running from somewhere else.

If you run into "HTTPError: HTTP Error 404: Not Found" when doing download_data, your appcfg.py is so old that is hitting a deprecated endpoint. Update your appcfg.py and try again.