bulk_upload to GAE development datastore no longer works?

[David Manns]

My application runs on GAE. Using Google's bulk loader I make occasional backups of the database. It used to be that I could use the GAE SDK (appcfg tool in this case) to upload all the data into the development server on my development machine, thus creating the full data context for the SDK.

This now fails. I'm not sure (or if) how the remote_api validates the provided email/password, but the results are the same failure with all sensible choices of password, e.g. the primary google account password (the account owns the app on GAE), the application specific password used for the bulk download for GAE (the gmail account uses 2 factor authentication), or no password.

I attach the script used to run the development upload (which used to work until recently) and the command window output from running it. Also the app.yaml file.

Anyone have any idea what the problem might be?

[Christian Foster Howes]

this is a documented GAE SDK bug.  you'll have to downgrade your GAE SDK to the last working version of this.  i *think* the last working version is 1.8.1

[David Manns]

Do you have a reference for this? I went back to 1.8.0 and problem is same. Also tried some 1.7x versions, higher versions with same result. Early 1.7x versions I think ran only on Python 2.5 and probably because of other changes in the yaml file and the --use_sqlite app parameter can't get these to run at all.

Thanks!

[Christian Foster Howes]

ok, it must be a different error. this is the bulkupload error that i
am watching: https://code.google.com/p/googleappengine/issues/detail?id=9666

there are several issues logged against the bulkuploader though:
https://code.google.com/p/googleappengine/issues/list?can=2&q=bulkloader&colspec=ID+Type+Component+Status+Stars+Summary+Language+Priority+Owner+Log&cells=tiles

[David Manns]

Here is the solution. The trick that is not well documented relates to how the new GAE SDK opens service ports.

After starting the application using the launcher the GAE SDK log shows:

2013-12-23 08:32:28 Running command: "['C:\\Python27\\pythonw.exe', 'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', '--skip_sdk_update_check=yes', '--port=12080', '--admin_port=8004', '--datastore_path=C:\\Users\\David\\Google Drive\\ocsnedb backups and test data\\ocsnedb.dat', 'C:\\Users\\David\\Google Drive\\My Documents\\ocsnedb31']"
INFO     2013-12-23 08:32:30,825 devappserver2.py:660] Skipping SDK update check.
INFO     2013-12-23 08:32:30,871 api_server.py:138] Starting API server at: http://localhost:52945
INFO     2013-12-23 08:32:30,871 dispatcher.py:171] Starting module "default" running at: http://localhost:12080
INFO     2013-12-23 08:32:30,871 admin_server.py:117] Starting admin server at: http://localhost:8004

Note that in this case the API server is opened on port 52945. It appears that each time the SDK is started, a different port number is allocated. This port number must be used in the upload command:

appcfg.py upload_data --num_threads=1 --url=http://localhost:<API server port>/_ah/remote_api --filename=<data to upload>

When appcfg prompts for the email address, hit Enter. It does not prompt for a password.