bulkloader.py Authentication

[Jeff Walter]

Hello,

I must have mistyped my Google password when I was authenticating for
the bulkloader.py script, because I got an "Authentication Failed"
message.

Now each time I run the script, it doesn't prompt me for my email or
password -- it just says "Authentication Failed." I tried the --email
and --passin parameters, but it won't let me re-authenticate with new
credentials.

How can clear the credentials that the script is remembering so I try
again?

Jeff

[Elias Torres]

Can you post your stacktrace?

-Elias

[Jeff Walter]

Elias,

I'm not getting any stacktrace when I run in debug. Here's what I see:

bulkloader.py --dump --kind=Video_ --url=http://idolornot-
dev.appspot.com/remote_api --filename=videos.csv --passin --debug

[INFO ] Logging to bulkloader-log-20090924.165037
[INFO ] Throttling transfers:
[INFO ] Bandwidth: 250000 bytes/second
[INFO ] HTTP connections: 8/second
[INFO ] Entities inserted/fetched/modified: 20/second
[INFO ] Opening database: bulkloader-progress-20090924.165037.sql3
[INFO ] Opening database: bulkloader-results-20090924.165037.sql3
[DEBUG ] [WorkerThread-0] WorkerThread: started
[DEBUG ] [WorkerThread-1] WorkerThread: started
[DEBUG ] [WorkerThread-2] WorkerThread: started
[DEBUG ] [WorkerThread-3] WorkerThread: started
[DEBUG ] [WorkerThread-4] WorkerThread: started
[DEBUG ] [WorkerThread-5] WorkerThread: started
[DEBUG ] [WorkerThread-6] WorkerThread: started
[DEBUG ] [WorkerThread-7] WorkerThread: started
[DEBUG ] [WorkerThread-8] WorkerThread: started
[DEBUG ] [WorkerThread-9] WorkerThread: started
[DEBUG ] Configuring remote_api. url_path = /remote_api, servername
= idolornot-dev.appspot.com
[DEBUG ] Bulkloader using app_id: idolornot-dev
[INFO ] Connecting to idolornot-dev.appspot.com/remote_api
[INFO ] Authentication Failed

I'm on OS X 10.6 and running the Java SDK

Jeff

[Nick Johnson (Google)]

Hi Jeff,

Instead of calling the bulkloader directly, call "appcfg.py upload_data", and supply the --no_cookies argument to ignore the cookie file.

-Nick Johnson

--
Nick Johnson, Developer Programs Engineer, App Engine
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047

[Jeff Walter]

Hi Nick,

Thanks for the suggestion. I'm not quite at the upload point, I'm
interested right now in pulling down data from the app engine.

Since I am using Java, your advice on Stack Overflow and your blog has
been very helpful.

http://stackoverflow.com/questions/1346632/how-do-i-put-data-into-the-datastore-of-googles-app-engine
http://blog.notdot.net/2009/9/Advanced-Bulk-Loading-Part-5-Bulk-Loading-for-Java

Following the articles, I was able to use bulkloader to download data!

However I would like to get the data in CSV format -- so I turned to
appcfg.py download_data, as per the instructions on:
http://code.google.com/appengine/docs/python/tools/uploadingdata.html

When I run this command I notice two things:

1) I am told --app_id is not a recognized option (which the
documentation leads me to believe is supposed to work)
2) (removing the --app_id) I am told there is a syntax error in my
app.yaml file (but the application deployed just fine)

My app.yaml file is the same as the example you posted on your blog
(except for the app name of course)

My error is posted below.

Is the issue that I also need to define my models for download as I
would need to for upload?

Best,
Jeff

appcfg.py download_data --kind=Video_ --url=http://
upload.latest.idolornot-dev.appspot.com/remote_api --
filename=videos.csv --config_file=../idolornot-dev-empty/app.yaml ../
idolornot-dev-empty/

/usr/local/bin/appcfg.py:41: DeprecationWarning: the sha module is
deprecated; use the hashlib module instead
os.path.join(DIR_PATH, 'lib', 'antlr3'),
Downloading data records.
[INFO ] Logging to bulkloader-log-20090929.132804
Traceback (most recent call last):
File "/usr/local/bin/appcfg.py", line 60, in <module>
run_file(__file__, globals())
File "/usr/local/bin/appcfg.py", line 57, in run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 2525, in <module>
main(sys.argv)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 2516, in main
result = AppCfgApp(argv).Run()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 1640, in Run
self.action(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 2404, in __call__
return method()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 2270, in PerformDownload
run_fn(args)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 2192, in RunBulkloader
sys.exit(bulkloader.Run(arg_dict))
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 3807, in Run
return _PerformBulkload(arg_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 3630, in _PerformBulkload
LoadConfig(config_file)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 3380, in LoadConfig
('', 'r', imp.PY_SOURCE))
File "../idolornot-dev-empty/app.yaml", line 1
application: idolornot-dev
^
SyntaxError: invalid syntax


On Sep 25, 8:23 am, "Nick Johnson (Google)" <nick.john...@google.com>
wrote:

[bgood]

I find myself in a similar circumstance. I would like to use the new
--dump option for the bulkloader to retrieve my data but I can't seem
to get it to authenticate me. Problem exactly as described above.
When I tried to use appcfg rather than calling bulkloader, it didn't
recognize the --dump parameter.

In the doc here
http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Downloading_and_Uploading_All_Data
it suggests that appcfg won't work with dump just yet.

So.. how can I get the bulkloader to authenticate me? I tried
setting the email and passin flags but that seemed to have no effect.

thanks for any advice
-Ben

On Sep 29, 11:17 am, Jeff Walter <jeffinmot...@gmail.com> wrote:
> Hi Nick,
>
> Thanks for the suggestion. I'm not quite at the upload point, I'm
> interested right now in pulling down data from the app engine.
>
> Since I am using Java, your advice on Stack Overflow and your blog has
> been very helpful.
>

> http://stackoverflow.com/questions/1346632/how-do-i-put-data-into-the...http://blog.notdot.net/2009/9/Advanced-Bulk-Loading-Part-5-Bulk-Loadi...
>
> Following the articles, I was able to usebulkloaderto download data!

> > Instead of calling thebulkloaderdirectly, call "appcfg.py upload_data",

> > and supply the --no_cookies argument to ignore the cookie file.
>
> > -Nick Johnson
>
> > On Thu, Sep 24, 2009 at 4:25 PM, Jeff Walter <jeffinmot...@gmail.com> wrote:
>
> > > Hello,
>
> > > I must have mistyped my Google password when I was authenticating for

> > > thebulkloader.py script, because I got an "AuthenticationFailed"

> > > message.
>
> > > Now each time I run the script, it doesn't prompt me for my email or

> > > password -- it just says "AuthenticationFailed." I tried the --email

[bgood]

I found part of my answer in a previous thread
http://groups.google.com/group/google-appengine/browse_thread/thread/f61b11f7c1871ed0#

It seems that the url I was requesting didn't exist - which you can
identify by inspecting the stack trace for the 404.

But the --dump option is failing as follows.

I am trying to dump the data created in a Java app engine
application. So, I have two versions of the app - the live java
version and the python version that hosts /remote_api . (this caused
the url confusion)

I succeeded with authenticating a request to the python remote_api (in
version 'bulkload') with this:

bulkloader.py --dump --filename=my-new-archive.csv --kind=MyClass --
url=http://bulkload.latest.myappid.appspot.com/remote_api ./

however, the request failed with the error:
[ERROR ] Error in Thread-1: app myappid cannot access app
bulkload.latest.myappid's data

My app.yaml file looks like this

application: myappid
version: bulkload
runtime: python
api_version: 1


handlers:
- url: /remote_api
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
login: admin

Any idea what I am doing wrong?
thanks
-ben

[Takashi Matsuo]

Hi bgood,

Perhaps you can try specifying app_id explicitly by adding
"--app-id='yourappid'".
Sorry if it won't work for you.

Regards,

--
Takashi Matsuo
The father of kay framework

[bgood]

Takashi, you rock! Thanks so much, that solved it

On Sep 30, 8:13 pm, Takashi Matsuo <matsuo.taka...@gmail.com> wrote:
> Hi bgood,
>
> Perhaps you can try specifying app_id explicitly by adding
> "--app-id='yourappid'".
> Sorry if it won't work for you.
>
> Regards,
>
> --
> Takashi Matsuo
> The father of kay framework
>
>
>
> On Thu, Oct 1, 2009 at 4:38 AM, bgood <ben.mcgee.g...@gmail.com> wrote:
>
> > I found part of my answer in a previous thread

> >http://groups.google.com/group/google-appengine/browse_thread/thread/...

[Jeff Walter]

Hi everyone,

Well I'm glad bgood was able to get his issue solved! In the meantime,
I'm still having a bit of trouble with appcfg.py and the download_data
function.

To recap:

I followed Nick Johnson's suggestion for Java developers and created a
simple python version of my application for the purpose of using
python tools to interact with my datastore.

I'm trying to download data in csv format from my datastore. I just
upgraded to the latest version of the python tools (1.2.6.542). It's
not working -- I am told there is a syntax error in my app.yaml file.
However I'm able to upload my app fine and use bulkloader.py directly
with success. Here's the exact command I'm using with appcfg.py and a
link to my configuration file. I'm on OSX v10.6.

appcfg.py download_data --kind=Video_ --url=http://
upload.latest.idolornot-dev.appspot.com/remote_api --
filename=videos.csv --config_file=../idolornot-dev-empty/app.yaml ../
idolornot-dev-empty/

http://jeffinmotion.com/work/appengine/appcfg_issue/app.yaml

The error:
Application: idolornot-dev; version: upload.
Downloading data records.
[INFO ] Logging to bulkloader-log-20091015.135351

Traceback (most recent call last):
File "/usr/local/bin/appcfg.py", line 60, in <module>
run_file(__file__, globals())
File "/usr/local/bin/appcfg.py", line 57, in run_file
execfile(script_path, globals_)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/

google/appengine/tools/appcfg.py", line 2548, in <module>
main(sys.argv)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 2539, in main

result = AppCfgApp(argv).Run()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 1640, in Run
self.action(self)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/

google/appengine/tools/appcfg.py", line 2427, in __call__

return method()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/

google/appengine/tools/appcfg.py", line 2293, in PerformDownload
run_fn(args)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/appcfg.py", line 2215, in RunBulkloader
sys.exit(bulkloader.Run(arg_dict))
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 3894, in Run

return _PerformBulkload(arg_dict)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/

google/appengine/tools/bulkloader.py", line 3717, in _PerformBulkload
LoadConfig(config_file)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/
GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/
google/appengine/tools/bulkloader.py", line 3467, in LoadConfig

('', 'r', imp.PY_SOURCE))
File "../idolornot-dev-empty/app.yaml", line 1
application: idolornot-dev
^
SyntaxError: invalid syntax

Any ideas?

Jeff