Why do I get https://www.google.com/accounts/DisplayUnlockCaptcha when using remote_api

[Tim Hoffman]

Hi

I am pushing about 20,000 entities from Plone to Appengine via the
remote_api
and every so often (200-300 entities (my guess is it is exactly 200) I
start getting reponses from the remote_api
that say

Please go to
https://www.google.com/accounts/DisplayUnlockCaptcha
and verify you are a human. Then try again.
2009-11-17 13:47:26 INFO root Exception sending Rollback:
Traceback (most recent call last):
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/datastore.py", line 1989, in
RunInTransactionCustomRetries
tx.handle, resp)
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/apiproxy_stub_map.py", line 72,
in MakeSyncCall
apiproxy.MakeSyncCall(service, call, request, response)
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/apiproxy_stub_map.py", line 266,
in MakeSyncCall
rpc.CheckSuccess()
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/apiproxy_rpc.py", line 111, in
CheckSuccess
raise self.exception
NameError: global name 'txdata' is not defined

Surley repeated use of the remote_api shouldn't make our google
overlords think that I am some evil spamming robot. (What I am doing
is robot like, but it is authenticated and it is my application and I
am not spamming anyone ;-.)

Is there some sort of limit to the number of calls that can be made
before we have to reconnect or create a new remote_api connection ?
As obviously something is triggering this reponse from google.

Rgds

Tim

[Matthew Blain]

Hi Tim,
Can you tell me the appid so we can look into this further?
Note that the transaction error is interesting--are you seeing lots of
transaction failures and thus retrying the same call repeatedly?

--Matthew

On Nov 17, 5:54 am, Tim Hoffman <zutes...@gmail.com> wrote:
> Hi
>
> I am pushing about 20,000 entities from Plone to Appengine via the
> remote_api
> and every so often (200-300 entities (my guess is it is exactly 200) I
> start getting reponses from the remote_api
> that say
>

> Please go tohttps://www.google.com/accounts/DisplayUnlockCaptcha

[Tim Hoffman]

Hi Mathew

Appid sent in mail. (Need to keep the site relatively unpublic for
about 1 more week)

Here is some additional information.

I encountered this problem last week - see this thread
http://groups.google.com.au/group/google-appengine/browse_thread/thread/47d0c685c802e8d7/27567828edc5884c?lnk=gst&q=Tim+Hoffman#27567828edc5884c

I found then that I would hit the barrier at exactly 200 puts

In this round of tests (will be our prod instance) I was doing the
transactions a little different and pushing via from two seperate
connections from the same ec2 instance
(rather than from my home connection.)

It appears that again I was hitting the problem and a combined total
(across the 2 connections ) of 200 puts (not 100% certain on this as
I wasn't
counting puts because some of the records had lots of puts - and I was
tracking documents not puts).

As I mentioned in my email I am not retrying transactions, and there
is no point becuase the minute I get this particular error I can't get
anything else through to appengine over the remote_api
(not even a get)

Rgds

Tim

[Tim Hoffman]

I am still getting this problem.

One of the things I have noticed is the errors (302) are being
propogated by google infrastructure and not recorded any way
in the apps logs.

My gut feel is that some other google infrastructure thinks I am
doing something I shouldn't requires me to re-authenticate but
the remote_api proxies can't deal with a 302 inside a Namerror in
CheckSuccess see stack below.

Obviously what I am doing is different to the bulkloader client.

Anyone got any suggestions on how I might go about getting to the
bottom of this.

T

2009-12-02T01:55:49 INFO root Exception sending Rollback:

Traceback (most recent call last):
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/datastore.py", line 1989, in
RunInTransactionCustomRetries
tx.handle, resp)
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/apiproxy_stub_map.py", line 72,
in MakeSyncCall
apiproxy.MakeSyncCall(service, call, request, response)
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/apiproxy_stub_map.py", line 266,
in MakeSyncCall
rpc.CheckSuccess()
File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/
google_appengine/google/appengine/api/apiproxy_rpc.py", line 111, in
CheckSuccess
raise self.exception
NameError: global name 'txdata' is not defined

------
2009-12-02T01:55:49 ERROR Zope.SiteErrorLog
http://10.8.0.134:49081/swan/portal_skins/custom/send_uos
Traceback (innermost last):
Module ZPublisher.Publish, line 119, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 42, in call_object
Module Shared.DC.Scripts.Bindings, line 313, in __call__
Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
Module Products.PythonScripts.PythonScript, line 328, in _exec
Module None, line 25, in send_uos
- <PythonScript at /swan/portal_skins/custom/send_uos>
- Line 25
Module None, line 11, in push
- <PythonScript at /swan/portal_skins/custom/send_uos>
- Line 11
Module Products.PlonePSC.tools.psc_manager, line 639, in
push_to_appengine
Module google.appengine.ext.db, line 1064, in get_or_insert
Module google.appengine.api.datastore, line 1884, in
RunInTransaction
Module google.appengine.api.datastore, line 1982, in
RunInTransactionCustomRetries
Module google.appengine.ext.db, line 1059, in txn
Module google.appengine.ext.db, line 981, in get_by_key_name
Module google.appengine.ext.db, line 1180, in get
Module google.appengine.api.datastore, line 234, in Get
Module google.appengine.api.apiproxy_stub_map, line 72, in
MakeSyncCall
Module google.appengine.api.apiproxy_stub_map, line 266, in
MakeSyncCall
Module google.appengine.api.apiproxy_rpc, line 111, in CheckSuccess
HTTPError: HTTP Error 302: Found

On Nov 18, 7:14 am, Tim Hoffman <zutes...@gmail.com> wrote:
> Hi Mathew
>
> Appid sent in mail.  (Need to keep the site relatively unpublic for
> about 1 more week)
>
> Here is some additional information.
>

> I encountered this problem last week - see this threadhttp://groups.google.com.au/group/google-appengine/browse_thread/thre...

[Tim Hoffman]

Something I have observerd and can now prove repeatably

Once I get the error 302 inside the remote_api call. It takes
somewhere between 2 and 10 mins before I can resubmit the transaction.
During that that time, if I try the same call from another process (on
the same box with the same auth details) the call still gets the error
until such time as something in google lets it go.

It really is looking like I am being clobbered by some google watch
dog.

T

> 2009-12-02T01:55:49 ERROR Zope.SiteErrorLoghttp://10.8.0.134:49081/swan/portal_skins/custom/send_uos

[Tim Hoffman]

OK

Started debugging inside the apiproxy_rpc.py(113)CheckSuccess()
and what I believe is happening is correct.

the rpc call MaekSyncCall is getting a redirect to a request to login

apiproxy_rpc.py(113)CheckSuccess()

The exception is a HTTPErrror which is a redirect (302)
to

'https://www.google.com/a/polytechnic.wa.edu.au/ServiceLogin?
service=ah&passive=true&continue=http://psc-prod1.appspot.com/_ah/login
%3Fcon.... stuff deleted',

Obviously requesting me to log in again.

So somewhere I am running afoul of higher layers in google that watch
what is going on.
I am going to need some help from google engineers to get to the
bottom of this one.

Rgds

Tim Hoffman

[Tim Hoffman]

Ok

Got the the bottom of it all.

You need to re-use connections. If you make too many authenticated
connections then googles higher up infrastructure
shuts you down for about 10 min.

It does highlight some problems in the remote_api code as the error
messages passed back are particularly informative unless you really
dig. but at least I understand where the problem is.

T

[Ikai L (Google)]

Tim, great find. What do you mean re-use connections? If you let me know, I'll update the docs and open an issue to provide a better error message.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Ikai Lan
Developer Programs Engineer, Google App Engine

[Tim Hoffman]

Hi Ikai


I am using remote_api called from within plone.

Because I could potentially get multiple concurrent calls from plone
to push content I was creating a new connection
(requiring new authentication ) on each request to push content from
plone to app engine.

Unfortunately when I started pushing large amounts of content each
content entity being pushed was creating a new
connection (invoking authentication). When I hit 200 of these in the
space of say 10mins I would get hit by google infrastructure
which would basically stop transactions. CheckSuccess didn't like
getting a redirect to the authentication stuff.
And I couldn't re-authenticate for about 10 mins anyway. Looks like I
was running afoul of systems designed to prevent
spamming etc.... Any other connection attempts from the same host
using that user id would also fail.
And none of this would be seen in the app engine logs, because the
remote_api auth/connection was being redirected
even before we got to appengine.

I found that if I cached the connection and didn't continually re-
authenticate I had no problems.

Hope that makes sense.

Rgds

Tim

On Dec 3, 3:35 am, "Ikai L (Google)" <ika...@google.com> wrote:
> Tim, great find. What do you mean re-use connections? If you let me know,
> I'll update the docs and open an issue to provide a better error message.
>

> ...
>
> read more »

[Ikai L (Google)]

Tim,

Ah, great find. If I'm to understand you correctly, you were using a different process off App Engine to log into App Engine, hitting the login page N times in a span of M minutes. This resulted in you not being able to authenticate anymore, but after you cached the auth cookie, it started working fine.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Tim Hoffman]

Hi Ikai

Not exactly

I am using this to connect to the remote_api

def connect(appid,auth_func,host):
remote_api_stub.ConfigureRemoteDatastore(appid, '/remote_api',
auth_func, host)

Its seems you never get a handle back on which you make the call, it
just hooks up the remote data store under the hood to all the
db api methods.

Each time I was starting a new set of pushes connect was being called.
I now track if I have called ConfigureRemoteDatastore and don't call
it again.

That way the current connection is re-used.

Regards

Tim

On Dec 4, 2:06 am, "Ikai L (Google)" <ika...@google.com> wrote:
> Tim,
>

> Ah, great find. If I'm to understand you correctly, you were using a
> different process off App Engine to log into App Engine, hitting the login
> page N times in a span of M minutes. This resulted in you not being able to
> authenticate anymore, but after you cached the auth cookie, it started
> working fine.
>

> On Thu, Dec 3, 2009 at 3:01 AM, Tim Hoffman <zutes...@gmail.com> wrote:
> > Hi Ikai