RetrievePageOfUsers - ApplicationError: 2 The read operation timed out

[Dan Lines]

I am using the Python Library.  I am getting this message - ApplicationError: 2 The read operation timed out when trying to retrieve a page of 100 users.  Never seen this before but it now happens every time.  

Anyone run into this?

- Dan

[Claudio Cherubino]

Hi Dan,

Can you show us your code?

Have you tried performing the same requests with cURL to check whether the problem is in the library?

Also, which version of the Python client library are you using?

Thanks

Claudio

- Dan

--
You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/1WjG6DdA4vcJ.
To post to this group, send email to google-app...@googlegroups.com.
To unsubscribe from this group, send email to google-apps-mgmt...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-apps-mgmt-apis?hl=en.

[Dan Lines]

Claudio,

Thank you for responding so fast.  It is really appreciated.  We have not tried cURL but I do have some information.

1) The Python client library that we are using is 2.0.14.  We have also tried 2.0.15 and this still does not work.

2) If we provide a 'start_username' parameter to the API call it works!

3) When we do not provide a 'start_username' parameter to the API call it does not work.

I am thinking it has something to do with the first set of usernames that are returned?

I will be meeting with our customers tomorrow to look at some of their users to see if there is something special about them.

If you need anymore information please let me know.

- Dan

--
Dan Lines

Customer Support

dli...@cloudlock.com

Office: 781.996.4332 ext 114

http://www.cloudlock.com

[Dan Lines]

Code:

client.RetrievePageOfUsers()

[Claudio Cherubino]

I tried reproducing the issue on my test domain and it worked correctly.

Are you running this on AppEngine? It seems like you may be hitting on of its timeout limits.

Claudio

[Dan Lines]

Claudio,

Yes we are running this on app engine.  It is running in a task on app engine.  But it is not hitting the 10 minute time out limit or anything like that.  

I'm thinking that it has to do with the data that is being returned in the first set of 100 users returned from 'RetreivePageOfUsers()'.  If I pass the character 'a' into this method then the call works: client.RetrievePageOfUsers(start_username='a').  One other piece of info is that many of the user emails in this domain are structured like 11.dan.lines at domain.com (numbers then letters).

If you need anymore info please let know.  We are currently still working with the customer to try to gather all of their users.

Thanks,

- Dan

[Claudio Cherubino]

Hi Dan,

If you haven't changed anything in the way the library sends the requests, then it is probably using the default deadline for fetch requests, which is 5 seconds:

http://code.google.com/appengine/docs/python/urlfetch/fetchfunction.html

Claudio

- Dan

--

You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/ZtPzck0oXdIJ.

[Dan Lines]

Claudio,

This kinda makes sense...if the call is timing out.  Although, we have many many domains that we use this API call on and none of them have ever timed out before.  We are using the python client library to make this api call.  How can I change the deadline time using the python library?

- Dan

[Claudio Cherubino]

You have to edit the source of the library at:

http://www.google.com/codesearch#5IMuXQUucS8/src/gdata/urlfetch.py&q=fetch%20package:http://gdata-python-client%5C.googlecode%5C.com&type=cs&l=209

I'd also like to ask you again to perform the same request with cURL so that we can exclude any issues with the library and/or App Engine.

If that works without problems, then we focus on the Python library without App Engine, and I'd like you to try running the same code from a local script (not a local App Engine instance).

Please perform those tests and let us know the results.

Thanks

Claudio

- Dan

--

You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/zIMDSR1xfNsJ.

[Dan Lines]

Claudio,

I changed the gdata library and made the default deadline time = 60.  This made the call work!  I am not sure what cURL is but I'd be happy to look into it and perform any tests that will help you out.  Maybe the fetch just needed some more time to process?

- Dan

[Dan Lines]

Claudio,

Is there a way I can still use the python client library and set the deadline time without having to change the library itself.  Or do I need to build the URL and do a standard URLFetch?

- Dan

[Claudio Cherubino]

Hi Dan,

Unfortunately that parameter is not exposed to the developer so the only way to set it is by editing the library code.

I can easily see this as a valid feature request, so I'd suggest you filing it on our issue tracker:

http://code.google.com/p/gdata-python-client/issues/list

Please describe your use case and perhaps also link this thread, I'm confident your request will be accepted and the library updated accordingly.

FYI, cURL is a command-line tool that can be used to send HTTP requests. 

Follow these steps to send an authenticated request to retrieve a page of users in your domain:

1) Download curl from http://curl.haxx.se/ then run the following command to perform ClientLogin (of course, put your credentials):

curl https://www.google.com/accounts/ClientLogin --request POST -d Email=ad...@domain.com -d Passwd=mypassword -d accountType=HOSTED -d service=apps

2) From the output of the previous command, copy the Auth token.

3) Run the following command to retrieve users from your domain:

curl --header "Authorization: GoogleLogin auth=PUT_THE_AUTH_TOKEN_HERE" "https://apps-apis.google.com/a/feeds/YOURDOMAIN.COM/user/2.0"

Claudio

- Dan

--

You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/nXXV4Rn2vy0J.

[Dan Lines]

Claudio,

Thank you for the information.  I have submitted a ticket at the link you provided.  It is issue 556.  I am still wondering why this call is timing out for a particular domain when I know that it works for hundreds of other domains.  Will the cURL command-line-tool provide any insight into this question?

- Dan

[Claudio Cherubino]

Hi Dan,

Thanks for filing the feature request, I noticed it has already been picked up by a member of the team.

Perhaps that specific domain has more users than the others or the format of the usernames makes our indexing less effective and therefore retrieval slower or any other internal internal implementation details that are not exposed publicly, but cURL won't help us on that. 

You may want to measure the time to retrieve a page of users in that domain and some other domains, if the difference is huge then I'll use those stats to start a deeper investigation on our side.

Thanks

Claudio 

- Dan

--

You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/gh1g_yH972cJ.

[Dan Lines]

Claudio,

I ran a few tests locally to get some numbers together.  Below shows the API call time on my local dev server for a few different sized domains.

<50 users: 0.5 seconds

21,000 users: 0.9 - 1.06 seconds

Problem domain (6,000 users): 8.6 - 10.06 seconds

I guess what is strange about the problem domain is that it only has around 6,000 users.  We have lots of domains larger than this and the call mostly works ok.  Another thing to note is that on the domain with 21,000 we do sometimes see the API call timeout.  When this happens it eventually passes if we keep retrying.

So overall, I'm not sure if the problem is directly related to the amount of users in the domain.  Let me know what you think.

- Dan

[Claudio Cherubino]

Dan,

Please let me know the name of the problem domain, so that we can perform some investigations.

Thanks

Claudio

- Dan

--

You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/_bmLed-lol8J.