Intermittent failure while fetching Permission data from Google Docs and Google Sites

[Ashish Jhunjhunwala]

Hi All,

    I am trying to fetch Accounts/Groups/Group members as well as Permissions data from Google Docs and Google Sites using Two-Legged OAuth mechanism. The data size of accounts and groups is small hence i am able to fetch this data everytime however the # of permissions are more (order of few thousands) and in this case the API call is able to fetch some data and then throws the below error :

com.google.gdata.util.ServiceException: An unknown error has occurred.

<errors xmlns='http://schemas.google.com/g/2005'>

<error>

<domain>GData</domain>

<code>ServiceException</code>

<internalReason>An unknown error has occurred.</internalReason>

</error>

</errors>

The amount of data which is fetched before the error comes varies randomly. However in some cases the same call works and i am able to fetch the complete set of data.

The above error doesnt tell me anything about the issue. I am using Google Aps Administrative Java APIs for performing these operations.

Is there any limitation on data sizes which can be fetched from Google Apps Apis? Does Google has some checks on data sizes it can return from these APIs?

Thanks

Ashish

[Ashish Jhunjhunwala]

Forgot to add more stacktrace. Adding it now:

Caused by: com.google.gdata.util.ServiceException: An unknown error has occurred.

at com.google.gdata.client.http.HttpGDataRequest.handleErrorResponse(HttpGDataRequest.java:624) ~[gdata-core-1.0.jar:na]

at com.google.gdata.client.http.GoogleGDataRequest.handleErrorResponse(GoogleGDataRequest.java:563) ~[gdata-core-1.0.jar:na]

at com.google.gdata.client.http.HttpGDataRequest.checkResponse(HttpGDataRequest.java:552) ~[gdata-core-1.0.jar:na]

at com.google.gdata.client.http.HttpGDataRequest.execute(HttpGDataRequest.java:530) ~[gdata-core-1.0.jar:na]

at com.google.gdata.client.http.GoogleGDataRequest.execute(GoogleGDataRequest.java:535) ~[gdata-core-1.0.jar:na]

at com.google.gdata.client.Service.getFeed(Service.java:1135) ~[gdata-core-1.0.jar:1.46.0]

at com.google.gdata.client.Service.getFeed(Service.java:998) ~[gdata-core-1.0.jar:1.46.0]

at com.google.gdata.client.GoogleService.getFeed(GoogleService.java:631) ~[gdata-core-1.0.jar:1.46.0]

at com.google.gdata.client.Service.getFeed(Service.java:1017) ~[gdata-core-1.0.jar:1.46.0]

Thanks

Ashish

[Richie Foreman]

Hi Ashish,

I've done a very heavily amount of development with these APIs as well.  Sometimes things do hiccup.  I would really recommend implementing an exponential backoff mechanism around your API calls.

http://en.wikipedia.org/wiki/Exponential_backoff

I'm a python guy, or I'd post a code sample, but -- I typically retry things 5 -7 times, using a simple sleep interval like timems = (((n ^ 2)*1000) + (Random Number Under 1000)).  The random seed at the end will help prevent all of your requests from re-dogpiling, especially if you're doing some threading.

Good luck!

[Ashish Jhunjhunwala]

Hi Richie,

  Thanks for the reply. What i did for fetching data was fetch 1000 records at a time and then sleep for 1-2 mins before sending another requuest for another 1000 records. This has helped me to get data sometimes but i cannot be 100% sure that everytime i run it, i will get data. I will certainly try your approach of retries in my code.

Thanks

Ashish