Error on RetrieveAllUsers

[Ray]

For the past few days my apps sync script has been failing with the
error message below. IncompleteRead error from httplib.py. If
RetrieveAllUsers happens to work, I'll get the same error from
RetrieveAllNicknames as well.

Running v2.0.1 of the gdata python client. While looking though code,
I see around line 390 of
/usr/lib/python2.4/site-packages/gdata/apps/service.py :

def RetrieveAllUsers(self):
"""Retrieve all users in this domain. OBSOLETE"""

I wasn't aware that this function was being obsoleted, and can't find
any docs supporting this on the web. Just wondering if anyone can
shed any light on this, and on the error I'm getting. If you can
suggest any workarounds, that'd be great! We are a midsized
University with about 35,000 accounts.

thanks,
ray

---------------------------------------------
Traceback (most recent call last):
File "/usr/local/bin/gass-ray.py", line 47, in ?
users_feed = service.RetrieveAllUsers()
File "/usr/lib/python2.4/site-packages/gdata/apps/service.py", line
396, in RetrieveAllUsers
return self.AddAllElementsFromAllPages(
File "/usr/lib/python2.4/site-packages/gdata/apps/service.py", line
120, in AddAllElementsFromAllPages
next_feed = self.Get(next.href, converter=func)
File "/usr/lib/python2.4/site-packages/gdata/service.py", line 1013,
in Get
result_body = server_response.read()
File "/usr/lib/python2.4/httplib.py", line 460, in read
return self._read_chunked(amt)
File "/usr/lib/python2.4/httplib.py", line 509, in _read_chunked
value += self._safe_read(chunk_left)
File "/usr/lib/python2.4/httplib.py", line 557, in _safe_read
raise IncompleteRead(s)
httplib.IncompleteRead: ["ogle.com/g/2005#kind' term='http://
schemas.google.com/apps/2006#user'/><title type='text'> -- snip --

[Alex Elder]

Hi Ray,

I've had a look around and I can see that the method was made
obsolete, however, it looks like it was changed somewhere around April
2008, I think! Here's where it was changed:

$> svn diff -r 357:358 | grep RetrieveAllUsers -A4
def RetrieveAllUsers(self):
- """Retrieve all users in this domain."""
+ """Retrieve all users in this domain. OBSOLETE"""


Seeing as the wiki and change logs weren't much use, I checked out the
SVN log, which wasn't much help either:

$> svn log -r 357:358

------------------------------------------------------------------------
r357 | api.jscudder | 2008-05-08 19:25:53 +0100 (Thu, 08 May
2008) | 2 lines

Changed version number to 1.0.13


------------------------------------------------------------------------
r358 | api.jscudder | 2008-05-13 02:00:40 +0100 (Tue, 13 May
2008) | 2 lines

Adding generator to iterate over an app feed.


------------------------------------------------------------------------

_SO_, if possible, could you please some of your code so I can see
what you're doing, and hopefully grok more knowledge as to what's
causing this issue?

Cheers,
Alex.

[Ray]

Alex,

Thanks for the reply. The script I wrote is posted here:
https://sites.google.com/a/selu.edu/gmigration/code

It basically queries Google for all users, queries our LDAP server for
all apps users, then compares the two lists. I've since added try/
except blocks around the RetrieveAllUsers and RetrieveAllNickenames
calls. It just catches the errors, emails me, and exits since the
script can't do much if either of these calls fails.

Let me know if you need any more info...

thanks,
ray

[Takashi Matsuo]

Hi ray,

You can use GetGeneratorForAllUsers method instead.
Here is an example:

for page in service.GetGeneratorForAllUsers():
# do something with each page here
# for example:
for user in page.entry:
# do something with each user here
..

Hope this helps.

--
Takashi Matsuo
The father of kay framework

[Ray]

Takashi,

That seems to work great... thanks a bunch for the code. Do you know
if there is a similar work around for
the RetrieveAllNicknames function? Occasionally that one fails as
well.

Also wondering if you can share any info regarding the status/fate of
the RetrieveAllUsers function.

Thanks again, your code snippet really helped!

ray

On Sep 11, 7:11 pm, Takashi Matsuo <matsuo.taka...@gmail.com> wrote:
> Hi ray,
>
> You can use GetGeneratorForAllUsers method instead.
> Here is an example:
>
> for page in service.GetGeneratorForAllUsers():
>   # do something with each page here
>   # for example:
>   for user in page.entry:
>     # do something with each user here
>     ..
>
> Hope this helps.
>
> --
> Takashi Matsuo
> The father of kay framework
>

[wbc]

I have (very) similar problems when calling RetrieveAllMembers for a
group. Should there by a generator equiv there too? Do we know what
the problem is/was with the older implementation for bulk user
retrieval that has apparently now been marked as 'obsolete'?

Thanks,
Will.

[Ray]

Will.

I'm not sure how similar the group function for RetrieveAllMembers is,
but I did add a generator for the RetrieveAllNicknames function. On
my Centos system, the file is /usr/lib/python2.4/site-packages/gdata/
apps/service.py. Edit that file, and right under the
RetrievePageOfNicknames function, add this one:

def GetGeneratorForAllNicknames(self):
"""Retrieve a generator for all nicknames in this domain."""
first_page = self.RetrievePageOfNicknames()
return self.GetGeneratorFromLinkFinder(first_page,

gdata.apps.NicknameFeedFromString)


I just modified the GetGeneratorForAllUsers function. It's used the
same way. I started having similar problems with the generator
functions also though. My next step is to try the php or java library
functions, to rule out a problem with something on my network dropping
the http connection.

As to your other question, no one has commented on why the original
function is marked obsolete. I could submit a patch for this new
generator function, but it'd be nice to know what is wrong with the
older implementation.

ray

[Takashi Matsuo]

Hi,

I'm the one who marked RetrieveAllUsers as obsolete originally.
That's because I have noticed this method occasionally resulted into
an error which everyone mentions about. I have not realize the root
cause of this error yet because It is very hard for me to examine.
Another reason is that original RetrieveAllUsers takes very long time
if there are many users on your domain. In the newer implementation,
we can do some jobs with each page of users, so I think the newer one
is better.

Let's talk about a way to handle errors.
In my opinion, the APIs like provisioning API that communicate over
the internet, are tend to raise errors because of various reasons. So
it is a good practice to handle this kind of errors with some
decorators or something like that.

It might be good for us to introduce some retrying mechanism on all
the 'RetrieveAll' kind of methods.
I've attached a diff against rev 840 for trial implementation which
I've never tested well. Also it might be better to implement such a
mechanism on a base class.

I'd like you all to discuss about this trial implementation.

As a note, this retry decorator originally derives from here:
http://wiki.python.org/moin/PythonDecoratorLibrary#Retry

--
Takashi Matsuo
The father of kay framework

[Takashi Matsuo]

Hi,

I've created improved patch. So please use this one instead.
sorry for confusing.

I added GetWithRetry method to gdata.service.GDataService with retry decorator.
So another service can use this method for reliability now.

--
Takashi Matsuo
The father of kay framework

[Ebrahim Kariel]

Hi,

 

This email is not for ekariel. Please fix. Thank you.

 

Kind regards,

 

Ebe