Re: exceeding rate limit issue with GetFriendIDs and potentially other APIs

[bear]

You shouldn't have to make a separate call for the screen names - GetFriends calls the twitter api /friends/list.json which allows for an optional include_user_entities parameter

api.GetFriends(screen_name='bear', cursor=-1, include_user_entities=True)

returns the first 20 entries and included in that will be the cursor marker which you then pass on to the next call.

On Wed, Jun 5, 2013 at 2:46 PM, <allison....@gmail.com> wrote:

I'm currently having a problem with GetFriends and rate limiting. The cursoring doesn't seem to be working properly. I follow about 1000 people, which isn't a ton in the scheme of things I didn't think. I can use the GetFriendIDs method without error and fetch the IDs of all of my friends, but I wanted to get their screennames. I tried getting the IDs and then looking the users up one by one but thats exceeding the limit also. I can't get UsersLookup to work (I get a JSON encode error) and GetFriends gives me a rate limit error every time. What can I do in this instance? Sorry, I'm relatively new to both Python and the Twitter API. If I'm missing something simple I'd appreciate the advice.

Thanks
Allison

On Sunday, May 5, 2013 5:38:34 PM UTC-4, mishra...@gmail.com wrote:

When a twitter user has over 75000 friends (not uncommon it seems), GetFriendIDs iterates through the cursors and tries to fetch them all (twitter allows 5000/call).. and basically hits the rate limit (15 calls/15 minutes) and errors out.

I've modified the code in my installation to not retrieve over 2000 IDs (for my app it's not required) but I think this could be a generic problem and some configuration would help.

Also, this problem is probably applicable to some other APIs.

--
You received this message because you are subscribed to the Google Groups "python-twitter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-twitte...@googlegroups.com.
To post to this group, send email to python-...@googlegroups.com.
Visit this group at http://groups.google.com/group/python-twitter?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Bear

be...@xmpp.org (email)
bea...@gmail.com (xmpp, email)
be...@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111  D770 9331 E822 40B3 CD29

[bear]

your getting a Rate Limit error from Twitter which we handle very badly (IMO - it's on my list of things to improve) because it seems we wrote some code that used to work but now does not :(

the GetFriends() routine loops to try and receive all of the friends which is bad, I'm going to make a change to have a sane default - i'll post the new code here so you can test it

On Wed, Jun 5, 2013 at 3:05 PM, <allison....@gmail.com> wrote:

Here is the code in question that I ran:

users = api.GetFriends(screen_name='musegarden', cursor=-1, include_user_entities=True)

I am sure I authenticated correctly and I copied the code as you wrote it. When I run this it does nothing but give me this error:

Traceback (most recent call last):
  File "pyfollowers.py", line 11, in <module>
    users = api.GetFriends(screen_name='musegarden', cursor=-1, include_user_entities=True)
  File "/usr/local/lib/python2.7/dist-packages/python_twitter-1.0.0-py2.7.egg/twitter.py", line 3200, in GetFriends
    data = self._ParseAndCheckTwitter(json)
  File "/usr/local/lib/python2.7/dist-packages/python_twitter-1.0.0-py2.7.egg/twitter.py", line 4396, in _ParseAndCheckTwitter
    self._CheckForTwitterError(data)
  File "/usr/local/lib/python2.7/dist-packages/python_twitter-1.0.0-py2.7.egg/twitter.py", line 4421, in _CheckForTwitterError
    raise TwitterError(data['errors'])
twitter.TwitterError: [{u'message': u'Rate limit exceeded', u'code': 88}]

Not sure where the discrepancy is.

Allison

[bear]

glad to help!

ok, this is a bit of test code that I have - you will need to edit your twitter.py and change GetFriends() to be this code. It changes the return from GetFriends to be a tuple so now it needs to be called as:

cursor, friendsList, twitterError = GetFriends(screen_name='bear', cursor=-1)

cursor will be the pointer to the next batch, twitterError will be True if a ratelimit has happened and friendsList will be a list of friend entities.

Twitter is limiting requests for friends to *15* so I set the default to 10 - you will need to be careful on how often this is called!

  def GetFriends(self, user_id=None, screen_name=None, cursor=-1, skip_status=False, include_user_entities=False, count=10):
    '''Fetch the sequence of twitter.User instances, one for each friend.

    The twitter.Api instance must be authenticated.

    Args:
      user_id:
        The twitter id of the user whose friends you are fetching.
        If not specified, defaults to the authenticated user. [Optional]
      screen_name:
        The twitter name of the user whose friends you are fetching.
        If not specified, defaults to the authenticated user. [Optional]
      cursor:
        Should be set to -1 for the initial call and then is used to
        control what result page Twitter returns [Optional(ish)]
      skip_status:
        If True the statuses will not be returned in the user items.
        [Optional]
      include_user_entities:
        When True, the user entities will be included.
      count:
        How many friend entries to return during this call. Defaults to 10.
        [Optional(ish)]

    Returns:
      A tuple of the last cursor value and a sequence of twitter.User instances, one for each friend
    '''
    if not self._oauth_consumer:
      raise TwitterError("twitter.Api instance must be authenticated")
    url = '%s/friends/list.json' % self.base_url
    result = []
    parameters = {}
    if user_id is not None:
      parameters['user_id'] = user_id
    if screen_name is not None:
      parameters['screen_name'] = screen_name
    if skip_status:
      parameters['skip_status'] = True
    if include_user_entities:
      parameters['include_user_entities'] = True
    remaining=count
    ratelimited=False
    while remaining > 1:
      remaining -= 1
      parameters['cursor'] = cursor
      json = self._FetchUrl(url, parameters=parameters)
      try:
        data = self._ParseAndCheckTwitter(json)
        result += [User.NewFromJsonDict(x) for x in data['users']]
        if 'next_cursor' in data:
          if data['next_cursor'] == 0 or data['next_cursor'] == data['previous_cursor']:
            break
          else:
            cursor = data['next_cursor']
        else:
          break
      except TwitterError:
        ratelimited = True
        break
    return (cursor, result, ratelimited)

On Wed, Jun 5, 2013 at 3:27 PM, musegarden <allison....@gmail.com> wrote:

Thank you for your quick response in this matter.

Allison

[musegarden]

I tested this code and it seems to work as you described. It pulls down 20 users on the first run and if I plug in the returned cursor into the another call of GetFriends it just returns the same user (the last one returned from the first call) many times. I have a feeling I'm using the cursor wrong. Would I not be keeping track of this and using the returned cursor value to eventually get all of my friends?

Thanks for all your help.

Allison

[bear]

yes, the code should show that the cursor is extracted from the returned json from twitter and then used as the starting point for the next call.

it is possible that it's a bug tho :/

[musegarden]

I resolved the issue now, my mistake. :)

Allison

[bear]

\o/

glad to hear it was something resolvable and learnable :)

[musegarden]

I would mention, however, that when I reach my rate limit it does not show the TwitterError and quit as I thought it would. It simply says "NoneType is not iterable" when trying to call GetFriends (since it can't fetch anymore?).

I know this is the case, as I've checked the rate limits and every time I hit the limit it does this. I tried hacking on the modified GetFriends method to see if I could find the problem, but I didn't have any progress.

The way I'm getting around this right now is by just using a very general try/except block around this code, but this pretty much defeats the purpose of checking for a TwitterError in the first place...

Allison

[bear]

yea, my change was a hack so that you could get results.  I need to refactor the code and make all of them that do cursor retrieval follow the same pattern with proper error handling.