'rate_limit.reset_in' from Twitter::Error::TooManyRequests is always nil

[Adam Dullenty]

Hey,

I am trying to use the error.rate_limit.reset_in delay from Twitter::Error::TooManyRequests to sleep the thread for x seconds before restarting the streaming client, but for me this param is always blank.

I'm basing my reconnection logic on the example provided: https://github.com/sferik/twitter/blob/master/examples/RateLimiting.md but in case it's related to something else in my code, I've ripped out lots of other code but this is basically identical to the main reconnection logic:

def client
  Twitter::Streaming::Client.new do |config|
    config.consumer_key         = [hidden]
    config.consumer_secret      = [hidden]
    config.access_token         = [hidden]
    config.access_token_secret  = [hidden]
  end
end

def start_search(keywords)
  client.filter track: keywords do |item|
    save! item
  rescue Twitter::Error::TooManyRequests => error
    puts 'Twitter::Error::TooManyRequests error'
    # TODO: Twitter gem isn't providing a rate_limit here, how come??
    delay = error.rate_limit.reset_in || 180
    sleep delay + 1
    start_search keywords
  end
end

Always resorts to 180 seconds delay (well, 181) - reset_in is always nil.

Any advice would be gratefully received.

[sferik]

Rate limiting works slightly differently in the Streaming API versus the REST API, which is shown in the rate limiting example.

Quoting from the Twitter FAQ:

How are rate limits determined on the Streaming API?

 

The public streaming APIs cap the number of messages sent to your client to a small fraction of the total volume of Tweets at any given moment. The sample hose, as documented in https://stream.twitter.com/1/statuses/sample.json, delivers a random sampling of all Tweets at a volume equal to the public streaming cap. Filtered streams return all matching Tweets up to a volume equal to the streaming cap. If there are more tweets that would match your criteria, you’ll be streamed a rate limit message indicating how many tweets were not delivered.

[Adam Dullenty]

Hi,

Thanks for the reply. That doesn't seem to explain the problem I am seeing here. I am seeing that even when the stream has been running uninterrupted for hours and hours, when I stop and restart the stream (e.g. when the 'track' keywords have changed), I still get Twitter::Error::TooManyRequests.

So I'm not talking about RateLimit messages being received while the stream is active, I'm purely talking about TooManyRequests errors when trying to set up the stream.

So I'm trying to resolve two things here, firstly to avoid getting myself banned by trying to get a sensible restart delay out of the TooManyRequests error when setting up a stream, but mainly to try to prevent getting this error at all.

So in summary is there any way to a) to get a 'reset_in' delay out of the TooManyRequests error when setting up a stream or b) to avoid this message in the first place for stream requests?

Thanks.

[Adam Dullenty]

Hi,

Is there a way of removing the 'Completed' flag on this post? Not sure we've reached a conclusion to the queries so don't want that flag to prevent people from replying.

I've been looking into the documentation for streams and found this page detailing the various ways in which a stream can be shut down. It seems that the 'disconnect' message provides details of why the stream was shut down which we are not able to access via the TooManyRequests error: https://dev.twitter.com/streaming/overview/messages-types

If we could have a way of accessing these reasons it would provide a much more useful insight into why the stream connection attempt failed. Is that already in place (apologies if I've missed that in the docs) or something which could be added to resolve this confusion?

To me it would seem sensible to separate these errors from the typical REST API rate limiting errors for the sake of clarity.

Thanks.