Tweepy streaming time drift

[Larry Zhang]

Hi everyone,

I have a program which uses tweepy's stream listener to download
tweets from the Twitter Streaming API (garden hose),
and I am experiencing the following problem: the timestamps of the
tweets that I downloaded constantly drift behind
real-time, the time drift keeps increasing until it reaches around 25
minutes, and then I get a timeout from the request, and then
connection is reset.

To verify where the problem is, I tried to directly use curl to
download messages and also check the time drift, it turn out there is
no problem in this case.
So I guess there is something inside the tweepy or I am not using it
in the right way.

I will appreciate it if someone could provide some insight on this.

Thanks!
-Larry

[Larry Zhang]

One more observation: it is actually a slow download rate that is
causing the time drift.
The stream downloader wasn't able to keep up with the incoming rate of
the tweets and some buffer gets queued up and eventually causes a
connection reset.
Cheers,
-Larry

[Josh Roesslein]

Thanks Larry for pointing out this issue. My current plan is to reallly deprecate the current

streaming API in Tweepy and eventually replace it with this new framework I started [1].

Instead of using python's HTTP client, it uses curl. This appears to perform much better

and should help with this issue. You are free to look into it and let me know if you have any questions.

Of course any contributions are welcome as well. :)

Josh

[1] http://github.com/joshthecoder/turbine

[Pascal Jürgens]

Hi Josh,

this sounds really exciting. I'll be using the streaming API for some time to come, and I'm building a new architecture, so I'll definitely check it out. See you in IRC,

Pascal

[Pascal Jürgens]

Larry,

I also answered on the twitter-list. Did you find out what exactly was causing this? Was your internet connection too small? I know that the 15% stream is about 1.5 MBit/s - so that can definitely be the cause.

Pascal