Handling "Late Arrivals"?

[Jonathan Eunice]

I run a script to collect the tweets in my friends_timeline. It uses
since_id to keep track of tweets already retrieved, advancing since_id
on each batch of tweets received. Works pretty well, but unfortunately
misses some tweets during heavy usage periods (like last night's US VP
debate). I suspect a scenario like this:

friends_timeline since_id=1000
gets tweets 1001...1050, with some holes in the sequence
friends_timeline since_id=1050
gets tweets 1051...1075, with some more holes in sequence
at the same time tweets 1022,1027, and 1031 are now available,
but are no longer being requested, so missed
iterate many times
each time losing tweets that were previously requested, but at
the time not ready
and missing out on the "late arrivals"

Is this scenario feasible/likely? And if so, what should I do to guard
against it?

I see no way to efficiently communicate "I have these ids, give me the
ones I don't have," which some APIs have.

The brute force approach would be to rescan friends_timeline
periodically to pick up late arrivals. Is it just that simple? Do I
really need to be that brutish? Or am I missing something?

[Alex Payne]

Yes, you simply need to rescan the friends_timeline. We can't
currently guarantee the ordering of responses from that method, and we
don't currently have a mechanism to "fill in the gaps". We do intend
to improve the failure case in the future, though.

--
Alex Payne - API Lead, Twitter, Inc.
http://twitter.com/al3x

[Jonathan Eunice]

Fine. I will rescan.

Can you estimate for me the maximum latency of message arrival? I.e,
how far
back do I need to rescan in order to get a high confidence that I have
"gotten
everything" in the friends_timeline? 10 minutes? An hour? Several
hours?

[Alex Payne]

We make no formal guarantee, but it's extremely unusual for us to be
inserting updates in timelines older than an hour.

--