Can't access old tweets via statuses/user_timeline

[@seiz]

Hi,

I am Trying to backup all my tweets (for @seiz) but it seems tweets of
a certain age aren't accessible via the api (the oldest tweet i get is
ID 1226937920 from 02/2009).
I am even using since_id and max_id restrictions in the API call in
order to avoid hitting a pagination limit and still can't get any very
old tweets.
Same goes for mentions (and i guess everything else like DMs too).

How can i get all my tweets in order to back them up?
Note, i basically have to do this only once and then only get a daily/
weekly or whatever delta using since_id, so it should not put too much
load on the api.

PS: there's also a BUG. when using max_id in the api call, the result
will include tweets where ID==max_id which, according to the
documentation should not be the case and only every thweet with an id
between since_id and < max_id (but not including max_id) should be
returned.

(I filed a ticket on help.twitter, but am also posting here, as past
experience seemed to indicate, that the ticketing system is not
maintained – sorry for the cross posting)

Thanks
Stefan

[Mark McBride]

Currently we only support retrieval of the last N tweets, where N is 3200 if I recall.

  ---Mark

http://twitter.com/mccv

[M. Edward (Ed) Borasky]

Quoting Mark McBride <mmcb...@twitter.com>:

> Currently we only support retrieval of the last N tweets, where N is 3200 if
> I recall.

It is a maximum of 3200 of the user's *original* tweets. If, for
example, you retrieve tweets in pages of 200 tweets, you will get a
maximum of 16 pages. And only tweets that originated from the user
will be in those pages. If you're one of those people, like me, who
use the built-in retweet capability heavily, those retweets *won't* be
in what you get back, but they count against you as part of the 3200.
;-)

--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky/

"A mathematician is a device for turning coffee into theorems." ~ Paul Erd?s

[@seiz]

Thanks all for the replies.

Mark: Then the API Documentation is misleading as it says "Clients may
request up to 3,200 statuses via the page…". My assumption was i can
get a maximum result set of 3200 tweets but could change the timeframe
of the result set by using since_id and max_id.

What you say seems to be the case, but then the wording in the docs
should be changed to clearly state:
Only the newest 3200 tweets can be received via the API. Older
Tweets are still archived at Twitter but not accessible via the
archive.

After all, this is not a Pagination limit at all then! It's simply a
limit of how many tweets are accessible via the API.

Since there might be cases, where archiving tweets might even be
required by law (in corporate enviroments e.g.), accessing old tweets
might be a rare but needed case and i could see the following as a
solution not affecting your performance.

Allow access to an "archive database" of tweets via a new set of API
Methods - e.g.:
http://api.twitter.com/1/statuses/home_timeline_archive.format
And implement a different rate limit to these API Methods – e.g. 4,000
requests per Month
This should suffice most usecases, as downloading old tweets is often
a one time thing in order to catch up. Once othe old archive is
retrieved, we can use the regular API Methods more frequent to keep
our backup archive current and stay within the 3200 tweets boundary.

What do you think?

Stefan.

[@seiz]

Apologies if my reply comes through multiple times. But google tells
me my reply was posted but i never see it arrive here.

[M. Edward (Ed) Borasky]

I filed a request for the ability for an authenticated user to be able
to retrieve *all* of his or her tweets a long time ago. I can find the
date and other details on this group if necessary. Someone back then
from Twitter said that or some equivalent was on the road map.

--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky/

"A mathematician is a device for turning coffee into theorems." ~ Paul Erd?s

[Cameron Kaiser]

> Apologies if my reply comes through multiple times. But google tells
> me my reply was posted but i never see it arrive here.

Just remember that the group is moderated (I removed your duplicate).

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- armadillo, n. the act of providing weapons to a Spanish pickle. ------------

[@seiz]

Ah, i see. I thought only the first post would be moderated (and only
with that i got a notification about the moderation).
Thanks!

[Cameron Kaiser]

> Ah, i see. I thought only the first post would be moderated (and only
> with that i got a notification about the moderation).

The group is configured to auto-promote after a certain number of accepted
posts. The number is internally determined but is in the ballpark of 20 or 30.
I will make this clearer on the group home page.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com

-- He is rising from affluence to poverty. -- Mark Twain ----------------------

[Raffi Krikorian]

it is on our roadmap to be able to retrieve all your tweets.

just to be precise, however, all your tweets are retrievable, if you know their IDs -- you can then retrieve them using statuses/show endpoint.  what we don't maintain, however, is retrieval of more than 3200 tweets via the timeline calls.  this is a limitation of our timeline and storage architecture, but it is actively being addressed.

--
Raffi Krikorian
Twitter Platform Team
http://twitter.com/raffi

[M. Edward (Ed) Borasky]

I have over 20,000 tweets under my belt. Even whitelisted at 20,000
API calls per second ...

I'm thinking Twitter could charge a dollar a thousand for a full
backdump like that - I'd pay $20 to get a full CSV file of them.

--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky/

"A mathematician is a device for turning coffee into theorems." ~ Paul Erd?s