Incorrect Tweet Counts, Maximum Timeline Results

[Taylor Singletary]

Hi Folks,

Many of you have noticed incorrect counts of how many tweets a given user has. In addition, you've noticed that statuses/user_timeline is only allowing around ~800 statuses to be returned in total. These issues should be resolved by the end of next week -- we'll be rebuilding the cached representations of these timelines and users which will repair the issue. This process takes some time and will be gradual and incremental.

After consulting internally to get some clarity on the counts you should be expecting when using pagination, I wanted to share that while there's a maximum of about 3,200 statuses that you can retrieve for a given user_timeline (when under normal operating conditions), you'll find that statuses/home_timeline and statuses/friends_timeline do indeed have a maximum result set of around 800 tweets and have for some time. We'll be making sure the documentation is correct in this area where it's incorrect currently.

Of course, all of these theoretical maximums have some variance as caches are re-aligned in the system. Sometimes you might find a little more available than these maximums. Sometimes a little less. Important to note that the total tweet count includes retweets, but some methods don't return retweets unless you explicitly request them with include_rts=true.

Happy tweeting,

Taylor

[andrew brackin]

I am not very happy, only 3000?? I had around 10,000 so your telling
me i've lost 7000?
Twitter may not be able to do stuff about this but I am loosing my
life! I started my company, internet radio station and in a couple of
years I want to be able to go through (using services) to see the
tweets from when it was starting out and things like the day huge
events happen, my tweets are my life and I am disappointed.

If I am incorrect with the statements i've just said please go ahead
and reply.

Twitters a great service but I feel let down.
Andrew Brackin
http://twitter.com/andrewbrackin

On Jun 18, 9:36 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

[Taylor Singletary]

Hi there,

I can assure you that your tweets do in fact exist and have in no way
been lost.

The 3,200 limit is what's possible to retrieve via the API and what we
keep in active, accessible memory locations at any one time. Your
tweets are not lost. Your count will return to normal. Your tweets
exist in a database, exactly as they always have.

This limitation of the API was a part of the way applications and
services accessed Twitter prior to any of these present issues. It's a
limitation based on performance and scalability and not on thr actual
existence or non-existence of tweets.

Your tweets are safe!

[M. Edward (Ed) Borasky]

Quoting Taylor Singletary <taylorsi...@twitter.com>:

> Hi there,
>
> I can assure you that your tweets do in fact exist and have in no way
> been lost.
>
> The 3,200 limit is what's possible to retrieve via the API and what we
> keep in active, accessible memory locations at any one time. Your
> tweets are not lost. Your count will return to normal. Your tweets
> exist in a database, exactly as they always have.
>
> This limitation of the API was a part of the way applications and
> services accessed Twitter prior to any of these present issues. It's a
> limitation based on performance and scalability and not on thr actual
> existence or non-existence of tweets.
>
> Your tweets are safe!

But that does bring up an interesting point - being able to retrieve
all of one's own tweets is something a lot of people want. I know it's
on your roadmap, but I hope it doesn't get swamped in other issues.

[Neuromaster]

Hi,

Twitter is really a great tool for sharing and acquiring information.
Lately more and more people are starting to use it. That’s good news
because it becomes a trustable medium for information exchange. But,
paradoxically, it is bad news as well. While the number of users is
significantly increasing so is the number of possible candidates
(users) to follow.

Empirical proof might help in understanding the problem:

Suppose the user X is following 1000 users (which is not a rare
situation)
Suppose those users post in average 5 tweets/ day (again, a situation
not hard to accept)
User X is busy, but every morning for a limited period of time reads
the tweets of his friends.
That is ~5000 tweets. He chooses to skim through them and if something
good pops up he’ll pay more attention to it.

At this moment the web site it’s not designed for this kind of
interaction. Pushing the “More” button it’s just not so practical.
One may think that the API comes to rescue, but actually it
doesn’t.While it comes with some features, those are limited

The API is well suited for lightweight and mobile application where
the latest news is important. But for application that want to offer
some digging tools the API doesn’t offer much help. In the example
scenario the number of tweets accumulated in one day was ~5000 while
the limitation for the /statuses/home_timeline is ( at least in the
documentation) 3200. The problem here is that this limitation was ok
in the past, but now when more and more users are interacting through
tweeter this limitation becomes a real bottleneck.

From what I’ve understood the reason for this behavior is server
overloading. The servers can’t handle this amount of data.


The question:
Is there a way to grant access to more data for developers? If not
directly at least through some kind of request (like in the case of
Rate Limiting)

Thanks


On Jun 18, 11:36 pm, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

[John Kalucki]

I think this scenario is quite rare. Anyone who is following a
thousand accounts is just sampling their feed, they don't care about
every tweet. Also, a random population of 1000 accounts will not all
tweet 5000 times a day, but a selected group might.

If someone insists on reading all of this, perhaps as a work function,
they should use streaming. They can get all tweets from an arbitrary
set public users with the follow function, or, on user streams, all
tweets that the user follows.

-John Kalucki
http://twitter.com/jkalucki
Infrastructure, Twitter Inc.

[M. Edward (Ed) Borasky]

Quoting Neuromaster <neuro...@gmail.com>:

This sounds like a perfect use case for User Streams? Are you a
developer, or just trying to find out what's possible?

[Neuromaster]

Thank u for the suggestion.

On Jun 19, 5:31 pm, John Kalucki <j...@twitter.com> wrote:
> I think this scenario is quite rare. Anyone who is following a
> thousand accounts is just sampling their feed, they don't care about
> every tweet. Also, a random population of 1000 accounts will not all
> tweet 5000 times a day, but a selected group might.
>
> If someone insists on reading all of this, perhaps as a work function,
> they should use streaming. They can get all tweets from an arbitrary
> set public users with the follow function, or, on user streams, all
> tweets that the user follows.
>

> -John Kaluckihttp://twitter.com/jkalucki
> Infrastructure, Twitter Inc.