User Streams Preview
We're opening up a
developer preview of a Streaming API feature that we're calling User
Streams -- real-time updates of all data needed to update a desktop
application display. The grand vision has applications, eventually,
hitting the REST API for startup back-fill, then transitioning to
Streaming for nearly all subsequent reads. Streaming will revolutionize
the user's desktop experience -- rate limits and latency will
practically be eliminated. We'll also be able to provide new event types
and data that are currently impractical on REST. We hope that you can
stop managing rate limits and use this new data to create an entirely
new user experience. On our end, we hope to reduce costs and increase
We'd like to get your feedback and
get some products in the pipeline as we finish and eventually release
this feature. You can start experimenting today and we'll discuss a Beta
release and a General Availability (GA) release plan as we go along.
This is a very early preview, expect changes!
not release any products during the preview period. Capacity on
is very limited. If we reach capacity due to released products or
general abuse, we'll rev the URL and/or close the preview.
GA launch, we'll need to have a steady-ramp up of connections on the
new endpoint to avoid falling over. We must work together
closely on launching clients to avoid capacity issues. We need to get
appropriate GA capacity in place before starting a controlled,
- During Chirp, access is limited to venue IP
- After Chirp, we'll transition the preview to
generally accessible endpoint.
- Services must continue to
for large scale integrations: Firehose, Gardenhose, Track, Follow, etc.
User streams permissions are not tuned for service-to-service
integration, rather they are tuned for end-user-display applications.
the preview period, expect unannounced change and restarts on chirpstream.twitter.com.
Be prepared to rev your software quickly and maybe even have things
- Our cost is a function of new connections established
per second and total maintained connections. We hope to work closely
with developers to minimize connection churn and the total number of
connections held open. Each desktop client should maintain exactly one
streaming connection per active Twitter account.
- JSON only.
Consider the delimited=length option.
- Searches that don't
fit as track keyword searches will have to fall back to the Search REST
API. In general, if a search contains a space, it should fall back to
Search. We hope to soon expand track to support more boolean operators.
building logic to fall back to REST polling if Streaming is over
capacity, or if the user's network connection is too flaky.
and browser-based clients must not use Streaming until we've sorted out
Desktop clients at some scale. One problem at a time.
- The data
format and data content is open for discussion and considerable
evolution. Please share your thoughts on the dev list or directly to
- Follow all the guidelines, especially around
connections, in the Streaming API wiki.
- Logical AND is now
supported in track. Phrases are separated by commas. Words within
phrases are delimited by spaces. A tweet matches if any phrase matches. A
phrase matches if all of the words are present in the tweet. (e.g. 'the
twitter' is the AND twitter, and 'the,twitter' is the OR twitter.).
- "twitter api,twitter streaming" will
match the tweets "The Twitter API is awesome" and "The twitter streaming
deal is fast", but not "I'm new to Twitter"
- The same approach
to dealing with case, punctuation, @replies and hashtags still applies.
So "chirp search,chirp streaming" will match "Listening to the @chirp
talk on search", "I'm at Chirp talking about search!", and "loving this
search talk #chirp"
resource for the duration
of the Chirp conference is http://chirpstream.twitter.com/2b/user.json
Expect this endpoint to change as we release new features.
- Allow experimentation during a small-scale User
- In Q2 2010, begin a beta test of User
- In Q3 2010, launch User Streams at scale.
- Rationalize JSON schema. We haven't
rationalized all the message formats in a way that makes the stream
trivial to parse. Any feedback around parsing the stream would be
appreciated. Is it easy enough to determine what the message type is? Is
it easy enough to route the event to the appropriate display element?
social graph changes to a stream in progress to avoid the need to
restart to get the current timeline.
- Adjust track rate
limits and keyword limits.
- Move from Basic Auth to oAuth.
the count parameter to allow a few minutes of back-fill after a
- By default remove replies to users not
followed, and optionally allow.
Beta Test Changes
to expected production URL endpoints and clusters.
Subject to change
before beta test period.
- Tweets and Retweets
you or from your followings
- By track keyword
- From you or to you
- To you, from you, and from your
- Favorite Events
- To you, from you, and from your followings
- To you, from you and from your followings