User Streams Preview4/15/2010

We're opening up a developer preview of a Streaming API feature that we're
calling User Streams -- real-time updates of all data needed to update a
desktop application display. The grand vision has applications, eventually,
hitting the REST API for startup back-fill, then transitioning to Streaming
for nearly all subsequent reads. Streaming will revolutionize the user's
desktop experience -- rate limits and latency will practically be
eliminated. We'll also be able to provide new event types and data that are
currently impractical on REST. We hope that you can stop managing rate
limits and use this new data to create an entirely new user experience. On
our end, we hope to reduce costs and increase site reliability.

We'd like to get your feedback and get some products in the pipeline as we
finish and eventually release this feature. You can start experimenting
today and we'll discuss a Beta release and a General Availability (GA)
release plan as we go along. This is a very early preview, expect changes!
Important Items

   - *Do not release any products during the preview period.* Capacity on
   chirpstream.twitter.com is very limited. If we reach capacity due to
   released products or general abuse, we'll rev the URL and/or close the
   preview.
   - *At GA launch, we'll need to have a steady-ramp up of connections on
   the new endpoint to avoid falling over.* We must work together closely on
   launching clients to avoid capacity issues. We need to get appropriate GA
   capacity in place before starting a controlled, large-scale launch.
   - During Chirp, access is limited to venue IP addresses only.
   - After Chirp, we'll transition the preview to generally accessible
   endpoint.
   - Services must continue to use stream.twitter.com for large scale
   integrations: Firehose, Gardenhose, Track, Follow, etc. User streams
   permissions are not tuned for service-to-service integration, rather they
   are tuned for end-user-display applications.
   - During the preview period, expect unannounced change and restarts on
   chirpstream.twitter.com. Be prepared to rev your software quickly and
   maybe even have things break.
   - Our cost is a function of new connections established per second and
   total maintained connections. We hope to work closely with developers to
   minimize connection churn and the total number of connections held open.
   Each desktop client should maintain exactly one streaming connection per
   active Twitter account.
   - JSON only. Consider the delimited=length option.
   - Searches that don't fit as track keyword searches will have to fall
   back to the Search REST API. In general, if a search contains a space, it
   should fall back to Search. We hope to soon expand track to support more
   boolean operators.
   - Consider building logic to fall back to REST polling if Streaming is
   over capacity, or if the user's network connection is too flaky.
   - Mobile and browser-based clients must not use Streaming until we've
   sorted out Desktop clients at some scale. One problem at a time.
   - The data format and data content is open for discussion and
   considerable evolution. Please share your thoughts on the dev list or
   directly to Twitter.
   - Follow all the guidelines, especially around connections, in the
   Streaming API wiki.
   - Logical AND is now supported in track. Phrases are separated by commas.
   Words within phrases are delimited by spaces. A tweet matches if any phrase
   matches. A phrase matches if all of the words are present in the tweet.
   (e.g. 'the twitter' is the AND twitter, and 'the,twitter' is the OR
   twitter.).  Some examples
   - "twitter api,twitter streaming" will match the tweets "The Twitter API
      is awesome" and "The twitter streaming deal is fast", but not "I'm new to
      Twitter"
      - The same approach to dealing with case, punctuation, @replies and
      hashtags still applies.  So "chirp search,chirp streaming" will match
      "Listening to the @chirp talk on search", "I'm at Chirp talking about
      search!", and "loving this search talk #chirp"

Resource
The temporary resource for the duration of the Chirp conference is
http://chirpstream.twitter.com/2b/user.json. Expect this endpoint to change
as we release new features.
General Plan

   - Allow experimentation during a small-scale User Streams Preview.
   - In Q2 2010, begin a beta test of User Streams.
   - In Q3 2010, launch User Streams at scale.

Preview Period Changes

   - Rationalize JSON schema. We haven't rationalized all the message
   formats in a way that makes the stream trivial to parse. Any feedback around
   parsing the stream would be appreciated. Is it easy enough to determine what
   the message type is? Is it easy enough to route the event to the appropriate
   display element?
   - Apply social graph changes to a stream in progress to avoid the need to
   restart to get the current timeline.
   - Adjust track rate limits and keyword limits.
   - Move from Basic Auth to oAuth.
   - Support the count parameter to allow a few minutes of back-fill after a
   transitory disconnect.
   - By default remove replies to users not followed, and optionally allow.

Beta Test Changes

   - Switch to expected production URL endpoints and clusters.
   - Support Lists, perhaps.

Preview Schema

Subject to change before beta test period.

   - Tweets and Retweets
   - Created
      - Deleted
      - From you or from your followings
      - By track keyword
      - Direct Messages
      - Created
      - Deleted
      - From you or to you
   - Friendship Events
   - Created
      - To you, from you, and from your followings
      - Favorite Events
   - Created
      - Deleted (TBD)
      - To you, from you, and from your followings
   - Retweet Events
      - To you, from you and from your followings
      - Deleted (TBD)