User Streams Preview (correction)

[John Kalucki]

User Streams Preview

4/15/2010

We're opening up a developer preview of a Streaming API feature that we're calling User Streams -- real-time updates of all data needed to update a desktop application display. The grand vision has applications, eventually, hitting the REST API for startup back-fill, then transitioning to Streaming for nearly all subsequent reads. Streaming will revolutionize the user's desktop experience -- rate limits and latency will practically be eliminated. We'll also be able to provide new event types and data that are currently impractical on REST. We hope that you can stop managing rate limits and use this new data to create an entirely new user experience. On our end, we hope to reduce costs and increase site reliability.

We'd like to get your feedback and get some products in the pipeline as we finish and eventually release this feature. You can start experimenting today and we'll discuss a Beta release and a General Availability (GA) release plan as we go along. This is a very early preview, expect changes!

Important Items

• Do not release any products during the preview period. Capacity on chirpstream.twitter.com is very limited. If we reach capacity due to released products or general abuse, we'll rev the URL and/or close the preview.
  
• At GA launch, we'll need to have a steady-ramp up of connections on the new endpoint to avoid falling over. We must work together closely on launching clients to avoid capacity issues. We need to get appropriate GA capacity in place before starting a controlled, large-scale launch.
• During Chirp, access is limited to venue IP addresses only.
  
• After Chirp, we'll transition the preview to generally accessible endpoint.
  
• Services must continue to use stream.twitter.com for large scale integrations: Firehose, Gardenhose, Track, Follow, etc. User streams permissions are not tuned for service-to-service integration, rather they are tuned for end-user-display applications.
  
• During the preview period, expect unannounced change and restarts on chirpstream.twitter.com. Be prepared to rev your software quickly and maybe even have things break.
  
• Our cost is a function of new connections established per second and total maintained connections. We hope to work closely with developers to minimize connection churn and the total number of connections held open. Each desktop client should maintain exactly one streaming connection per active Twitter account.
  
• JSON only. Consider the delimited=length option.
  
• Searches that don't fit as track keyword searches will have to fall back to the Search REST API. In general, if a search contains a space, it should fall back to Search. We hope to soon expand track to support more boolean operators.
  
• Consider building logic to fall back to REST polling if Streaming is over capacity, or if the user's network connection is too flaky.
• Mobile and browser-based clients must not use Streaming until we've sorted out Desktop clients at some scale. One problem at a time.
• The data format and data content is open for discussion and considerable evolution. Please share your thoughts on the dev list or directly to Twitter.
• Follow all the guidelines, especially around connections, in the Streaming API wiki.
• Logical AND is now supported in track. Phrases are separated by commas. Words within phrases are delimited by spaces. A tweet matches if any phrase matches. A phrase matches if all of the words are present in the tweet. (e.g. 'the twitter' is the AND twitter, and 'the,twitter' is the OR twitter.).  Some examples
  
• "twitter api,twitter streaming" will match the tweets "The Twitter API is awesome" and "The twitter streaming deal is fast", but not "I'm new to Twitter"
• The same approach to dealing with case, punctuation, @replies and hashtags still applies.  So "chirp search,chirp streaming" will match "Listening to the @chirp talk on search", "I'm at Chirp talking about search!", and "loving this search talk #chirp"

Resource

The temporary resource for the duration of the Chirp conference is http://chirpstream.twitter.com/2b/user.json. Expect this endpoint to change as we release new features.

General Plan

• Allow experimentation during a small-scale User Streams Preview.
  
• In Q2 2010, begin a beta test of User Streams.
• In Q3 2010, launch User Streams at scale.
  

Preview Period Changes

• Rationalize JSON schema. We haven't rationalized all the message formats in a way that makes the stream trivial to parse. Any feedback around parsing the stream would be appreciated. Is it easy enough to determine what the message type is? Is it easy enough to route the event to the appropriate display element?
  
• Apply social graph changes to a stream in progress to avoid the need to restart to get the current timeline.
  
• Adjust track rate limits and keyword limits.
• Move from Basic Auth to oAuth.
• Support the count parameter to allow a few minutes of back-fill after a transitory disconnect.
• By default remove replies to users not followed, and optionally allow.

Beta Test Changes

• Switch to expected production URL endpoints and clusters.
• Support Lists, perhaps.

Preview Schema

Subject to change before beta test period.

• Tweets and Retweets
  
• Created
  
• Deleted
  
• From you or from your followings
• By track keyword
  
• Direct Messages
• Created
• Deleted
• From you or to you
• Friendship Events
  
• Created
• To you, from you, and from your followings
  
• Favorite Events
  
• Created
• Deleted (TBD)
• To you, from you, and from your followings
• Retweet Events
• To you, from you and from your followings
  
• Deleted (TBD)