[twitter-dev] WebSockets protocol for streaming API

[Cezar Sá Espinola]

Hey guys,

Quick question, are there any plans on supporting WebSockets protocol for the Streaming API?

That'd be awesome for browser based Twitter clients (i.e. Google Chrome extensions). Without this it'll be very difficult for this kind of client to lavarage benefit from the upcoming user streams.

Thanks a lot for all your hard work,

Cezar Sá Espinola

@cezarsa

[John Kalucki]

The first release of User Streams is not intended for web clients due
to capacity constraints.

http://apiwiki.twitter.com/ChirpUserStreams
"
All services, mobile and browser-based clients must not use Streaming
until we've sorted out Desktop clients at some scale. One problem at a
time.
"

That being said, of course we'd like to encourage experimentation with
other client types, so that clients can evolve as we scale out the
service.

While others at Twitter are well-versed in the latest browser
technologies, I'm totally and willfully ignorant. If you could give a
brief summary of how the existing Streaming API does not work for
WebSockets, that might be helpful. What's missing?

-John Kalucki
http://twitter.com/jkalucki
Infrastructure, Twitter Inc.

[Abraham Williams]

I'm not particularly familiar with the specifics of WebSockets but here is the draft documentation: http://dev.w3.org/html5/websockets/

I don't see Chrome Extensions as being any different from desktop applications as they are both manually installed by the user on their desktop.

Abraham

--
Abraham Williams | Developer for hire | http://abrah.am
@abraham | http://projects.abrah.am | http://blog.abrah.am
This email is: [ ] shareable [x] ask first [ ] private.

[M. Edward (Ed) Borasky]

I'm going to keep the whole thread here because I think some important
distinctions are being raised / discussed.

1. As a developer, I have to create a (minimum) viable product. "Viable"
implies there must be a user interface beyond the command line. Yes, I know
Cameron Kaiser thinks otherwise, but the people that I'm hoping will pay me
money for a Twitter desktop product will most likely not want a command line
product.

2. The most efficient way for me to deliver that UI, given oAuth, is to fire
up a native browser and use it for at least the oAuth authentication and
"view" portion of my application. I'm also guessing that one of the "industry
standard" JavaScript libraries will be involved. Lots of wheels to not re-
invent, etc..

3. Twitter shouldn't have to care about the "model" and "controller" portions
of my application. I'm guessing they'll not be in browser-side JavaScript,
though, unless the JavaScript engines are a lot more efficient than I think
they are. ;-)

4. That leaves the interface to user streams. Twitter should only have to care
about connection-related and bandwidth-related parameters, *not* about whether
I connect with Perl, Ruby, PHP, Python, JavaScript, C, Java, or even Lisp,
Scheme or Forth. And Twitter shouldn't have to care whether the code is
executed inside a browser by an interpreter, by compiled Java applet code,
outside the browser by interpreted or compiled code, etc.

Remember, I've got to have the browser anyway! In my case I'm guessing the
connection will be Perl and outside the browser, because I know how to do
things efficiently in Perl better than the others. And I'm guessing my model
will require Perl efficiency as well.

But if someone can build the whole enchilada in a browser, it shouldn't matter
to Twitter as long as the connection and bandwidth properties are predictable
/ controllable. And, in fact, if *they're* ready to go to market the day user
streams graduates to production status and I'm still futzing with my Perl
code, well, I lose. ;-)

5. So let me throw a *huge* "what if?" on the table. What if there was an
official Twitter-supported @anywhere - user streams capability? Then we
*could* write a large chunk of a Twitter client in browser-side JavaScript.
Maybe even simple models could be done that way too, but certainly the view
and controller pieces could.

See Antonio Cangiano's (@acangiano) blog post for a little motivation:

http://antoniocangiano.com/2010/05/14/the-most-important-programming-language-
today/

M. Edward (Ed) Borasky
http://borasky-research.net/m-edward-ed-borasky/ @znmeb

"A mathematician is a device for turning coffee into theorems." ~ Paul Erdős

[Michael Brunton-Spall]

I'll just briefly disagree with you about point 2.

OAuth does not require that you have a browser available.  With Twitter supporting xAuth your desktop or mobile application can take a persons username and password and exchange them for an Access Token for accessing protected resources via OAuth without ever firing up the browser.

You can find out more about twitter and xauth at http://dev.twitter.com/pages/xauth

I think you are missing the point as to why twitter don't want web client to use the user streams right now.  Much fewer people use desktop applications than use mobile or web applications.  If you upgrade your desktop app to use the user streams then all of your customers have to upgrade their app, which will happen slowly over time.  If you upgrade a web app all of your clients get upgraded all simultaneously.  It sounds like it's purely a capacity issue at the moment.

Michael Brunton-Spall
Developer Advocate
guardian.co.uk

Please consider the environment before printing this email.
------------------------------------------------------------------
Visit guardian.co.uk - newspaper website of the year
www.guardian.co.uk  www.observer.co.uk

To save up to 33% when you subscribe to the Guardian and the Observer visit
http://www.guardian.co.uk/subscriber

---------------------------------------------------------------------

This e-mail and all attachments are confidential and may also
be privileged. If you are not the named recipient, please notify
the sender and delete the e-mail and all attachments immediately.
Do not disclose the contents to another person. You may not use
the information for any purpose, or store, or copy, it in any way.

Guardian News & Media Limited is not liable for any computer
viruses or other material transmitted with or as part of this
e-mail. You should employ virus checking software.

Guardian News & Media Limited
A member of Guardian Media Group PLC
Registered Office
Number 1 Scott Place, Manchester M3 3GG
Registered in England Number 908396

[M. Edward (Ed) Borasky]

On Sunday, May 16, 2010 12:58:13 am Michael Brunton-Spall wrote:
> I'll just briefly disagree with you about point 2.
> OAuth does not require that you have a browser available. With Twitter
> supporting xAuth your desktop or mobile application can take a persons
> username and password and exchange them for an Access Token for accessing
> protected resources via OAuth without ever firing up the browser.
>
> You can find out more about twitter and xauth at
> http://dev.twitter.com/pages/xauth

I'm aware of xauth, but I'm likely going to stick with browser-based oAuth and
with a browser-based view component of the application. It's too hard to get a
"portable" "native" desktop UI any other way. People know how to use browsers.

For example, think about the SugarCRM "Community Edition". If you run that as
a stand-alone desktop application on a Windows, Linux or Mac desktop, what
you've actually got is a whole PHP stack model/view/controller application
with your browser pointing to localhost. Using a browser for the UI and a
model/view/controller stack vastly increases the number of open source tools
I can bring to bear.

> I think you are missing the point as to why twitter don't want web client
> to use the user streams right now. Much fewer people use desktop
> applications than use mobile or web applications. If you upgrade your
> desktop app to use the user streams then all of your customers have to
> upgrade their app, which will happen slowly over time. If you upgrade a
> web app all of your clients get upgraded all simultaneously. It sounds
> like it's purely a capacity issue at the moment.

First of all, this is a new application, not an upgrade to an existing one.
Second, since it depends on Twitter and user streams, it's not going to be
deployed until Twitter has the capacity to support it. And third, it *is*
going to be a desktop application, not deployed on servers or mobiles. I
simply don't want to have technologies ruled out because they're "usually"
used on servers, like a web application stack.

[John Kalucki]

I did a quick reading, and I couldn't tell if WS connections are
restricted back to the domain of the calling page, or if arbitrary
connections are allowed. If the former, there may not be a reason for
the Streaming API to support this. If the latter, perhaps there's a
valid use case.

Looking at the protocol itself, this wouldn't be trivial to support,
but it might not be that bad either. We'll have to keep an eye on the
installed browser base and see when this might make sense.

It is too bad that they didn't allow HTTP connections with incremental
byte reads in addition to WB connections on that interface. I'm sure
they had their reasons.

-John Kalucki
http://twitter.com/jkalucki
Infrastructure, Twitter Inc.

[John Kalucki]

Ed,

Efficiency isn't the only goal of user streams, but it is an important
one when considering systems at this scale. When optimizing such a
system, you generally go looking for the biggest reward first.

We've identified applications that run on the desktop, such as
Tweetdeck, Seesmic, and friends, as a larger consumer of resources on
a per-user basis. They're always on and always polling at the maximum
rate and the traffic comes directly from the desktop. So be it, as
this is core to their utility and nature.

Browser-based clients are service-based. The aggregate user streams
demand from service-based applications, of all types, not just browser
clients, far exceeds that of desktops. We cannot possibly launch into
service-based demand. It's too vast. To ration access in this
environment would be hard for everyone.

Also most services do not place quite as large of a per-user load on
the REST API. Browsers don't reload, or reload infrequently. Services
mediate the load is various ways. Etc.

Finally, services are not best served by user streams as they are
today. They should have all of their requested traffic delivered over
a small number of multiplexed streams. It's not ideal to open 10^5 or
10^6 connections from a given service to Twitter, only to have their
aggregate traffic be far less than a Gardenhose feed. The service-side
complexity to manage all these connections is large, and the resource
utilization on our end is quite high. User streams, with a connection
per user, would be an efficiency loss here, where multiplexed streams
would be a tremendous efficiency gain.

So, services, as a category, are too big to tackle first, they
probably offer an efficiency loss, and, with a little more work,
services can be supported with a huge efficiency gain for everyone. So
we're initially going for the biggest efficiency win with the smallest
feature set against the smallest reasonably partitionable market given
the limited resources we have available.

-John Kalucki
http://twitter.com/jkalucki
Infrastructure, Twitter Inc.

[Harshad RJ]

On Mon, May 17, 2010 at 12:35 AM, John Kalucki <jo...@twitter.com> wrote:

Browser-based clients are service-based.

There are many counter examples that are not service-based. (Hootsuite, Brizzly and err tDash).

Instead of generalizing browser-based clients, would it not be better to specifically address service-based clients (and thus exclude them from the user stream API)?

--
Harshad RJ
http://hrj.wikidot.com

[Zac Bowling]

Web sockets allows you to support cross domain access however you want
by checking the "Origin:" header. If you only want to allow your own
domain, then you can compare the Origin equals your own domain and if
you want to allow some domains or all domains you can whitelist as you
want.

According to the lastest version of the draft spec's security concern:
http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-55#section-7

One thing to understand is that WebSockets are not really HTTP.
WebSockets handshakes may start out looking like an HTTP in the
initial request and initial response but that only so that it's
possible for a web server serving HTTP today could be used to handle
the requests and be able to tell that it's a WebSocket and internally
switch over to handling the request as a WebSocket internally.

Websockets are not like HTTP though in that they are full duplex
communication (real two way communication). It avoids some of the
tricks you have to do with AJAX Comet/Long polling/http streaming by
giving the browsers something more like traditional TCP/IP sockets to
communicate back the server while still working through HTTP proxies
and firewalls.

Jetty, the web server that Twitter uses today on the streaming side
supports WebSockets in its latest version
(http://blogs.webtide.com/gregw/entry/jetty_websocket_server). Most of
the web socket server implementations that exist today are built on
top of traditional web servers, but they don't use epoll or any kind
of event based async sockets (most design around having a single
thread handle the connections for a user) and so they are unable to
handle massive numbers of simultaneous open concurrent connections
like you can with Jetty or FriendFeed's Tornado
(http://www.tornadoweb.org/).

I would love to see a streaming server implementation using WebSockets
from Twitter, not only just for access from web browsers but from
desktop clients that support websockets. There are some doors that
open with that like being able to send back messages to Twitter up the
socket (like sending a tweet, or changing the paramaters of what we
are filtering on the stream, or request all sorts of things that you
would have to do on the querystring and have to reconnect to change
them).

Zac Bowling
@zbowling

[Raffi Krikorian]

as a fallback - some people have had some success using http://github.com/r/twstreamer to get access to the stream from javascript (using flash as a bridge).  usual caveats around production-izing streams apply.

--
Raffi Krikorian
Twitter Platform Team
http://twitter.com/raffi

[Cezar Sá Espinola]

Hey guys, thanks for the responses.

You're right, the motive why I can't use the WebSockets interface is that although the header information for the WebSockets protocol is based on HTTP headers, it's not HTTP and you cannot open a WebSocket connection to an HTTP endpoint.

The only other option for network connectivity within browser based clients is using the XmlHttpRequest interface for opening HTTP connections, that's perfectly fine for the REST and the search APIs, however, trying to establish a persistent connection with the XmlHttpRequest interface for use with the streaming API doesn't seem possible.

As for cross-origin requests policy I think WebSockets rely on the Cross-Origin Resource Sharing specification [1] authorizing or not the connection based on a whitelist sent by the server on the response header. So that's probably a non-issue.

Another benefit that I can imagine is that WebSockets allow a 2-way persistent connection. Because it's a 2-way connection that could make it a lot easier to update the filter predicate without the necessity of reconnecting, but that's another story. :)

[1] http://www.w3.org/TR/access-control/

Again, thanks for your hard work,

Cezar Sá Espinola

@cezarsa