[proposal] Realtime Opensocial feature

[Moishe Lettvin]

Hi there. I'm Moishe Lettvin, an engineer at Google. I've been working on Google Talk for almost 3 years now, and since I started working here I've been interested in making the web in general more "immediate".

Lately I've been thinking about adding realtime communication to gadgets. This proposal is a result of that thinking, with HUGE help from Lev Epshteyn, David Byttow, Brandon Beck and others here at Google. Lev & David specifically suggested that I share this proposal with this group.

Thanks, and thanks in advance for any feedback & ideas you may have.

-Moishe

---

Opensocial has come a long way towards enabling developers to create rich social interactions on the web's social networking sites, but a lot of these still have a "play by mail" feel. We're proposing enhancing the opensocial feature set by offering a more realtime model of communication - which will enable a whole new class of applications not really seen anywhere on the web. The initial proposal (which needs to be fleshed out) is to create a stack that would let developers connect their users in near real time - similar to an IM session but with a more domain specific client. Specifically, such a stack would facilitate creation of a large variety of multi-player apps such as a shared whiteboard, a realtime translation console, shared map viewing, various games, and so on.

At this time the proposal below and the APIs are yet to be solidified (especially on the server side), and we are looking for kindred spirits to help us nail down requirements and protocols, as well as to gauge the level of interest in the developer community. Some Containers may already have a lot of the inner plumbing in place to make this happen. Obviously, any input at all would be greatly welcome. We're in the early stages of defining this API; this proposal is meant primarily as a jumping off-point. It's subject to change and we want to hear ideas.

We propose the creation of two spec endpoints:

• a client-side JS API for joining and communicating in realtime sessions
• an HTTP-based arbiter server API for moderating such communications
  

The gadget talks to the arbiter bot using a defined javascript API. The arbiter receives HTTP requests based on these API calls and responds with messages to send to the various gadgets it has knowledge of. In the middle is a Container-implemented relay, responsible for translating the javascript API calls into HTTP requests and distributing messages received via HTTP response to javascript callbacks in the various gadgets. One characteristic of this model is that messages are distributed only as a result of API calls from a gadget -- there's no way for the relay to get a message from the arbiter except as the result of a request. This makes certain use cases (eg., a timer-based game where the client code is untrusted) more complicated to implement but greatly simplifies the API conceptually and practically.

To make it more concrete, imagine a developer working on a simple chess game. They would request JS API via a require feature (also supplying a URL for their arbiter server):

<Require feature="realtime">
  <Param name="arbiter">http://example.com/arbiter</Param>
</Require>

This would make available to the client JS the opensocial.realtime namespace, which would be used to open a session:

var gameKey = gadgets.views.getParams()['gameKey'];
opensocial.realtime.requestSession(gameKey, function(session, result) {
  mySession = session;
  mySession.setListener(onGameMessage);
  startGame();
});

A client joins a Session by supplying a session key - an identifier (which is implicitly scoped to the application) that uniquely identifies a Session. Any clients passing the same Session key to requestSession() will be joined to the same Session (with Sessions lazily created as needed). In the case of our chess game, the key was supplied via a View param, but other ways of obtaining it are possible. The JS API Session object would allow client code to send out JSON messages, and register a callback to fire when messages come in (in our case, imagine the move variable to represent one chess move, black or white):

function onGameMessage(message, sender) {
  if (message.type == GAME_MOVE) {
    processMove(message.move);
  }
}

function sendMove(move) {
  var message = {};
  message.type = GAME_MOVE;
  message.move = move;
  mySession.send(message, function(result) {

     if (result != opensocial.realtime.SUCCESS) {

       undoMove(move);

    }

  });
}

Interaction with arbiter

The arbiter functions a the hub through which all session information flows. In this API, there is no concept of sending a message directly to another user: messages go only from a gadget to the arbiter, or from the arbiter to a gadget. The arbiter may -- in fact, in many cases will -- end up forwarding a message sent by one user to another, but ultimately everything goes via the arbiter.

Thus, the arbiter is called when a user wants to join a session, leave a session, or send a message. These messages all contain a session identifier, a user identifier, and a message identifier. The arbiter will respond with a status message indicating success or failure, and optionally with other information to be relayed to the calling or other gadgets. Again, the Container-implemented relay is responsible for distributing these messages and translating them into javascript callbacks.

It is assumed that the arbiter will have access to and maintain the relevant state of the Session - for example in our game of chess, that state would comprise of the board, the position of all the pieces, which player plays for black and white, etc. Every message sent to the arbiter on behalf of the client must have identifying information (such as the client's Person ID - however it may need to be more flexible than that to support anonymous play if that is desired)

So, continuing our example from above, the chess developer wants to keep some game logic -- determining wins, complex rules like repeating cycles, and the like -- on the server to simplify gadget code, and ensure the data is not compromised by a rogue client.

The arbiter receives HTTP requests with parameters corresponding opensocial.realtime method parameters and JSON corresponding to javascript objects passed to opensocial.realtime methods. It responds with XML describing methods and parameters for opensocial.realtime and JSON for developer-defined objects to pass to those methods.

When the gadget calls opensocial.realtime.requestSession, the arbiter server will receive an HTTP request:

POST /arbiter HTTP 1.1

container=www.igoogle.com
action=RequestSession
sender=a
sessionid=gameKey

The container parameter specifies the container making the request. This should be the url of the container's home page, and will be used by the arbiter to disambiguate sender id's.
The action parameter specifies the action for the server to take. Some possible actions are RequestSession, LeaveSession, and SendMessage.
The sender parameter identifies the user requesting the action. This is an OpenSocial Person ID. We will also use OpenSocial Person ID's to address messages from the server. It is the responsibility of the Container to determine if the server should be allowed to send a message to the given user, probably based on if they've installed the application.
The sessionid parameter identifies the session that this action is associated with. Some sessionids may have special values (for instance, an arbiter may implement a 'lobby' sessionid that relays global game state to users, including (for instance) sessionids of available games).

The servlet that handles the POST will verify that the sender is allowed to join the specified session, and return a response:

HTTP/1.x 200 OK

<?xml version="1.0" encoding="UTF-8"?>
<requestsession id='sessionId' status='joined'>
  <participant>a</participant>
  <participant>b</participant>
</session>

The Container-implemented relay will call the callback function specified in the requestSession call with the information from this response.

Now that the game is joined to a session, it can make a move by sending a message to the arbiter. For purposes of illustration, the player with id 'a' is playing white, and id 'b' is playing black. User 'a' moves his bishop to e5, resulting in a call to the 'send' method on the session object created above. The container translates this into a request to the arbiter:

POST /arbiter HTTP 1.1

container=www.igoogle.com
action=Message

sender=a
sessionid=gameKey
message={type: 'MOVE', move: location: 'Be5'}

The arbiter verifies that this is a valid move. It also sees that this move places the black player in check, and modifies and re-broadcasts the message because of this (note the addition of the '+' in the location string).

HTTP/1.x 200 OK

<?xml version="1.0" encoding="UTF-8"?>

<message to='a'>
  {type: 'MOVE', move: 'Be5+'}
</message>

<message to=otherId>
  {type: 'MOVE', move: 'Be5+'}
</message>

The 200 response code indicates that the bot accepted the message. The container calls the callback passed to sendMessage with a SUCCESS result. The contents of the response from the arbiter indicate that messages should be sent to both members of the session. It's up to the Container to distribute these messages to the appropriate users -- it will call the functions specified in the session.setListener calls with objects corresponding to the deserialized JSON of the contents of the <message> elements.

The black player then tries unsuccesfully to move out of check by capturing the bishop:

POST /arbiter HTTP 1.1

container=www.igoogle.com
action=Message

sender=b
sessionid=gameKey
message={type: 'MOVE', move: 'Ke5'}

Unfortunately the bishop is protected by a pawn, so the arbiter rejects the move:

HTTP/1.x 400 Bad Request

<error>

  {description: 'Illegal move, you will move into check.'}

</error>

<message to='a'>

  {type: 'ATTEMPTED_MOVE', attempt: 'Ke5'}

</message>

The container calls the callback passed to sendMessage with a "Bad Request" result. Note that even this response could contain messages destined for other users, so the container needs to process these as it would a success response.

Note that the above is illustrated using XML to wrap messages to/from the HTTP server, with JSON representing objects created by the gadget itself. Another possibility is to use JSON exclusively.

[Scott Seely]

IIRC, David has created a game already that handles multiplayer running on AppEngine. It seems like the concepts in this proposal might fit better into an Open Source framework that developers would use to build OpenSocial games. I’ve seen the idea implemented several times: in a form of ‘battle tetris’ and in the poker games that several parties have published.

 

I would love to see more Open Source projects out there which help make Open Social development easier/better.

[Moishe Lettvin]

Totally agreed that an Open Source implementation of the relay would be a great addition to this. There are two complicated pieces -- passing messages around, and creating a persistent HTTP connection to eliminate the need for polling.

In my mind it's orthogonal to the API that I'm proposing below, though. You still need the touch points between the relay and the gadget, and between the relay and the server, and by decoupling them from the implementation it'll let container owners use their existing transport layers if they have them.

-Moishe

[Kevin Brown]

On Tue, Dec 16, 2008 at 4:00 PM, Moishe Lettvin <moi...@gmail.com> wrote:

Totally agreed that an Open Source implementation of the relay would be a great addition to this. There are two complicated pieces -- passing messages around, and creating a persistent HTTP connection to eliminate the need for polling.

In my mind it's orthogonal to the API that I'm proposing below, though. You still need the touch points between the relay and the gadget, and between the relay and the server, and by decoupling them from the implementation it'll let container owners use their existing transport layers if they have them.

I'd really like an abstraction like this. Given that many existing OS containers also provide an IM service, I suspect most already have transports. An open source transport based on a standard like XMPP would probably work for containers that don't.

I don't know if I'm sold on the specific concepts or design of the proposed API, but the basic concept of being able to authenticate a user and open a session is critical if I want to write something portable.

I'd also say that now is probably not the best time to get into details on this. Everyone is hard at work on 0.9 items, and this is definitely a new beast entirely. I'd like to see this fleshed out more by the time 1.0 work begins.

One somewhat ugly thing you could do without first-class container support is to do something like this:

1. Create an OAuth-enabled endpoint that would initialize the session for a user and pass back a session id.
2. Open the session channel with the provided session id.

The flow would look something like this:

function handeLoginResponse(obj) {
  if (obj.data.success) {
    initSession(obj.data.sessionId);
   }
}
gadgets.io.makeRequest("http://real-time-thing.org/login/oauth", handleLoginResponse, {AUTHORIZATION: "OAUTH"});

An interesting option that this opens up is the ability to get 'free' cross-container support for RTC. The security implications are a bit scarier, though.
 

[Bess Ho]

I like the concept. Ability to get closer to real time allow developers to make apps that are more interactive and engaging. Imagine many Facebook App games require notifications and messages for multi-player games. I would say this is innovative approach on social network. You'll get my vote for the concept.

--
Bess Ho

[Lev Epshteyn]

David's game actually uses polling and could have hugely benefited from a stack such as this. To me this proposal has great potential because it is going to formalize the endpoints of an infrastructure most containers already have or are developing (IM functionality), and allow 3rd party developers to build applications on top of this.

I have suggested for Moishe to bring it to the group earlier rather than later to get good feedback on the right, opensocial-y way of defining the API - I don't think anyone is suggesting this go into 0.9, but if we are to agree on something along these lines for v.Next, it would be good te see how people feel about the idea in general, and the various specifics.

I think the implementaitons would require pretty close container integration, so building this as a standalone open source project doesn't make much sense (though creating a Shindig extention to support this seems entirely reasonable).

[Scott Seely]

I’m not against the idea—I’m trying to figure out where to draw the line between things that an OpenSocial container needs to explicitly support vs. use cases it should allow. To support some interesting, server side scenarios, developers are also using an IFrame within the Canvas page IFrame to host a site at an independent location. Example:

 

<iframe src="http://www.google.com" height="100%" width="100%"></iframe>

 

Once I do this, I have a way to control all the RTC that I need to handle within that iframe via the content at src—any JS, Flash, whatever. MySpace also publishes a way to access the OpenSocial bits from the IFrame: http://developer.myspace.com/community/myspace/da6.aspx. The discussion in there can easily become a standard template or some such that people just use.

 

When combined with the versioning support we have in 0.9, we have a mechanism that allows a developer to handle the deltas between containers. They can then move forward on this kind of thing without waiting on us.

[Moishe Lettvin]

You're correct that it's currently possible to do any RTC you want without support from the container. For instance I worked a little bit on the "Talk Gadget" which uses a persistent HTTP connection to a Google server to push down RTC updates. This isn't an OpenSocial app, but is a gadget that does RTC with no container support.

One problem with this model is that multiple persistent HTTP connections can be expensive on the server side, and some browsers only support a limited number of them (I believe IE6 only supports two to the same domain).

Another more fundamental problem is that it's hard to write a good RTC channel -- a reliable persistent HTTP connection is tricky, not to mention the plumbing to deliver a given message to the correct endpoint.

This proposal aims to give gadget developers a standard way to leverage plumbing that the Container may already have to do RTC.

[Scott Seely]

Maybe I’m missing part of the proposal. Let’s see if I am understanding the required pieces and how they relate:

 

1.       Developer provides an arbiter and the machines to host the arbiter. Arbiter has these responsibilities:

a.       Establishes rules

b.      Matches players

c.       Knows which games/sessions are alive

d.      Receive messages from container

e.      Respond to messages from container.

2.       Container provides persistent connections between users and container. Container has these responsibilities:

a.       Maintain connection between user and container (assumption is that the system being used for IM can be reused for OS apps)

b.      Receive messages from application.

c.       Send messages to arbiter

d.      Broadcast response to all users involved in session

e.      Behavior: maintains millions of concurrent connections

 

A message sent to the arbiter probably looks something like this:

 

{

                sessionId:'orkut.com:1234',

                userId:'orkut.com:4567',

                appData : {application specific serialized object}

}

 

Essentially, this is a pub/sub system over HTTP. The value add of the container is that the container has already solved the millions of connections problem. Connections to the development servers happen as needed (session creation, person matching, member does work) and the container provides the mechanism to keep sessions talking to each other.

 

If this is true, then I can see where this has value. Suggestion: let’s spec this as an optional gadget feature. It goes without saying that we are discussing v.NEXT proposals at this time. J

[Bess Ho]

I am not the expert in polling. However I am aware of the challenges and limitations.

Twitter is becoming main stream and media culture partly because of the ability to exchange and share communication near real time like reporting news and earthquake. The ability to get near real time in app is useful, possible gain for better user experience. Perhaps the near real time may move closer to support the ideal concept of Web 4.0 - intelligent web.

Facebook is using polling for their Facebook Chat. I doubt Facebook will open up this resources for developer platform soon.

This video showed the challenges of HTTP connections and proposed ideas

http://www.youtube.com/watch?v=coZ-jVLsmVA

You can find Alex Russell on Google campus. His Dojo framework is working around the browser issues on polling.

http://www.linkedin.com/pub/0/842/320

Kaazing is working on open source API that handle HTTP connections. May be it worths consider collaboration. Kaazing co-founder is serving the W3C spec committee relating to these things, web sockets.

Moishe Lettvin and Lev Epshteyn,

I would be happy to introduce Kaazing co-founders if you drop me email outside this mailing list.

Bess

--
Bess Ho

[Moishe Lettvin]

Yeah, that's a great summary!

It's the "broadcast response" and "maintain millions of connections" where the container adds value.

The arbiter will respond to a request with something like:

<message from="arbiter" to="user1">
  {application specific serialized object}
</message>
<message from="arbiter" to="user2">
  {maybe a different application specific serialized object}
</message>

Then it's up to the container to distribute these messages. If the container already has a message-delivery system this can be tied in to that. Without the container, the arbiter would need to do its own message queuing which is a daunting task.

Agreed it should be optional.

[snacktime]

I do tend to agree that the idea being proposed is mostly orthogonal,
but I really like the idea of a set of extensions to opensocial or
something similar for stuff like this. I work at a smaller company
that creates fairly successful games on facebook, myspace, and bebo.
I am constantly trying to find solutions for stuff like this. It's
very difficult because it's either not supported by opensocial itself,
or the container doesn't allow it. For example any kind of efficient
polling method is a no go with makerequest (jabber http-bind for
example). And we can't access any number of web api's such as twitter
via opensocial because the container strips out all the required
headers.

Maybe a better way to think about this is that it's not so much an
orthogonal problem, but an evolutionary one. Opensocial as a whole
will have to encompass more and more as it goes along. Maybe even
just a forum for opensocial extensions would help. Something to
encourage containers, vendors, and developers to work on stuff like
this together instead of creating their own proprietary stuff. We
have enough of that going on already:)

Chris