dispatch example, (work in progress)

[robin]

Hi,

I've been tooling around with MochiKit.Async. I wanted a good place to
put the dispatch/poll logic needed when pasing events back and forth
over http to a backend server. This is how its looking atm:

http://svn.wiretooth.com/svn/devel/mochikit_contrib/examples/dispatch/dispatch.js

http://svn.wiretooth.com/svn/devel/mochikit_contrib/examples/dispatch/test_dispatch.html

It's a work in progress. I'm posting in the hope of some pointers to
how other folks do this sort of thing from js with MochiKit in the
picture. If it's usefull/appropriate for mochikit, I'd be happy to
contribute it.

Cheers,

Robin

[Bob Ippolito]

This is pretty cool and certainly useful for anyone who needs to
multiplex requests over HTTP. The code seems to be well written and
well tested, but I'm not sure we're really at the stage where this
makes sense as part of the MochiKit distribution.

The reason for that is that this code seems to belong to some kind of
relatively complex infrastructure that knows how to talk like this,
and what you've provided isn't even a full client-side (I don't see a
physical transport layer in here). If this was an implementation of
some kind of standard, then I'd take it for sure. I'd likely take it
even if it was available in at least two flavors of relatively
popular server technologies (e.g. TurboGears, Twisted or Nevow, PHP,
Bricolage, Rails, whatever).

Basically, I've never seen this particular approach before so I'm
questioning its practicality. Since we're currently only
distributing one flavor of MochiKit, kilobytes (kibibytes?) count. I
could be convinced otherwise, though, especially if it is some
standard that I just don't know about :)

-bob

[Robin Bryce]

Thanks for the encouragement :-)

> seems to belong to some kind of relatively complex infrastructure that knows how to talk like this,

Hrm, thats fair comment. MochiKit has that lovely "just exactly what
you need. And no more" feel to it. I can see that dispatch.js or
something like that might well be a layer too far.

The client-side bit needs finishing. I started looking at
implementations of Protocol using HttpRequest but thought it would be
good to get some feedback/advice before I got too carried away. I have
that "some one must have done this before, and better" feeling. If
there isn't something out there that doesn't leave me thinking "hey I
just came in for a stake sandwidch and now you say I have to eat a
cow!" then I think I'll try to take dispatch.js further.

The basic idea is roughly:

Wrap up an HttpRequest such that it can be conveniently invoked at
regular intervals. (That's what Protocol is for)

Each time the request is triggered it gets two query parameters: The
first is all the 'event' messages generated by the client side, and
the second is all the responses the client has made for events
generated on the server side.

In the client-side, the responses to backend originated events, and
new client-side events, pile up in Dispatch until 'interval' seconds
after the last request completes. Both the client-side and the server
side are allowed to take more than one http request/response cycle to
redeem a request. Each side of the circuit (client or backend) is
responsible for numbering the event messages it originates.

event messages look like either of:

{'tag_query':id,'body':opaquedata}
{'tag_resonse':id,'body':opaquedata}

dispatcher.js `Dispatcher.query` is passed `opaquedata`. Dispatcher
generates an id and queues a 'tag_query' formed message for the next
request cycle. `Dispatcher.query` is how the client-side creates a new
event targeted at the backend. It takes an optional deferred, and
returns a new one if none is given. The returned deferred will be
fired later when the backend responds with a 'tag_response' whose id
matches the original query. The 'result' is the opaquedata from the
body of the tag response.

`Dispatcher.respond` takes whatever id the backend chose to give the
event message, and the opaquedata, as parameters for the reply to the
backend originated event. But whatever `id` is, it should to be
coercable, losslesly, to a js property name. The line begining with
`log("backend query` in Dispatch.dispatch_result is where the hooks
need to go to allow the rest of the client program to intercept
backend originated queries, Dispatch.respond is how the client should
reply when its ready.

So the client side js realy just cares about deferreds and the format
of opaquedata. The message 'container' format and numbering is delt
with by the combination of Dispatch & Protocol. They just add enough
wraping to the 'opaque' application event data in order for the
backend and dispatch.js to arrange for transport.

It is infact only Protocol that needs to care about 'tag_query' and
'tag_response', altho a little extra indirection in
Dispatch.dispatch_result is needed to fulfill this property. And only
Dispatch cares about the numbering/id scheme (but the backend has to
honour this). Imposition on the backend is then:

* manage tagging of events originated at the backend.
* honour tagging of events originated at the client (tag_response =
tag_query).
* Respond in an appropriately formed way to http/GET or http/POST

Event messages are allways in json format. Arguably this need not be
the case and the desicion rests with Protocol.

The desired end, is a bi-directional circuit ,pigy backing on a
polling http/s request, that delivers events sourced at either the
client or the backend.
I guess the question would be, is this sort of thing usefull enough,
and often enough to find a home in MochiKit, even in the examples ?

I think I'll start out by rustling up a couple of minimal backends.
Twisted I can do, and I've been itching for an excuse to fiddle with
TurboGears and others. I'll stick to a directory layout that makes
droping into mochikit/examples easy and see how things look after
another weekends worth of fiddling. Advice, Admonishments and stake
sandwiches ;-) greatfully accepted!

Cheers,

Robin

ps "must remember to reply to list", oops

[Dieter Lunn]

Sounds like an interesting idea.  Its steak if you are refering to the meat dish.  I don't see a use for this unless you are wanting to write an application which updates without any user interaction, but that can be acheived with a simple timer and standard requests.  Thats just my opinion.

--
Dieter Lunn
http://www.coder2000.com

[Robin Bryce]

> and what you've provided isn't even a full client-side (I don't see a
> physical transport layer in here)

I've added, XMLHttpRequest based implementation and a trivial twisted
backend. And a bunch of documentation.

http://www.wiretooth.com/mochikit_contrib/examples/dispatch.html

There is a link in the abstract of dispatch.html to view dispatch.js
directly from svn.

In the download, backends/twistedweb.py is what the name implies.
Standing in examples/dispatch do ``python backends/twistedweb.py``,
then browse for
http://localhost:8080/examples/dispatch/test_dispatch.html to run the tests.
Documentation can also be found at
http://localhost:8080/examples/dispatch/doc/html/dispatch.html

Developed and tested on Ubuntu Linux 5.0.4, Python 2.4.1, Mochikit
trunk (yesterday, shortly before 1.1 release), Firefox 1.0.7. YMMV :-)

TODO (I think):
* Make the example more 'interesting' than present 'just the tests' mode.
* Implement another backend or two.
* dojo/jsan support

Do folks think this is this going in the right direction for an
'examples' contrib ?

Best,
Robin

[Robin Bryce]

Prompted by:

>Basically, I've never seen this particular approach before so I'm
>questioning its practicality.

<snip>

> I could be convinced otherwise, though, especially if it is some
> standard that I just don't know about :)

I did some more googling, looking for references re applications and
standards covering bi directional communication between browser and
backend.

In a discusion[1] of the TCPConnection object in the "Web
Applications 1.0 Working Draft — 18 November 2005" from
www.whatwg.org.

I came accross accross this comment:

> All the current implementations of that are huge
> hacks that would be much simpler to implement if they could just open a
> TCP connection and send free-form data back and forth.

--- http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2005-October/004886.html

And thought that sounded painfully close to dispatch.js alright.

Also encountered rfc 3080, 3081, and BEEP for the first time:

http://www.ietf.org/rfc/rfc3080.txt
http://www.xml.com/pub/a/2002/10/16/ends.html
http://www-128.ibm.com/developerworks/webservices/library/x-beep/

My interest in all of this is from the "casual online games"
perspective. And with the requirement that the user experience span
more than one device: mobile phone + PC based web browser. So I'm
quite encouraged by the TCPConnection, and PeerToPeerConnections,
mentioned in sections
6.3.4 & 6.3.6.1 of the "web applications" draft.

A javascript take on this is http://jsbeep.mozdev.org/ but it looks
like its gone cold.

Anyways, thanks to all for the feedback.

Cheers,
Robin