Unifying our interfaces across client and server

[Seth Ladd]

Hello Dartisans,

Based on the feedback we're getting from the hackathons and mailing list, it's clear we're building two worlds that are different and thus more confusing than needs be. We have server libs and client libs with similar intents, but very different APIs.

I'd like to propose we try to reduce the number of APIs users need to learn and create shared abstractions where appropriate. For example:

Unify WebSocketHandler's onOpen and ServerSocket's onConnection to same name

Unify HttpClient and XMLHttpRequest into common interface

Unify HTML5 FileSystem API and File/Directory from dart:io into common interface

Unify Timer and window.setTimeout into common interface

Also, we should unify how we assign callback handlers. For example:

dart:html is element.on.click.add((e) => ...);

dart:io is webSocket.onOpen = (conn) => ...;

dart:core is new Timer(1000, (timer) => ...)  (then is fine, simply pointing out the callback is passed as parameter)

Each time we have two ways to express more or less the same intent, the user is then forced to A) learn two things and B) question what the subtle differences are. Users look to our code as the "shining example". Also, I've seen over and over that dart:io is initially seen as client side functionality.

There are bugs for many of these requests, but I'd like to come to a team consensus that unified abstractions are a good thing, and than a more common way to pass callbacks is a good thing.  Thoughts?

Thanks,

Seth

[John Evans]

I haven't done a lot of server side coding in Dart yet, but everything
you propose here makes sense and I support it.

[Seth Ladd]

John, Which callback mechanism feels more Darty to you?

[John Messerly]

Just to throw out another callback mechanism: Futures. Although I think that only works for one-shot events.

- John

[John Evans]

Well since you asked... :)

My choice would be something like a Future pattern that supports 1 to n returns.

I am of the opinion that all sequences (events, iterables,  and externals [like network transmissions]) should provide a uniform interface:

Element.on.click.observe((nextValue){});  //also optional params for onComplete() and onError(e)

Timer(1000).observe((nextValue){}); 

myListOfStuff.observe((nextValue){});

WebSocket.onOpen.observe((nextValue){});next

You get a lot of bang for your buck with just uniformity alone, but you also get compos-ability other observable operators, as I've shown with the Reactive Dart library:

myElement

  . on.click /* assuming click as an observable */

  .throttle(300)

  .observe((next){});next

or

Observable

  .merge([socketObservable1, socketObservable2])

  .observe(...); 

John

On Saturday, April 28, 2012 8:10:32 PM UTC-5, Seth Ladd wrote:

John, Which callback mechanism feels more Darty to you?

[mythz]

Seth,

That's definitely a good idea although I'm hopeful it can be done
without limiting the usefulness/features of the Dart VM
implementations as a result of any client-side limitations.

I do think there should be a server implementation for any non-DOM
related client library and have them both share a functional 'subset
API' (i.e. whats limited by the client) but still allow the server
implementation opportunity to provide additional functionality.

The other approach is to have very specific client / server APIs
around their respective libraries and a higher level API that provides
a unified high-level interface on top of both of them so client
libraries can build solutions off the shared abstraction.

This is the approach I plan to take with the JsonClient where its high-
level API is impl-agnostic and just exposes a programmer intent API:
https://github.com/mythz/DartJsonClient

The only API that isn't currently portable are the hooks:
>> client.requestFilter = (HttpClientRequest httpReq) { ... };
>> client.responseFilter = (HttpClientResponse httpRes) { ... };

So yeah in this area a shared HttpClientRequest/HttpClientResponse
interfaces are welcomed.

As for the event APIs they should be consolidated where possible
although they're currently optimized for their different use-cases,
e.g:

>> dart:html is element.on.click.add((e) => ...);

Uses a multi-cast event API and the use of 'add' suggests multiple
events can be assigned to this elements click handler, which it does.

>> dart:io is webSocket.onOpen = (conn) => ...;

This API suggests only 1 handler can be assigned at one time, so it
doesn't make sense for this to share the above API.

If you follow the spirit of dart:html events API you could have
something this to suggest a single handler:
>> webSocket.on.open.set((conn) => ...);

But that looks like a lot more work than what it's replacing. I think
a good compromise is taking inspiration from C#'s event handlers by
making use of the '+=' operator so they become something like:

>> element.on.click += (e) => ...;
>> element.on.click -= removeExistingHanlder;
>> webSocket.on.open = (conn) => ...;
>> webSocket.on.open = null; //removed

For classes that just encapsulate a single/primary event handler I
suggest leaving it as-is since you wont be able to beat having it
assigned in a single expression:

>> dart:core is new Timer(1000, (timer) => ...)

Cheers,
Demis

[Ross Smith]

I totally agree. Dart should provide an observer pattern, either
directly in the language (ala C#) or in the core library, and all SDK
API should adhere to it. For better or for worse, people will look to
the core APIs for patterns to use in their own code, and right now
mayhem reigns (well intended mayhem, of course). Today, lots of
people (like me) are all coding their own event/happening/occurence/
whatever and thus all of our 3rd party libraries are going to be hard
to swallow for anyone who tries to put them together. And we are
doing that because we know we need it, whereas some newcomers may not
know what to do at all.

-Ross

[Eric J. Smith]

I couldn't agree more. There are several patterns for events alone. And many other patterns need to be standardized as well. I personally like the obj.on.click pattern for events because it keeps all the events for an object in one location and keeps the object cleaner. I really think that events should be observable as well.

Would also be nice if there was some language feature that objects could opt into that would make properties automatically observable.

[Ladislav Thon]

I'd like to propose we try to reduce the number of APIs users need to learn and create shared abstractions where appropriate. For example:

Unify WebSocketHandler's onOpen and ServerSocket's onConnection to same name

Unify HttpClient and XMLHttpRequest into common interface

Unify HTML5 FileSystem API and File/Directory from dart:io into common interface

Unify Timer and window.setTimeout into common interface

I tend to agree, but I fear that reducing the APIs to the least common denominator could make them hard(er) to use. That would be throwing out the baby with the bathwater.

 

Also, we should unify how we assign callback handlers. For example:

dart:html is element.on.click.add((e) => ...);

dart:io is webSocket.onOpen = (conn) => ...;

dart:core is new Timer(1000, (timer) => ...)  (then is fine, simply pointing out the callback is passed as parameter)

I understand that this is a BIG issue and should definitely be fixed, but I can't yet see a consensus on what should the right way look like.

Playing the devil's advocate, I'd say that it's inevitable to have differences between client-side Dart and server-side Dart, and we shouldn't try to cover them, but rather embrace them. We have to clearly explain to people that even if you use a single language for server-side and client-side programming, it still doesn't make any sense to create a HttpServer in the browser or play music on the server. Having client-side APIs clearly different from server-side APIs should be a part of the these explanation efforts.

LT

[Gilad Bracha]

Seth,

I am very supportive of your suggestions. There are probably legitimate differences, but I suspect that there are many that are gratuitous and simply drven by uncoordinated development. In general, cases where are differences could perhaps be harmonized by combining things into common type hierarchies, or having some APIs fail dynamically. 

--
Cheers, Gilad

[jako]

On that regard, I think the serverside is still weak when it comes to working with typed arrays... 

so (at least from my understanding) on client you have all these typed arrays while on the server  you have to work with List<int> and such to get the same result.

I have to admit that I didnt wrote any client-server application so far, but it would make sence to unify these, lets say for example for a client-server game that use the same mathlib or to have simply one cummunication library that works on server and client.

Or simply a game/app that runs a big portion of shared code on server and client.

[Ruudjah]

Unified event/callbacks: +1
Unified library interfaces +1. This might be combined with the library
injection proposal.

> it still doesn't make any sense to create a
HttpServer in the browser or play music on the server.

Why not? I can see a usecase for an http server in the browser
(provided security is taken care of), for example true p2p torrents.
Especially with ipv6 in the pipeline (less NAT), this becomes
increasingly more interesting. IIRC, there was a w3c proposal for p2p
in the browser, I can't remember if that involved a complete HTTP
server. Music is a more clearcut example, however, even there valid
usecases can be thought of. What about serverside audio processing,
for example? This might use the same library interface for clientside
music handling. Admitted, this might be better done using a seperate
serverside audioprocessing lib, but it's still not entirely impossible
there's a usecase for serverside usage of the audio library. A better
usecase might be a server playing a sound on a critical error.

Last, who are we to decide that API's cannot be used on the server, or
the client? Tomorrow a brilliant coder might stepup and find a
compelling usecase for it. We should not explicitly prevent this from
happening ;).

> ...
>
> read more »

[Sean Eagan]

On Sat, Apr 28, 2012 at 5:20 PM, Seth Ladd <seth...@google.com> wrote:

Based on the feedback we're getting from the hackathons and mailing list, it's clear we're building two worlds that are different and thus more confusing than needs be. We have server libs and client libs with similar intents, but very different APIs.

This disjointedness is contagious as well, it forces library developers to expose separate client and server libraries even when there is no difference in functionality.  This would be a regression from JavaScript where many of the most popular APIs can be used transparently across browsers and node.js.  

Unify WebSocketHandler's onOpen and ServerSocket's onConnection to same name

After the connection is established, this interface is symmetric across client and server, so would probably be best to expose WebSocketConnection on both client and server, and have a WebSocketClient interface for setting up a connection which could be made available on the server in the future, just like HttpClient is.

Unify HttpClient and XMLHttpRequest into common interface

Unify HTML5 FileSystem API and File/Directory from dart:io into common interface

Unify Timer and window.setTimeout into common interface

I think the w3c APIs which were built specifically for JavaScript should be replaced by APIs which take advantage of the many nice library building features in Dart (HttpClient, Timer, etc.).  As with Dart as a whole, this will probably be controversial, but it's what's best for users.

I think a mix of Gilad's suggestion of common type hierarchies and dynamic failure, as well as dynamic environment testing via "isClient" and "isServer" getters somewhere would be a good approach.

Also, we should unify how we assign callback handlers. For example:

Each of these can be broken down into a few aspects, each of whose merits can be discussed separately:

dart:html is element.on.click.add((e) => ...);

* namespacing of events into an "on" getter

  * overkill for objects with smaller number of events

  * more difficult to find which events are available for a given interface

* multi cast (multiple handlers can be added)

* protects against clobbering of existing handlers

dart:io is webSocket.onOpen = (conn) => ...;

* single cast (only one handler can be added)

* does not protect against clobbering of existing handlers

* setter without getter is surprising to users, as noted by the Dart style guide

dart:core is new Timer(1000, (timer) => ...)  (then is fine, simply pointing out the callback is passed as parameter)

* single cast, and cannot reassign the single handler

The above all deal with async events which may occur any number of times.  There are also async "operations" which are events guaranteed to occur exactly once, or fail with an exception:

The most common approaches outside of Dart are:

* pass a callback and an "errback" to the operation itself

* use Futures (or similar).

  * multi cast

  * operation can provide exact same parameter signature (don't need to think about parameter ordering of callback, errback, future which may be added in the future) as if it were a sync operation, merely changing the return type from Foo to Future<Foo>

  * can take advantage of an "await" keyword or similar

  * allows composition of arbitrary async operations

dart:io async operations generally take a callback, but not an errback, which means error handling cannot be localized for individual operations or even for operation types.

Futures are provided in dart:core, but are rarely used by the core libraries, which is a clear violation of the dogfooding principle :) .

I would suggest the following approach:

* on<EventName> naming convention everywhere, i.e. no "on" getters

* create a multi cast Event interface (similar to dart:html's existing EventListenerList), and use it for all events

* For async operations use Future everywhere

* Have Future extend Event so callback registration mechanism is consistent regardless of whether the event occurs multiple times, which for example would allow using the "await" keyword for arbitrary events.

I updated my previous more detailed proposal here:

https://docs.google.com/document/d/17crOxOmR5AYwultXbNcfdbvD4ACBw5a9QbvO90FVgtk/edit

Cheers,

Sean Eagan

[Bob Nystrom]

Finally got a chance to go through this. :) Some random feedback:

On Mon, Apr 30, 2012 at 10:34 AM, Sean Eagan <seane...@gmail.com> wrote:

dart:html is element.on.click.add((e) => ...);

* namespacing of events into an "on" getter

I actually like this, but arguments against it are valid too.

 

  * overkill for objects with smaller number of events

Good point.

 

  * more difficult to find which events are available for a given interface

I don't think that's too bad because you'll look for an "on" getter and go from there. Having them all in a separate interface is kind of nice in that respect because it means you can see the set of events for a given type independent of all of the other operations the type supports.

 

* multi cast (multiple handlers can be added)

* protects against clobbering of existing handlers

Both good.

 

dart:io is webSocket.onOpen = (conn) => ...;

* single cast (only one handler can be added)

Generally bad because it isn't clear what happens to the previously registered one.

 

* does not protect against clobbering of existing handlers

Yeah, bad.

 

* setter without getter is surprising to users, as noted by the Dart style guide

Double plus bad.

 

dart:core is new Timer(1000, (timer) => ...)  (then is fine, simply pointing out the callback is passed as parameter)

* single cast, and cannot reassign the single handler

The above all deal with async events which may occur any number of times.  There are also async "operations" which are events guaranteed to occur exactly once, or fail with an exception:

The most common approaches outside of Dart are:

* pass a callback and an "errback" to the operation itself

One problem I find with this is that the error handling doesn't compose. When I do a chain of async operations, I have to make sure to register an errback at each level and pass along errors. Ends up feeling like C all over again. One of the very nice things about Futures is that they handle that (mostly) automatically.

 

* use Futures (or similar).

  * multi cast

  * operation can provide exact same parameter signature (don't need to think about parameter ordering of callback, errback, future which may be added in the future) as if it were a sync operation, merely changing the return type from Foo to Future<Foo>

Yes!

 

  * can take advantage of an "await" keyword or similar

Absolutely. Any consistent generic pattern makes it easier to add syntactic sugar to the language later.

 

  * allows composition of arbitrary async operations

This is a big win, though it is still pretty cumbersome. Pub uses futures heavily and I definitely prefer them over callbacks, but all of the chain() and transform() calls can get confusing.

 

dart:io async operations generally take a callback, but not an errback, which means error handling cannot be localized for individual operations or even for operation types.

Yes. This also means many of the dart:io types are mutable in surprising and unpleasant ways. If my function takes a File, to be safe I should really make a copy of it so that I don't accidentally trash any on__ handlers the caller may have registered on it. Lame.

 

Futures are provided in dart:core, but are rarely used by the core libraries, which is a clear violation of the dogfooding principle :) .

Pub uses IO heavily and does so through a wrapper layer we're writing that uses Futures instead of callbacks for everything: https://code.google.com/p/dart/source/browse/branches/bleeding_edge/dart/utils/pub/io.dart.

I would suggest the following approach:

* on<EventName> naming convention everywhere, i.e. no "on" getters

* create a multi cast Event interface (similar to dart:html's existing EventListenerList), and use it for all events

* For async operations use Future everywhere

* Have Future extend Event so callback registration mechanism is consistent regardless of whether the event occurs multiple times, which for example would allow using the "await" keyword for arbitrary events.

I think I'm generally in favor of this. We definitely need to do something more consistent here.

Thanks for spending the time to think this through and put together something coherent.

Cheers!

- bob

[Eric J. Smith]

So C# has issues where you have to be very careful to unwire all of your events or the garbage collector will never be able to clean the objects up.  This is a pretty big gotcha and a pain to track these sort of issues down.  Is Dart falling into this same trap?  Are there better approaches?  Have you guys explored embracing the Observable pattern instead of events?  There seems to be a lot of consensus and praise for C#'s Rx library (observable pattern).  It makes the entire event system composable which offers a lot of benefits.

Thoughts?

[Ruudjah]

I think tools might solve the orphan objects-with-events problem. If
no reference other then events are present, a debugger might warn. By
the way, I don't understand why the .NET GC does not solve this
automatically. After all, you know if the only references are
subscribed event handlers.

Personally, I read multiple articles and played with some Rx code, but
for some reason I don't grasp Rx entirely, I don't understand them.
What's the difference between observable pattern and events? Afaik my
limited knowledge goes, they are the same.

[Eric J. Smith]

I'm no expert on Rx either, but the primary benefit would be composability.  The ability to do interesting things like aggregate the event callbacks and do things like throttle them, take the 1st event and unsubscribe, delay events, group events, etc.  It really opens up a new world of possibilities.  If Dart gets Mixins implemented, then we could see entire libraries of cool ways to manipulate any observable stream.

There is already an implementation of the Rx library for Dart here:

https://github.com/prujohn/Reactive-Dart 

More than anything, I'm just wanting the Dart team to standardize their patterns so that we have some consistency.