Reactive Programming for Clojure(Script) UIs

[Timothy Baldridge]

I would like to start heading up an effort to get a reliable FRP library built out for ClojureScript and would like some input on my thoughts/conclusions thus far. 

To start with, I have a very simple example solution available here: 

https://github.com/halgari/reactive/tree/master/src 

To start with, let me explain the ideas behind my implementation:

In this situation we are primarily concerned with UI bindings. Therefore the entire system is based on the idea of pushing events to listeners, and then those listeners deref-ing the most recent value from the event streams. This method differs from .NET Rx or Stuart's work in that my implementation does not guarantee that all events will be processed. Instead it tries to ensure that binding constraints are always true. 

Let's give a concrete example. 

Requirements: Create a window that shows "[x y] side" in the title, where x and y are the mouse position, and side is either "left" or "right" depending on the position of the mouse relative to the center of the window.


In .NET's Rx view of this data we would write this (in Clojure Code) somewhat like this:

(bind window :title

    (let [mouse-x (rx/map :x (.toObservable window "mouse-pos"))

          mouse-y (rx/map :y (.toObservable window "mouse-pos"))]

      (rx/map str

          (rx/combine-latest list 

            (rx/combine-latest vector mouse-x mouse-y)

            (rx/combine-latest #(if (< %1 (/ %2 2)) "left" "right")

              mouse-x

              (.getObservable window "width"))))))

          

We we see here that combine-latest applies a function to the latest value from one or more streams, while map applies a function to every value in the stream. The above code expresses the concept very well if we are viewing UI bindings are considered to be event streams. However, there is a cleaner way.

My proposal is to define a very small DSL that (like FrTime), allows users to write UI bindings as if they were normal functions, and then the framework translates these s-expressions into a set of Observables and Observers that are connected to semantically similar to the functional code expressed in the input code. 

Example:

(bind window :title 

   (let [mouse-x (:x window:mouse-pos)

         mouse-y (:y window:mouse-pos)]

     (str [mouse-x mouse-y]

          (if (< mouse-x (/ window:width 2))

              " left"

              " right"))))

Firstly, the macros lift every constant in the input. So 2 becomes a IObservable that always returns the number 2. str becomes an IObservable that always returns the function str, etc. Seqs are considered to be the same as combine-latest in Rx, with the most recent value in the first stream being the mapping function used to combine the other streams. The symbol window:mouse-pos becomes (.getObservable window :mouse-pos). Special macro code is needed for if, and let. 

Unlike FrTime, this DSL does not allow users to include lambdas or loops in binding expressions as this adds a lot of complexity with marginal usefulness. 

My initial prototype linked above is not the cleanest implementation of this, but it's a proof-of-concept. 

I'd like some input on this. I keep hearing people suggest that FRP is a good solution for functional UIs in ClojureScript but thus far no one has stepped up to make it happen. This is my goal.

Thank you for your time,

Timothy Baldridge 

[Kevin Lynagh]

Timothy,

Thanks for taking the time to implement this.

I've been reading about FRP and implementations like FrTime and FlapJax, but I've only written a simple-implementation solution (read: hack) in this space: Reflex (https://github.com/lynaghk/reflex/).

I'll briefly outline how I use Reflex, and I'd be interested to hear your opinion about how Reactive would make some of these operations simpler, safer, faster, &c.

Reflex is based on the "computed observables" of Knockout JS, and the library monkey patches atoms so that dependencies (via atom derefs) are picked up automatically:

    (def a (atom 1))

    (def b (computed-observable (inc @a)))

    @b ;;=> 2

    (reset! a 2)

    @b ;;=> 3

I wrote Reflex to support building UIs and interactive data visualizations in ClojureScript.

Here's an example usage from a todo list demo (https://github.com/lynaghk/c2-demos/tree/master/todoMVC).

(The `bind!` macro uses computed-observable internally):

    (bind! "#main"

           [:section#main {:style {:display (when (zero? (count @core/!todos)) "none")}}

            [:input#toggle-all {:type "checkbox"

                                :properties {:checked (every? :completed? @core/!todos)}}]

            [:label {:for "toggle-all"} "Mark all as complete"]

            [:ul#todo-list (unify (case @core/!filter

                                    :active    (remove :completed? @core/!todos)

                                    :completed (filter :completed? @core/!todos)

                                    ;;default to showing all events

                                    @core/!todos)

                                  todo*)]])

Here, two atoms (`core/!todos` and `core/!filter`) affect the state of this piece of the UI, and whenever either of them update, the entire body gets re-evaluated and th result merged into the DOM.

Internally, the body of `bind!` is just wrapped in a function to create a closure, so there are no restrictions on the forms within it.

The computed-observables in reflex are lazy; they get marked as dirty when any dependencies update, but the fn doesn't get called until the CO itself is derefed.

However, since there is no lifting or other fanciness, the entire fn gets called on deref.

So, if you have

    (computed-observable

      (let [a (expensive-op @state)

            b (+ a (more-expensive-op @moar-state))]

        (* a b @c)))

and only `@c` updates, on deref the expensive operations to calculate `a` and `b` are run even though that's not necessary.

In Reactive all of the forms would be lifted, and only the last one, `(* a b @c)` would be re-run, correct?

So Reactive is potentially faster than Reflex (in situations where the domain computations outweigh the added overhead of lifting and dependency graph construction/walking).

Aside from speed, what are advantages to using Reactive over Reflex?

My gut feeling is that Reactive pieces are more composable, but I'm not sure if this is something I'd use often.

Architecturally, most of the stuff I'm making looks like

    data --bind-form--> UI

with DOM event handlers that update the underlying data in response to user actions.

(When the data is updated, the atoms notify their listeners, which automatically propagates changes to the UI).

There isn't a ton of composition here, but that could be a reflection of the limitations of my current toolkit.

My other question is about wrapping UI widgets to plug into ClojureScript.

From the source, it looks like any widget just calls `core/next-event` to emit events.

Do you have any plans for the other direction; setting the widget state (e.g., <select> options or textarea text values)?

In any case, great work.

I'm definitely excited to check it out and get a feel for what kinds of things it will make simpler / possible.

best,

Kevin

[Paul deGrandis]

It's exciting to see people tackling FRP on ClojureScript.  I thought my experiences might add something to the discussion.

The work I've done in this area is based on pieces from the FrTime papers, Deprecating the Observer Pattern, Safe and Efficient FRP, Iteratees, and the RxJS code base (currently codified in Shoreleave's PubSub system - http://shoreleave.github.com/shoreleave-pubsub/) .

My application flow is similar to Kevin's:
DOM listener -> data function(s) -> … -> DOM update / storage update / client-to-server update
and occasionally:
Server-to-client update -> data function(s) -> … -> DOM update / storage update / client-to-server update

I also have conditions where those updates could trigger further data functions or updates.
Additionally my apps focus more on streams of data (search and process related apps) and less on heavy UI updating.  That is, I'm more sensitive to declaratively binding loosely coupled functions/atoms/"components" across my entire app than I am to binding temporal data stores to UI elements.

I originally started from Tim's position, "I'll just port over the bulk of RxJS."
This was leading to a thorny solution where complexity within the abstractions and approaches was hurting general composability.  It was also ending up too tightly coupled to an implementation (most likely a fault of my implementation).
My design goal was to leverage Clojure's natural functional composition, with a solution built upon sane protocols that was open to future extension.  RxJS could not provide that, but I liked the general approach that it laid out.

What I ended up with was two protocols: IMessageBrokerBus and IPublishable
One describes a broker/bus that routes subscriptions.  For me, this takes the form of a C2 Event Bus with multiple publishers and multiple subscribers (or if you will, observables and observers).  Making this a protocol allows for application specific implementations to be dropped in (i.e.: do you need to use crossdoc messaging?  Do you want the bus to asynchronously process events or should it be synchronous?  Is your bus built upon another JS technology)
The other describes how a given thing publishes to the bus (you build "observables" with it using a decorator, much like how you would memoize a function).  The rest of the event and data logic is achieved by leveraging the CLJS ecosystem (core.logic, standard Clojure functions, etc).

Using these abstractions allowed me to make functions, atoms, and storage all publishable.  That is, a function can publish its results to the bus, and anything else can subscribe to those results (including DOM updating functions).  An atom (or anything that is IWatchable) can publish to the bus, and all subscribers will get updates.  With this approach, I can stitch together completely decoupled components to build out app functionality.  In the above flow, everywhere you see "->" is a subscription in my application.

There are certain tradeoffs with this approach:  All returns *should* be maps to keep components loosely coupled, the bus architecture is inherently eager, and you give up some of the more useful abilities of RxJS (like the "switch" ability to compose 'streams'), but you gain simplicity, easy extension via protocols, and general environment/ecosystem composability.
I haven't tried it yet, but my intuition is that Shoreleave's PubSub pairs nicely with Reflex to tackle a large portion of the FRP landscape.

This isn't so different from what Tim proposed, except that in Shoreleave's PubSub you're *strongly encouraged* to separate the functions that generate the data (building the new title) from those that update the DOM.
Mouse movement event -> Data function that builds a map {:x :y :side} -> DOM updating function (some sort of defaction, or whatever library you use)

Regards,
Paul

[Timothy Baldridge]

I agree, Kevin and I have more or less taken the "port Rx approach".
Paul's Pub/Sub example intrigues me greatly. It's very close to
something I use at work for cross-module communication in C# (.NET's
IEventAggregator).

>This isn't so different from what Tim proposed, except that in Shoreleave's PubSub you're *strongly encouraged* to separate the functions that generate the data (building the new title) from those that update the DOM.
>Mouse movement event -> Data function that builds a map {:x :y :side} -> DOM updating function (some sort of defaction, or whatever library you use)

Perhaps I should back-up and explain what got me interested in this
whole subject.

To start with I was very interested in implementing true Actors in
Clojure. When I say "true Actors" I mean the style of actors described
here: http://www.dalnefre.com/wp/2010/06/actors-in-clojure-why-not/

However, taking a cue from Rich's talk, I sat down and said "what is
complected in Actors?". Well, we've complected state (each actor has
state) with code (each actor has a processing function). This caused
me to start to re-think how I could decouple these two parts. This
lead me to Dataflow programming. In the Actor Model, we cannot
de-couple two actors and insert a different actor between the two
without modifying the logic of these actors. That is, we must tell
actor 1 "Detach from Actor 2 and start sending new events to Actor 3".

Dataflow programming gets around this by abstracting away the
"send-to" functionality. Paul's Pub/Sub code does this as well. We can
add a dozen listeners to a given publisher without ever needing to
alert that publisher. The Pub/Sub example seems to just be a
simplified and yet abstracted version of standard Dataflow. Now that
does not mean that we've now removed all state from nodes in DataFlow.
But at least the subscribers list is not tucked away inside the
"actor's" state. And I like that!

The one thing that bugged me with Dataflow however, is that it's
pretty hard to reason about in text without some decent syntax. This
is the reason for the DSL.

For example, let's say we dropped the DSL, we'd need something like
this in reactive to do a simple "set title to [x y]":

(let [mouse-x (.getObservable window "mouse-x")
mouse-y (.getObservable window "mosue-y")
mapper (lift vector 2)]
(connect mouse-x mapper :0)
(connect mouse-y mapper :1)
(bind window "title" mapper))

This is horribly ugly in my opinion. It's imperative, and doesn't
correctly express the intent of the programmer. It's much simpler to
write:

(bind window "title" (vector window:mouse-x window:mouse-y))

But none of this stops us from adopting the simplified Pub/Sub system
in the background. This DSL is implementation agnostic, and the
Pub/Sub system is generic enough to support any sort of front-end we
desire.

Well these are my first set of thoughts. I'd love to get anyone else's
input. In the mean time I'm going to think about this for a few days,
and then see what a pure Clojure/JVM Pub/Sub system would look like.


Timothy

[Brandon Bloom]

I don't know about RxJS, but the shipped/production .NET Rx is built on "Px", the Parallel Extensions. One talk on Channel 9 suggested that the Task abstraction and C# 5's async/await provided a substantial complexity and code size reductions. Although C#'s compiler implements async functions as a switch-statement-state-machine, I think Clojure(Script) would prefer a CPS transform. In my mind, the path towards an FRP GUI looks like this:

1) Implement reliable CPS transform

2) Generalize "promise" to "task" which is just a promise with a continuation callback

3) Implement async, defasync, and await (warning: name conflict for agent's await)

4) Define observable/observer/etc abstractions

5) Implement higher order reactive operators

6) Figure out how the heck to make this all work with the DOM

Does this seem reasonable? I'd be happy to chat with anyone about any/all of these steps to see how we can drive this effort forward.

[Timothy Baldridge]

> I don't know about RxJS, but the shipped/production .NET Rx is built on
> "Px", the Parallel Extensions. One talk on Channel 9 suggested that the Task
> abstraction and C# 5's async/await provided a substantial complexity and
> code size reductions. Although C#'s compiler implements async functions as a
> switch-statement-state-machine, I think Clojure(Script) would prefer a CPS
> transform. In my mind, the path towards an FRP GUI looks like this:
>

But do we need all that? I'm very hesitant to take anything MS does as
"the best way" or even as "a good way".

Look for example, at the complexity of IObservable<T>, IObserver<T>
and IDisposable<T> compared to Paul's Pub/Sub example. Now, if there
is a reason for that complexity, that's fine, but we should do
ourselves the favor of closely evaluating each and every aspect of an
implementation before we just jump into it "whole hog".

Let's remember that Rx is built for C# (and VB.NET) and C# is an
imperative, mutable-by-default, language with no concept of time. I
question how much of C# we should pull into Clojure, if any at all.
And this comes from someone who spends about 8 hours a day writing C#
apps.

Timothy

[Paul deGrandis]

On Monday, July 16, 2012 11:35:41 AM UTC-4, Brandon Bloom wrote:

I don't know about RxJS, but the shipped/production .NET Rx is built on "Px", the Parallel Extensions. One talk on Channel 9 suggested that the Task abstraction and C# 5's async/await provided a substantial complexity and code size reductions.

 

I dug through some Px stuff as well (all the citation material listed in the Rx readings), but I didn't study anything in depth.  Is there anything specific I should read?
 

Although C#'s compiler implements async functions as a switch-statement-state-machine, I think Clojure(Script) would prefer a CPS transform. In my mind, the path towards an FRP GUI looks like this:

1) Implement reliable CPS transform

I don't want the approach to be tightly coupled to a strategy.  We should be able to decide: async via CPS or JS events, synchronous, http web messaging backed, etc.
Tightly coupling to an implemented strategy is going to hamper future proofing and limit the reach and effectiveness of the system.  Making the underlying infrastructure as open for extension as possible needs to be a design goal.
 

2) Generalize "promise" to "task" which is just a promise with a continuation callback

3) Implement async, defasync, and await (warning: name conflict for agent's await)

I think the names should be different, but yes you're agreeing with Tim and I - a verb to participate in the system, a macro or DSL to simplify it, and some way to capture the "events" you're looking for (potentially blocking on them).

4) Define observable/observer/etc abstractions

I'm not sure what the best name here is either, but as discussed in Tim's latest message, at the heart of these abstractions is nothing more than dataflow.  What do you think are the tradeoffs in centering the abstractions around that and building the reactive pieces upon those abstractions?
 

5) Implement higher order reactive operators

Which of the higher-order reactive operators do you find yourself using most (or would you envision using most).  How are these composable with non-reactive higher-order functions?

6) Figure out how the heck to make this all work with the DOM

I actually think this is the one we all feel is nailed down - the DOM enters the system via DOM-event listeners.  How you use that listener and how you later update the DOM is dependent only on the libraries you choose; there should be no forced way the DOM gets updated.  Some apps are easier with Singult/C2, some are easier with crate, and some people prefer building UIs with Enfocus.
The FRP system should allow for easily integrating all these technologies when and where it makes sense to do so.
 

[Brandon Bloom]

But do we need all that? I'm very hesitant to take anything MS does as
"the best way" or even as "a good way".

Of course we shouldn't assume anything is a good way, but I wouldn't discount the great work the C# and BCL teams are doing just because they are doing that work for Microsoft. As of v3, C# is basically "good ideas from the FP world packaged up for the intermediates in the OOP world". There's a lot of good thinking there that is worth studying.

 

Look for example, at the complexity of IObservable<T>, IObserver<T>
and IDisposable<T> compared to Paul's Pub/Sub example. Now, if there
is a reason for that complexity, that's fine, but we should do
ourselves the favor of closely evaluating each and every aspect of an
implementation before we just jump into it "whole hog".

Agreed. Allow me to provide some additional commentary:

RE #2: There is *a ton* of async callbacks in typical JavaScript (and therefore ClojureScript) code. One successful approach to dealing with that callback soup is promises. JVM Clojure provides a promise mechanism, but it's semantics are tuned for multithreading. In particular, they do not provide an easy callback mechanism and has an exception-based re-throwing error handling mechanism. I'm proposing a generalization from "promise" to "task" which provides an easy way to register a single continuation callback. See Task and it's ContinueWith methods. I think much of the cancelation, error handling, and other stuff in Px can be mostly ignored or greatly simplified in Clojure.

RE #3: Once you have a generalized promise with a callback, you can conceive of forms like let-async and let-parallel which run code in their respective asynchronous styles coercing promises to bindings. Unfortunately, async is like a virus. If you call an async function, you become an async function (ignoring blocking in a multithreaded environment). So you want a way to have your function automatically converted into an synchronous function which operates on promises as if they were fully realized and has a return value which is automatically delivered to a promise as well. This, by the way, could be interpreted as a sort of async- or promise-monad. I want to be able to write functions that look like this:

(defasync f [x]

  (let-parallel [y (remote-call :bar)

                 z (remote-call :foo)]

    (* x y z)))

When called, this code should immediately return a promise. If called within the context of some macro like the C# await keyword, then it should be an error. Unless that macro is used within some other async macro, then it should be a shift/reset point for a delimited continuation, which brings me back to #1.

#4 and #5 are all a little fuzzier in my mind. I haven't given them as much deep thought yet. I've played with various observer/observable patterns trying to find one I really like, but the need for an unsubscribe or dispose seems to always get messy fast. Rich Hickey has mentioned Erik Meijer's Rx design and the potential push dual to pull sequences. One thing, however, is clear to me: Reactive is inherently in the asynchronous domain. In my toying around, I found myself wishing for the task abstraction while implementing some reactive operators.

Obviously #6 is going to require A LOT more though.

Let's remember that Rx is built for C# (and VB.NET) and C# is an
imperative, mutable-by-default, language with no concept of time. 

I question how much of C# we should pull into Clojure, if any at all.

I agree completely. I think Clojure is inherently better suited to the types of abstractions that are needed here. I think that the stack I've laid out is relatively disjoint from Microsoft's work, other than using some of it's names for things. Microsoft is reading the same FP literature that the rest of us are, but they went ahead and built and shipped a pretty robust Rx library that is in use by a large number of developers, with a less powerful language. I don't know of anyone else who has done that yet in this domain. That's practical, real world experience that we can't afford to ignore.

 

And this comes from someone who spends about 8 hours a day writing C#
apps.

Full disclosure: I used to work on Visual Studio integration at Microsoft on Xbox and XNA (C#). However, I had no direct contact with any of the Px/Fx guys. And to save my geek cred: I left Microsoft and founded a startup which didn't touch a single Microsoft technology for its 2 year lifespan (minus some IE and Outlook testing).

[Brandon Bloom]

I dug through some Px stuff as well (all the citation material listed in the Rx readings), but I didn't study anything in depth.  Is there anything specific I should read?

Hm... maybe Eduasync is a good start? It's been a while since I read it, but I remember Edulinq being very enlightening.

 

I don't want the approach to be tightly coupled to a strategy.  We should be able to decide: async via CPS or JS events, synchronous, http web messaging backed, etc.
Tightly coupling to an implemented strategy is going to hamper future proofing and limit the reach and effectiveness of the system.  Making the underlying infrastructure as open for extension as possible needs to be a design goal.

There are two categories of abstractions needed:

1) The oposite of seq. That is some sort of push model, rather than pull model for a sequential data source. Whatever this protocol is, it should be disjoint from it's implementation strategy, so it can use any sort of backing event emitter.

2) A model of control flow. I think this is where C#'s async/await pair of keywords is an extremely well designed approach. This can be implemented by mechanical transformation into A) continuation passing style (my suggestion), B) a state machine, C) some other form I haven't considered or D) some hodgepodge of all of the above as an optimization scheme. There may be an alternative other than the await/async pair, but I haven't found nor can conceive of one which so nicely preserves the traditional flow of code.

 

Define observable/observer/etc abstractions

... 

What do you think are the tradeoffs in centering the abstractions around that and building the reactive pieces upon those abstractions?

As opposed to which alternative?

 

5) Implement higher order reactive operators

Which of the higher-order reactive operators do you find yourself using most (or would you envision using most).

Underscore.js provides a bunch of reactive-style functions we used *a lot* in our frontend. For examples, see debounce, throttle, and once. I'd imagine we'd have used much more if we had written our code in a style more similar to that "deprecating the observer pattern" paper.

 

How are these composable with non-reactive higher-order functions?

In the same way that unary and binary functions are composable with map, reduce, filter, etc.

 

6) Figure out how the heck to make this all work with the DOM

I actually think this is the one we all feel is nailed down

I need to think on this more deeply. 

[Timothy Baldridge]

First of all, I should mention that in my day job I primarily write
back-end C# code on a in-house business app. So thankfully we both
have a common ground of understanding the .NET platform.

I'm going to address Brandon's comments a bit out-of-order:

>> As of v3, C# is basically "good ideas from the FP world packaged up for the intermediates in the OOP world". There's a lot of good thinking there that is worth studying.

Agreed, but let's remember, that what we're talking about is something
different. So C# took some FP ideas and brought them into the OOP
world. Great! But should we really be looking at C# then as a source
of code that we want to pull back into a FP language? I imagine that
MS took what they liked from FrTime and shoehorned it into C#. So I'm
simply saying, instead of going off of this shoehorned version, let's
look back at their source and start from there.

>> 1) The oposite of seq. That is some sort of push model, rather than pull model for a sequential data source.

I think this is digging into the implementation to early. A
"inside-out-seq" is just one way of looking at this. WPF's XAML
binding system is another method. Callbacks are another. I think the
main goal here is to simply say "how can we wrap as much of the
imperative nature of UI into a library and hide it from the
programmer, presenting only a functional interface, while still
remaining extensible and simple?"

>> Unfortunately, async is like a virus. If you call an async function, you become an async function (ignoring blocking in a multithreaded environment).

Agreed, but I question how hard/easy this would be in a dynamic
language like Clojure. For instance, await/async in .NET required a
compiler update, and even then in a static language it's easy to fail
when the user forgot to put in a "await". and ends up returning a task
from a function instead of returning the result of the task.


---

In all of this I can't help but wonder if Rx isn't just a "elegant way
to hack reactive code into C# using Linq". When I see the need for
function transformations, await/async, etc. I can't help but wonder if
there isn't a simpler way. When read up on the features of Rx (or any
new feature in a language), I have to ask myself "would this have been
easier in a language with persistent structures, and STM?".

This is where I start to look the Pub/Sub approach and see the
following features:

1) We've de-coupled the scheduling from the function sending the
event. By swapping out the buss we can get multi-threaded execution,
network transparency, etc.
2) We've de-coupled senders from receivers. We could have 20 listeners
listening to a single event, and they could be
subscribing/unsubscribing rapidly, and the publisher need never know
(ie. we get rid of all the IDisposable stuff)

Yes, async is cool, I'm just wondering if it's really what we're
looking for here.

And just so I don't sound biased, I think FrTime is cool but just way
to complex. It's a very feature complete solution, but I think it
sacrifices simplicity in the process.

There has to be a happy median...


Timothy

[Timothy Baldridge]

Over the past week I've done some more reading up on async, Task<T>
and Rx in .NET. I've also taken a look at lamina:
https://github.com/ztellman/lamina/wiki/Introduction

Lamina implements most of what Brandon was advocating from the Rx
point of view (including a async macro). I'm not exactly sure how easy
it would be to port to cljs, but it looks like it would take a fair
amount of time.

After some more thinking on the issue, my DSL solution won't work with
a Pub/Sub implementation. This is due to the way lifted constants are
handled. In the current code, when we lift a constant we are basically
creating a node that allows observers to attach, but the moment they
do, the lifted variable fires off an event. In the Pub/Sub solution we
can't do this since we don't know when subscribers attach to our
topic. My solution will also have a bit of an issue when it comes to
dealing with async calls, so some work would need to be done there.

From looking at the Lamina source, I see that it takes a position
fairly close to Rx by using Tasks and connected objects to represent
the graphs.

I'm going to back-peddle on my past comments about async/await: I
agree with Brandon that in CLJS these would be a great feature to have
available.

In the end we have three types of problems we're trying to solve here:

1) single value bindings. Think of this as my original example. We're
only interested in the current mouse position
2) event streams, if the user clicks a button three times we may want
all three clicks
3) async events. An auto complete box is a good example of this

So I guess I'm with Brandon on this now. I see that we need the following:

1) a library for handling event streams (easy enough on cljs due to it
being single threaded)
2) a way to integrate async tasks into this pipeline. On cljs this
could be as easy as extending (task) to support IObservable
3) a DSL to tie it all together.

At first I was against Rx since it is a bit convoluted for a simple
single value bindings. However, once we introduce async calls into the
mix (and we have no other choice in the browser), we are now looking
at a different paradigm.

At this point I'm thinking we should just give it a go and see how far
it takes us. We'll stick to Rx more-or-less, but improve on it where
we can. We'll code this for ClojureScript and that will simplify
things greatly as we won't have to worry about multiple threads.

I'll start up a project in the next few days and try to lay out the groundwork.

Timothy

[David Nolen]

It might be useful to put together a Confluence page that documents /
summarizes the various topics / approaches covered in this thread.

David

> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>

[Brandon Bloom]

Lamina implements most of what Brandon was advocating from the Rx
point of view (including a async macro). I'm not exactly sure how easy
it would be to port to cljs, but it looks like it would take a fair
amount of time.

Just looked at Lamina briefly for the first time right now. My initial reaction is "whoa, this seems like a lot of stuff! What does all this do?" I'm advocating an extremely carefully layered approach. This seems like a big bag of evented programming related stuff. We need to break it down even more than I suggested. But I think at the high level, it makes sense. At least in my head. I've got about 3 open projects to conclude before I can really dig into this stuff...

In the meantime, I'd simply implement promise in CLJS as (defn promise [] (continue-with (task) identity)) and explore from there. continue-with is Microsoft's name, but I think we could do better (ie. shorter).

[Timothy Baldridge]

It might be useful to put together a Confluence page that documents /
summarizes the various topics / approaches covered in this thread.

David

Done: http://dev.clojure.org/display/design/Reactive+Programming

Everyone please feel free to comment on the page, or correct any errors I have.

Timothy

[Paul deGrandis]

I apologize for my late reply, I was on vacation and just got caught up on the thread this morning.

On Tuesday, July 24, 2012 9:18:07 AM UTC-4, Timothy Baldridge wrote:

Over the past week I've done some more reading up on async, Task<T>

 

Can you point me in the direction of the specific async, Task<T>, and Rx readings you've been digging through?
I just want to make sure I have the same frame of reference for the conversation.

 

After some more thinking on the issue, my DSL solution won't work with
a Pub/Sub implementation. This is due to the way lifted constants are
handled.

Can you elaborate more on what you mean by lifted constants?  Are you just talking about items that have been decorated to publish?
 

In the current code, when we lift a constant we are basically
creating a node that allows observers to attach, but the moment they
do, the lifted variable fires off an event. In the Pub/Sub solution we
can't do this since we don't know when subscribers attach to our
topic.

Using the protocols for the Pub/Sub system I built, you could easily write a bus implementation that published when a subscription happened. (More on this further in the reply).  You could even make a bus that was publishable, allowing for components to subscribe to every event in the bus.

In the end we have three types of problems we're trying to solve here:

1) single value bindings. Think of this as my original example. We're
only interested in the current mouse position
2) event streams, if the user clicks a button three times we may want
all three clicks
3) async events. An auto complete box is a good example of this

Good!  This is awesome to see three different classes of applications we need to target.
Implementing a bus on top of something like ClojureScript:One's event abstraction, or even cross-document messaging can achieve all of those three.  The entry point for all three cases is a listener on a DOM event.

1) a library for handling event streams (easy enough on cljs due to it
being single threaded)

 

I agree we need something to handle event streams, but the assumption that CLJS (or just JS) is single-threaded should be avoided.  Shoreleave allows for ad-hoc web workers to be created on the fly (via automatic embedded worker construction behind the scenes).  People are going to start using web workers (or some multi-process/multi-threaded approach) and WebCL/WebGL to push the browser for their apps.
 

2) a way to integrate async tasks into this pipeline. On cljs this
could be as easy as extending (task) to support IObservable

Are you saying you want some tasks to be async and others to be synchronous?
I would think you'd want a "Task" and the bus/foundation that handles those tasks to be async or synchronous.
 

3) a DSL to tie it all together.

Agreed.

At first I was against Rx since it is a bit convoluted for a simple
single value bindings. However, once we introduce async calls into the
mix (and we have no other choice in the browser), we are now looking
at a different paradigm.

Where in the code are the async calls happening?  For example, are you making async calls within a function where you later want to join the results before the function returns?  Or do you just want the functions themselves to be async (that is, the functions that operate on event streams do so asynchronously)?
Take a look at how CLJS:One's event abstractions work. Does that achieve what you need in terms of the goals above and the ability to operate asynchronously?  I think the base abstractions in Rx are good, but at that level, you're just using a different vocabulary to express a C2 event bus.

The ideal solution for me would be something like Rx ala carte - I can add on more pieces as I go, I'm not forced to adorn all my code with Rx specific verbs, any piece of the language can participate in it (functions, IWatchables), and I can adjust the implementation details to better fit my application.
 
 ------

While on vacation I thought over how one might express and compose higher-order reactive functions with something that is bus oriented.  The solution I came up with was that these higher-order operations take publishables and return a publishable function:

  (subscribe bus (once funcp) my-subscribing-function)

Underneath lies a protocol (call it IReactiveOnce) that defines one function (-once).  Buses can optionally implement bus-specific implementations (like `subscribe-once` for the Closure Pub/Sub).  The default implementation would just rely on promises, allowing all buses to participate.
This approach works for the two test cases I was using: once and switch.

There are more details to this approach, but I want to code up a prototype first so we can all look at something tangible.

I'm very excited to see we're getting somewhere on this, it seems we're all honing in on a few key pieces.
I've recently gone back to re-read The Art of the Metaobject Protocol for inspiration, and that's proving to be a great source of insight for me personally with regards to this reactive design discussion.

Paul

[Paul deGrandis]

A quick update:
After using a mix of promise, delay, and the basic bus protocol functions, I was able to get pretty far.

I started thinking more about Task, `async`, and Brandon's request to change promise.
I'm coming to the conclusion that I think promise should be left as is, but perhaps we need something like clojure.core.control - continuation based control flow functions and CPS transformations.  Thoughts?

Below is my current reading list that I've been working through:
The F# Asynchronous Programming Model - D. Syme et al

Composable-continuations-tutorial - Scheme wiki
Syntax Matters: Writing abstract computations in F# -  T. Petricek, D. Syme
Implementing First-Class Polymorphic Delimited Continuations by Type-Directed Selective CPS-Transform - T. Rompf, I Maier, M. Odersky
Threads yield continuations - S. Kumar et al
Handling Control - D. Sitaram
Shift to Control - C. Shan
A static simulation of dynamic delimited control - C. Shan
Yield: Mainstream Delimited Continuations - R. James, A. Sabry

Kilim: Isolation-Typed Actors for Java - S. Srinivasan, A. Mycroft
Thorn - Robust, Concurrent, Extensible Scripting on the JVM - Bard Bloom et al

I'd also like to thank David and Brandon for their work on delimc (delimited continuations for Clojure) - which has provided me with a base implementation on which to reference and expand.

Regards,
Paul

[Zach Allaun]

I'm going to jump in :-).

 

I started thinking more about Task, `async`, and Brandon's request to change promise.
I'm coming to the conclusion that I think promise should be left as is, but perhaps we need something like clojure.core.control - continuation based control flow functions and CPS transformations.  Thoughts?

 

I think that this would be valuable. I recently undertook a port of async.js over to cljs, mostly as a learning exercise (I'm still relatively new to all of this). Even after only porting about a third of the library, it's becoming very obvious that there is a need for serious abstraction, as the direct port feels very clunky in clojurescript. 

I wouldn't feel particularly comfortable making design decisions on something like this, but I would love to help where possible, and I have a lot of time to do so. (I'm at Hacker School in ny for the next month, and could probably also commission one or two others to help on something like this.) Could someone with a bit more experience in this area chime in with some suggestions on how to proceed? For now, I'm going to start working through some of the readings that Paul mentioned.

Regards,

Zach

[Timothy Baldridge]

I got thinking tonight that we could prototype this async stuff very easily in clojure-py. Crash course in Python's yield keyword:

In python, any function that contains the bytecode "YIELD_ITEM" is considered a generator. These generators are also supported in clojure-py functions:

(defn generator-test [x]
  (dotimes [y x]
    (py.bytecode/YIELD_ITEM y)))


(vec (generator-test 3))
; [0 1 2]

Now, that's all well and fine, but in Python 2.6 they turned YIELD_ITEM into an expression.

Example:

;; assume we have a yield macro for the bytecode above

(defn gen-expr [x]
  (yield (yield x)))

(defn z (gen-expr 3))
; returns <generator object>

(.next z)
; returns 3

(.send 42)
; returns 42

That's right. The Python VM supports a form of co-routines directly in the VM.

Anyways, here's an example of the .NET style async functions in clojure-py: https://gist.github.com/3213067

If you want to play around with this a bit, do a checkout of https://github.com/halgari/clojure-py and then do:

python clojure.py ~/async-test.clj

Futures were just recently added, so you'll need to get the latest from github.

Anyways, this is a way to prototype the async stuff before we actually think about writing an CPS transformer in clojure.

Timothy

[Brandon Bloom]

Neat idea!

However, this sort of experimentation is exactly why I wanted a robust CPS transform available to all Clojure implementations. It would be reasonably performant for async on all platforms, but would also have many other uses outside of async.

I took a crack at let-async and let-parallel.  Here's my fork of your gist: https://gist.github.com/3214278

I'm exhausted, so I'm going to stop hacking. I qot some weird timing values, but I'll let you play with this, since I was just messing around. I also filed a number of bugs against clojure-py.

[David Nolen]

On Tue, Jul 31, 2012 at 2:38 AM, Brandon Bloom <snpr...@gmail.com> wrote:
> Neat idea!
>
> However, this sort of experimentation is exactly why I wanted a robust CPS
> transform available to all Clojure implementations. It would be reasonably
> performant for async on all platforms, but would also have many other uses
> outside of async.

I've been keeping a close eye on Chris Frisz's clojure-tco. Chris is one Dan Friedman's students so I suspect this will eventually (if doesn't already) qualify as a robust CPS transform.

David

[Paul deGrandis]

Timothy - Python's yield coroutines are a subset of proper delimited continuations.  I had a very similar idea originally, but realized I could build out a proper yield or reset/shift continuation based on the works in the papers cited above.

My suggestion is the same as Brandon's, but a step back.  Let's forget about async first - let's build out proper CPS transformations and delimited continuations, then build on top of that (limited `yield` coroutines, `async`, etc.)

Implementing the async stuff using futures is also backwards.  From the F#/.NET Asynchronous Programming Model:
Task generators are split into two groups - "hot tasks" that run immediately (ie: futures) and "cold tasks" that must be started explicitly (something like `delay`)

These tasks are continuation-backed units of work (akin to delimited continuation capture).  Once you have that foundation, you can build everything else with it (we could even introduce a pool of green threads if we really wanted to).

-----

Last night I built out an event-backed bus using my bus protocol (it actually extends the bus to any EventTarget), but I don't have any numbers or examples using it yet.  See the pubsub in Shoreleave for details: https://github.com/shoreleave/shoreleave-pubsub

Paul

[Paul deGrandis]

I've been keeping a close eye on Chris Frisz's clojure-tco. Chris is one Dan Friedman's students so I suspect this will eventually (if doesn't already) qualify as a robust CPS transform.

Thanks for bringing this up - very cool stuff.

[Timothy Baldridge]

> However, this sort of experimentation is exactly why I wanted a robust CPS transform available to all Clojure implementations. It would be reasonably performant for async on all platforms, but would also have many other uses outside of > > async.

I agree, complex lazy-seqs, async, and lightweight actors are all quite easy once we have proper CPS transforms. CPS does seem to be a common need behind all the ideas we've come up with, so I agree, we should start there. 

I'm not quite read up enough on CPS to understand the clojure-tco code. Does anyone know if it's generic enough to reuse? TCO seems to be a subset of general CPS. 

Timothy

[David Nolen]

On Tue, Jul 31, 2012 at 9:36 AM, Timothy Baldridge <tbald...@gmail.com> wrote:

I agree, complex lazy-seqs, async, and lightweight actors are all quite easy once we have proper CPS transforms. CPS does seem to be a common need behind all the ideas we've come up with, so I agree, we should start there. 

I'm not quite read up enough on CPS to understand the clojure-tco code. Does anyone know if it's generic enough to reuse? TCO seems to be a subset of general CPS. 

My impression that the CPSer in CLJTCO has been done right and includes modern optimizations like minimal thunkification etc. The repo has a couple of readings. I think Friedman et al's Essentials of Programming Languages is a good resource on CPS.

My only concern is that I think it uses a custom AST representation. It would be nice to finally wrap up what we want CLJS's analyzer to produce and then see if the CPSer in CLJTCO can be made to work with it.

David

[Paul deGrandis]

My only concern is that I think it uses a custom AST representation. It would be nice to finally wrap up what we want CLJS's analyzer to produce and then see if the CPSer in CLJTCO can be made to work with it.

It would definitely be ideal to hook this into the CLJS analyzer (once the general format can be decided upon) - which is very similar to what's under the hood for .NET's asynchronous programming model as far as I understand it.

As stated before, it sets the stage for async, yield, and a host of other controls.  When paired with some form of dataflow abstraction, we have all the building blocks to fully tackle RFP in whatever manner we see fit.

Paul

[Brandon Bloom]

My only concern is that I think it uses a custom AST representation. It would be nice to finally wrap up what we want CLJS's analyzer to produce and then see if the CPSer in CLJTCO can be made to work with it.

Not only is it custom, but it's also very, um, un-idiomatic, to put it politely. I had a really hard time following through the code because it uses and abuses protocols, records, etc. But that's just incidental complexity. In terms of inherent complexity, I couldn't quite wrap my head around the literature referenced. It seemed to assume a lot of prior knowledge and notation that for foreign to me. The resulting code uses a lot of symbols (often single-character) which, I assume, have meaning inherited from the literature and Chris & Dan's research. It would be really nice if we could help him help us by encouraging and supporting him to port to the ClojureScript AST and then regroup on next steps.

[Brandon Bloom]

My suggestion is the same as Brandon's, but a step back.  Let's forget about async first - let's build out proper CPS transformations and delimited continuations, then build on top of that (limited `yield` coroutines, `async`, etc.)

Yeah, Async is almost trivial to implement in terms of shift & reset !

 

Task generators are split into two groups - "hot tasks" that run immediately (ie: futures) and "cold tasks" that must be started explicitly (something like `delay`)

 

How is a hot task different than a cold task that is started right after being created? I presume there is a small performance optimization available, but is there any difference in terms of the abstraction itself?

These tasks are continuation-backed units of work (akin to delimited continuation capture).  Once you have that foundation, you can build everything else with it (we could even introduce a pool of green threads if we really wanted to).

This would be super cool coupled Clojure's data immutable, persistent, and printable/readable data structures. We could push the agents infrastructure a bit harder and begin to approximate some Erlang-esque infrastructure.