ClojureScript def, vars, and binding

[Brandon Bloom]

The ClojureScript wiki states that "the user experience of [binding] is similar to that in Clojure" but my very first experiment produced wildly different results between platforms.

Here's a Clojure on the JVM session:

user=> (import java.lang.Thread)

java.lang.Thread

user=> (defn set-timeout [ms fn] (.run (Thread. #(do (Thread/sleep ms) (fn)))))

#'user/set-timeout

user=> (def x "top level")

#'user/x

user=> (binding [x "in binding"] (println x) (set-timeout 1000 #(println x)))

in binding

in binding

nil

And here's the analogous ClojureScript session:

ClojureScript:cljs.user> (def x "top level")

"top level"

ClojureScript:cljs.user> (binding [x "in binding"] (println x) (js/setTimeout #(println x) 1000))

in binding

21

top level

So ignoring the sequencing and nil vs timeout-id return values, the binding of 'x wasn't preserved in the asynchronous callback.

I raised this issue in #clojure and @dnolen said that "that's the behavior there's nothing much to fix", but that didn't sit right with me. This seems like either 'binding is bugged, or maybe I don't understand something about its intent.

On the topic of "Vars" proper, I understand their usefulness in repl-centric development, where you can redefine functions at runtime. The wiki also makes some mention of this, but I can't wrap my head around the context and jargon. I've run into this problem before in Javascript, where some level of indirection is necessary to support run-time redefinitions. You can't do `var fn = package.fn;` and dynamically redefine `fn` from `package` later because a copy of the reference is made. How does ClojureScript address this problem?

Cheers,

Brandon

[Chris Perkins]

You should be calling Thread.start in the clojure version, not Thread.run.  Your set-timeout is just blocking for a while and then running the passed-in function on the caller's thread.

- Chris

[Brandon Bloom]

D'oh! That's what I get for assuming I remembered the Java Thread API from several years ago....

Thanks.

[Brandon Bloom]

For anyone else stumbling across this thread, I found a lot more information about dynamic binding with respect to asynchronous code in the Clojure Confluence wiki:

http://dev.clojure.org/display/design/State%2C+Concurrency%2C+and+Parallelism

In particular, the "Asynchronous Events", "Blocking vs Nonblocking Reads", and "Improve Bindings" sub-pages discuss my misconception regarding the behavior bindings within callbacks, or otherwise lazy code.

The topics haven't been updated in a while, but I suspect that these issues are going to become higher priority as ClojureScript gains adoption. The async and non-blocking nature of JavaScript introduces some serious complexities which aren't as relevant on the JVM.

[Brandon Bloom]

It seems like the missing word from my vocabulary was "conveyance".

I've spent a fair bit of time toying with Clojure, ClojureScript, and their respective implementations. It seems like Clojure 1.3 augmented futures and agents with binding-conveyor-fn, clojure.lang.Var/getThreadBindingFrame, and clojure.lang.Var/resetThreadBindingFrame.

Despite the lack of threading, it seems like this conveyance concept has wide applicability for the common asynchronous code typical in Javascript. In particular, I'd like to be able to use some dynamically bound variables in callbacks from XHR requests. This would also be extremely useful for dynamic var support in Continuation Passing Style transformation macros.

The JVM/Clojure implementation of Var's binding Frame class make clever use of persistent maps to make this machinery work, but it appears that ClojureScript uses simple Javascript objects/maps as namespaces. Capturing a binding frame would involve an O(vars) operation.

Before I dig any deeper into this (including potentially implementing conveyance in ClojureScript), I'd appreciate if anyone could fill me in on any current thinkings here. I'm sure async is interesting to the key ClojureScript contributors, but I've found precious little discussion about these topics.

Thanks,

Brandon

[Brandon Bloom]

To redeem myself for the noob threading mistake, I've implemented Var and friends in ClojureScript:

https://github.com/brandonbloom/clojurescript/compare/8ba4849e60e5957cdac36ef6946c647e824ca3c8...vars

This branch includes (almost) the full set of relevant functionality, but for performance and interop reasons, only applies to vars def-ed with ^:dynamic.

Here's a repl session:

ClojureScript:cljs.user> (def ^:dynamic x "root binding")

(var cljs.user.x)

ClojureScript:cljs.user> x

"root binding"

ClojureScript:cljs.user> #'x

(var cljs.user.x)

ClojureScript:cljs.user> ((binding [x "dynamic binding"] (fn [] x)))

"root binding"

ClojureScript:cljs.user> ((binding [x "dynamic binding"] (bound-fn [] x)))

"dynamic binding"

Complete list of newly supported forms:

• var
• alter-root-var
• bound-fn
• bound-fn*
• bound?
• get-thread-bindings
• push-thread-bindings
• thread-bound?
• var-get
• var-?
• with-bindings
• with-bindings*

I'm filling out my contributor agreement and will start a parallel thread on clojure-dev to see if we can get this integrated!

Cheers,

Brandon

[David Nolen]

This is interesting. However it seems, at least to me, like a big change with too little justification. When you're CA is in, please setup a design document for this line of development - http://dev.clojure.org/display/design/ClojureScript.

David

Brandon

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Brandon Bloom]

Can we continue the conversation on this thread while my CA waits on the USPS?

Three things to cover: 1) motivation 2) design 3) impact

1. Why do I want this? Why would anyone want this?
• Better parity with JVM Clojure
  
• async Javascript (ie. nearly all Javascript) makes the binding macro effectively useless without bound-fn
• Some common Clojure behaviors, like printing to a dynamically bound *out*, can't currently exist across an async boundary. See *print-fn* in CLJS
• I'd like to create a dom manipulation library that has a *dom* dynamic variable, which acts like current working directory for dom manipulation functions. I've got some crazy ideas that I want to experiment with here, especially once I implement add-watch: I think I can achieve pretty seamless UI data binding.
2. Design
• Are Vars still useful without threads?
• Yes, but only dynamic vars
• async callbacks and threads have a lot of common design considerations
• Performance
• Price is equivalent to that of JVM Clojure
  
• Extra indirection for def and deref
• Shared stack of dynamic binding frames
  
• Hash lookup on each access
• Opt-in price for the Var indirection
• Treat ^:dynamic as that opt-in mechanism; no impact for static vars
• Potential optimization: Leverage Javascript's prototypical inheritance instead of Frame type
• Required compiler analysis
• Metadata for resolved vars
• Not available for external libraries
• OK, because we only care about ^:dynamic vars
• Interop
• Vars declared as ^:dynamic differ from static ones: they are wrapped in the Var type
• binding, etc are only applicable to dynamic vars, not useful to non-ClojureScript callers
• External callers can still use deref and alter-var-root
3. Impact
• Breaking change for binding macro
• New behavior matches JVM clojure: binding only works on dynamic vars
• Simply marking any affected vars as ^:dynamic should be enough to upgrade
• Potentially breaking changes for any cljs.core vars that are changed to ^:dynamic
• *print-fn* and other printing vars
• Only breaking for Javascript interop usages, still source compatible

How's that? Anything I missed? Complaints? Concerns? Questions?

Cheers,

Brandon

[David Nolen]

I put your notes here, http://dev.clojure.org/display/design/Dynamic+Binding

How are you ensuring that the binding frames are local to a particular asynchronous block of code and that they are removed when that asynchronous block of code exits?

David

Cheers,

Brandon

--

[Brandon Bloom]

> I put your notes here,
> http://dev.clojure.org/display/design/Dynamic+Binding

Thanks!

> How are you ensuring that the binding frames are local to a particular
> asynchronous block of code and that they are removed when that asynchronous
> block of code exits?

I don't do anything special for asynchronous code, this simply provides the primitives necessary. Which is, primarily, bound-fn. Which is, in turn, built on {get,pop,push}-thread-bindings.

The logic is all taken from the JVM clojure implementation, with the main difference being that I didn't need the TBox class (see Var.java) since
there are no threads in JavaScript land. bound-fn (and the binding and with-bindings macros) simply utilize the low level push/pop/get-thread-bindings to capture the immutable bindings hash as well as apply them when necessary. Finally blocks handle most of the popping magic... assuming that machinery works in all the various edge cases already (if not, it's an existing compiler bug).

There is some future work, which requires far more thinking, to accomplish something akin to C#'s async/await keywords. These would likely be non-blocking/async variants of the futures/defer theme. Much more thought is required here before tackling that work. In the meantime, you can now use bound-fn instead of just #() or fn when defining callbacks, if you want to preserve the binding frame asynchronously.

Does that answer your question?

[Brandon Bloom]

As I experiment with this more, I could see non-dynamic Var objects also being useful once IWatchable is implemented on them.

Basically, I'd love to send a defn form to the browser during development and have my UI data-bound to both the data AND the function that rendered it :-) My dream is to be able to have a list of foo-widgets deep in the page and have them all update instantly to the latest code... without losing my scroll position!

Anyway, with this in mind, I'd like to also bring into consideration the possibility of both static and dynamic Var objects. With my current design, this has performance implications (an extra indirection for all lookups and function calls). As well as interop implications (def-ed values need to be deref-ed explicitly by non-cljs callers).

One idea would be to provide alternate metadata, like ^:export. Not sure what the right name or default for that metadata would be.

[David Nolen]

On Tue, Feb 14, 2012 at 7:17 PM, Brandon Bloom <snpr...@gmail.com> wrote:

> I put your notes here,
> http://dev.clojure.org/display/design/Dynamic+Binding

Thanks!

I don't do anything special for asynchronous code, this simply provides the primitives necessary. Which is, primarily, bound-fn. Which is, in turn, built on {get,pop,push}-thread-bindings.

One problem then is that the work becomes explicit instead of implicit as it is in Clojure: future and agent sends setup the machinery for you. Another problem with the proposal is that there will be use cases for dynamic binding that shouldn't have to pay for the overhead of your implementation.

 

There is some future work, which requires far more thinking, to accomplish something akin to C#'s async/await keywords.

You can do that with delimited continuations implemented via macros.

David 

[Brandon Bloom]

> One problem then is that the work becomes explicit instead of implicit as it is in Clojure: future and agent sends setup the machinery for you.

You need the ability to explicitly capture and restore the current thread bindings to implement that implicit machinery.

The future macro is implemented in terms of future-call, which is implemented in terms of binding-conveyor-function, which is very similar to bound-fn. I still need to examine the differences a bit more, but it looks like binding-conveyor-function may just be an optimization to avoid re-validating already validated dynamic bindings.

ClojureScript doesn't have future-call or agents yet, but if they (or similar constructs) are to be implemented correctly, you need the ability to save and load a binding frame. That's what my changes accomplish.

> Another problem with the proposal is that there will be use cases for dynamic binding that shouldn't have to pay for the overhead of your implementation.

What use cases do you have in mind?

I haven't done any benchmarks, but the implementation closely follows JVM Clojure's implementation. As far as I can tell, the same fees are paid for every Var in every Clojure app. The cost of those fees may vary, but there is a bunch of room for optimization.

Basically, the current implementation of the binding that's in the master branch is much more similar to the with-redefs macro. Couldn't people who don't need async dynamic bindings, but need the perf simply use that? (I could implement it on this branch)

[David Nolen]

On Wed, Feb 15, 2012 at 2:09 PM, Brandon Bloom <snpr...@gmail.com> wrote:

ClojureScript doesn't have future-call or agents yet, but if they (or similar constructs) are to be implemented correctly, you need the ability to save and load a binding frame. That's what my changes accomplish.

Given the single-threaded nature of JavaScript I don't see how future-call or agents could be implemented.

 

What use cases do you have in mind?

Dynamic binding is often used for configuration.

These considerations should probably be included in your design notes. It's also important to note that what you're proposing could easily be provided as a useful library - ClojureScript is a proper Lisp after all :)

David

[Brandon Bloom]

> Given the single-threaded nature of JavaScript I don't see how future-call or agents could be implemented.

There are analogous constructs in a single-threaded, async-callback world. In particular, many Javascript libraries have a concept of promises and futures for managing the async callback speghetti. Like I said, that part of the puzzle needs much more thought. This is step one: Implement robust dynamic binding.

> Dynamic binding is often used for configuration.

Configuration that you probably want preserved across an async callback.... Imagine binding *parse-response-in-some-special-way* before calling ajax-get...

> It's also important to note that what you're proposing could easily be provided as a useful library

Maybe... but it would need to override some of the Compiler's multi-methods... Proper Var support, on par with JVM Clojure, seems like it belongs as part of the core ClojureScript distribution.

[Cedric Greevey]

On Wed, Feb 15, 2012 at 5:08 PM, David Nolen <dnolen...@gmail.com> wrote:
> On Wed, Feb 15, 2012 at 2:09 PM, Brandon Bloom <snpr...@gmail.com> wrote:
>>
>> ClojureScript doesn't have future-call or agents yet, but if they (or
>> similar constructs) are to be implemented correctly, you need the ability to
>> save and load a binding frame. That's what my changes accomplish.
>
> Given the single-threaded nature of JavaScript I don't see how future-call
> or agents could be implemented.

Well, I suppose one could implement green threads in JavaScript ... :)

[Brandon Bloom]

I've updated my fork with IWatchable and validation implemented on Var:

https://github.com/brandonbloom/clojurescript/compare/8ba4849e60e5957cdac36ef6946c647e824ca3c8...vars

@dnolen I assume I need to wait for my CA to be delivered/approved before I can update that Wiki page? I'd like to add some more notes...

[David Nolen]

On Wednesday, February 15, 2012, Brandon Bloom <snpr...@gmail.com> wrote:.

> There are analogous constructs in a single-threaded, async-callback world. In particular, many Javascript libraries have a concept of promises and futures for managing the async callback speghetti. Like I said, that part of the puzzle needs much more thought. This is step one: Implement robust dynamic binding.

Step one is properly describing the problem. There are many possible solutions many which don't try to graft Clojure's concurrency concepts onto a single-threaded environment. You're working on a particular solution - others already exist in the JS world as you've mentioned. A robust solution can be implemented via CPS transformation, and that can be provided as a ClojureScript library without changing the meaning of binding as it's currently implemented.

I would expand your design notes  before you continue down any particular solution. Otherwise it will have insufficient justification.

[Brandon Bloom]

> A robust solution can be implemented via CPS transformation

Forgive me for my ignorance, but how exactly would a CPS transformation enable one to capture and restore the current set of dynamic bindings?

[Laurent PETIT]

2012/2/16 David Nolen <dnolen...@gmail.com>

On Wednesday, February 15, 2012, Brandon Bloom <snpr...@gmail.com> wrote:.

> There are analogous constructs in a single-threaded, async-callback world. In particular, many Javascript libraries have a concept of promises and futures for managing the async callback speghetti. Like I said, that part of the puzzle needs much more thought. This is step one: Implement robust dynamic binding.

Step one is properly describing the problem. There are many possible solutions many which don't try to graft Clojure's concurrency concepts onto a single-threaded environment.

Forgive me if I'm wrong, but global vars / dynamic variables have essentially more to do with scoping than multithreading, e.g. they existed in lisps and other languages way before thread support was added to them.

So I don't understand why you are rejecting the idea of adding support for them based on the argument that Javascript is monothread ...
 

You're working on a particular solution - others already exist in the JS world as you've mentioned. A robust solution can be implemented via CPS transformation, and that can be provided as a ClojureScript library without changing the meaning of binding as it's currently implemented.

I would expand your design notes  before you continue down any particular solution. Otherwise it will have insufficient justification.

--

[Cedric Greevey]

On Fri, Feb 17, 2012 at 1:28 AM, Laurent PETIT <lauren...@gmail.com> wrote:
> Forgive me if I'm wrong, but global vars / dynamic variables have
> essentially more to do with scoping than multithreading, e.g. they existed
> in lisps and other languages way before thread support was added to them.

+1

> So I don't understand why you are rejecting the idea of adding support for
> them based on the argument that Javascript is monothread ...

Particularly when there is no guarantee that JS will stay that way
forever, and the general historic trend has been for languages to gain
multithreading that formerly lacked it.

[David Nolen]

On Fri, Feb 17, 2012 at 1:28 AM, Laurent PETIT <lauren...@gmail.com> wrote:

Forgive me if I'm wrong, but global vars / dynamic variables have essentially more to do with scoping than multithreading, e.g. they existed in lisps and other languages way before thread support was added to them.

So I don't understand why you are rejecting the idea of adding support for them based on the argument that Javascript is monothread ...

Dynamic binding already works in ClojureScript. The discussion is about whether dynamic binding should be extended to address common async patterns in JavaScript applications.

[Laurent PETIT]

Envoyé de mon iPhone

Le 17 févr. 2012 à 08:24, David Nolen <dnolen...@gmail.com> a écrit :

On Fri, Feb 17, 2012 at 1:28 AM, Laurent PETIT <lauren...@gmail.com> wrote:

Forgive me if I'm wrong, but global vars / dynamic variables have essentially more to do with scoping than multithreading, e.g. they existed in lisps and other languages way before thread support was added to them.

So I don't understand why you are rejecting the idea of adding support for them based on the argument that Javascript is monothread ...

Dynamic binding already works in ClojureScript.

Oh, ok then

The discussion is about whether dynamic binding should be extended to address common async patterns in JavaScript applications.

--

[David Nolen]

On Thu, Feb 16, 2012 at 1:33 PM, Brandon Bloom <snpr...@gmail.com> wrote:

> A robust solution can be implemented via CPS transformation

Forgive me for my ignorance, but how exactly would a CPS transformation enable one to capture and restore the current set of dynamic bindings?

The problem is dealing with with asynchronous code, right? Not capturing / restoring dynamic bindings.

David 

[Brandon Bloom]

The problem is dealing with with asynchronous code, right? Not capturing / restoring dynamic bindings.

No, the problem is that there is no mechanism to capture and restore dynamic bindings.

This is a shortcoming irrespective of asynchronous code.

It just so happens that such a mechanism is necessary for writing code which sets a dynamic binding, then schedules asynchronous code which uses that binding.

Notice my example from above, here it is again in Clojure:

user=> (def ^:dynamic x "root")

#'user/x

user=> x

"root"

user=> #'x

#'user/x

user=> ((binding [x "dynamic"] (fn [] x)))

"root"

user=> ((binding [x "dynamic"] (bound-fn [] x)))

"dynamic"

Notice two things about this code:

1) It is completely synchronous and single-threaded.

2) Now works identically in both Clojure and my ClojureScript branch.

See (doc bound-fn)

You suggested that there is insufficient motivation for my changes in the face of CPS / delimited continuations. My question to you is: How would you implement bound-fn without doing exactly what my code already does?

[David Nolen]

On Fri, Feb 17, 2012 at 1:37 PM, Brandon Bloom <snpr...@gmail.com> wrote:

The problem is dealing with with asynchronous code, right? Not capturing / restoring dynamic bindings.

No, the problem is that there is no mechanism to capture and restore dynamic bindings.

This is a shortcoming irrespective of asynchronous code. 

I think the question is whether ClojureScript needs to support bound-fn at all. But maybe it really is useful in a single-threaded environment! :)

I looked over your implementation again. It looks like the beginning of interesting line of investigation. Your implementation will likely be *dramatically* slower (I would not be surprised if it's >100X slower). You should test this and consider ways to decrease the impact. Would also be useful to know the performance impact on a common use case - binding fns.

Also this seems like yet another breaking ClojureScript change.

David

[Brandon Bloom]

> I think the question is whether ClojureScript needs to support bound-fn at all. But maybe it really is useful in a single-threaded environment! :)

I'm already finding it very useful for an experimental UI framework design I'm working on. I have a dynamically bound "insert pointer", such that UI views are functions which can be re-run automatically and "emit" dom elements into the tree at the point they were dynamically bound. My defview macro checks which arguments are IWatchable and triggers an inplace dom re-render when the arguments change. I'd also like to use the analyzer to look for deref calls, instead of just arguments, so I can get much more finely grained change notifications. In particular, data can be pushed from the server and trigger an asynchronous re-render, which needs to have the insert pointer and other bindings.

> Your implementation will likely be *dramatically* slower

Hence why it only applies to ^:dynamic variables. There are generally pretty few of these in any application.

I do, however, plan to add an additional way to trigger it for non-dynamic variables, if you want IWatchable support, etc. Probably a ^:var metadata attribute. {^:var false, ^:dynamic true} would be disallowed.

Also, the implementation has numerous optimization opportunities, manly centered around using native JS objects instead of ObjMap.

> Also this seems like yet another breaking ClojureScript change.

The two breaking changes I can see:

1) binding macro

• I'm re-using the native Clojure `binding` macro, but could write a custom one which is backwards compatible. With our without a warning configuration.
  
• The behavior I have implemented matches JVM clojure: You can only use binding on Vars labeled as ^:dynamic
• Fixing usages is as simple as adding ^:dynamic to the vars you are binding over.
• The binding macro behavior in master is actually with-redefs, so that could be subbed in as well.
• All and all, a pretty minimal breaking change with easy resolutions

2) ^:dynamic vars exposed to external code must now be accessed with deref

• This is extremely minor because Clojure JavaScript libraries are generally the bottom of the stack. If you need interop, you can easily create an additional var, or choose with-rdefs instead.
• I haven't currently implemented with-redefs because it would require the analyzer be used to differentiate "raw" vars or vars wrapped in my Var implementation. However, still pretty straight forward.

[Stuart Campbell]

Sorry to dig up such an old thread.

I'd also like to maintain the bindings of dynamic vars across asynchronous function calls.

Is there a workaround that people use in the absence of bound-fn, etc?

Cheers,
Stuart

On Friday, 27 January 2012 16:49:10 UTC+11, Brandon Bloom wrote:

The ClojureScript wiki states that "the user experience of [binding] is similar to that in Clojure" but my very first experiment produced wildly different results between platforms.

Here's a Clojure on the JVM session:

user=> (import java.lang.Thread)

java.lang.Thread

user=> (defn set-timeout [ms fn] (.run (Thread. #(do (Thread/sleep ms) (fn)))))

#'user/set-timeout

user=> (def x "top level")

#'user/x

user=> (binding [x "in binding"] (println x) (set-timeout 1000 #(println x)))

in binding

in binding

nil

And here's the analogous ClojureScript session:

ClojureScript:cljs.user> (def x "top level")

"top level"

ClojureScript:cljs.user> (binding [x "in binding"] (println x) (js/setTimeout #(println x) 1000))

in binding

21

top level

So ignoring the sequencing and nil vs timeout-id return values, the binding of 'x wasn't preserved in the asynchronous callback.

I raised this issue in #clojure and @dnolen said that "that's the behavior there's nothing much to fix", but that didn't sit right with me. This seems like either 'binding is bugged, or maybe I don't understand something about its intent.

On the topic of "Vars" proper, I understand their usefulness in repl-centric development, where you can redefine functions at runtime. The wiki also makes some mention of this, but I can't wrap my head around the context and jargon. I've run into this problem before in Javascript, where some level of indirection is necessary to support run-time redefinitions. You can't do `var fn = package.fn;` and dynamically redefine `fn` from `package` later because a copy of the reference is made. How does ClojureScript address this problem?

Cheers,

Brandon

[David Nolen]

There is not.

--

[white...@polyc0l0r.net]

Any chance that this could be supported now core.async is here? I think the arguments of brandon bloom are still valid. I have implemented Erlang-like supervisors for core.async and now got stuck in the last step when I realized that the supervisor binding is broken in cljs. (1) I can pass explicit arguments everywhere as a work-around, but this is not necessary on the JVM and makes the error-handling a lot less seamless. A binding fits nice for supervision. core.async would need to be extended as well, but I would be willing to put the effort in, if the approach is acceptable.

Cheers,
Christian

(1) https://github.com/whilo/full.monty/blob/master/full.async/src/full/lab.cljc#L74

clojure+unsubscribe@googlegroups.com