ClojureScripts persistent data structures

[Jonas Enlund]

I’ve been studying the implementation of ClojureScript and it is an absolutely wonderful experience. Thanks to all who are involved in this amazing experiment!

I have some concern about the implementation details of the immutable data structures which I'd like to see some discussion on. This is how I understand it, and please correct me if I’m wrong:

1. The data structures are implemented in pure clojure(script) using deftype and protocols.
2. deftype does not (and should not!) support static fields or methods.
3. What then, does forms like these mean (i.e. What are the semantics?) [1]:
- cljs.core.PersistentVector/EMPTY
- cljs.core.PersistentVector/fromArray

In the clojure world of java interop the above would be static fields and methods but there are no such things for neither deftype nor javascript so I would have guessed that these forms would be considered invalid. In fact, I’m amazed that it works at all, since they confuse the analyzer into thinking that there is a namespace called cljs.core.PersistentVector with a var named EMPTY.

What would be the alternative when implementing the clojure data structures using clojure itself? A good idea might be to look at the implementation of gvec[2]. In that implementation top level forms such as

    (def EMPTY-NODE (VecNode. nil (object-array 32)))

are defined instead of using static fields.

Some discussion took place on IRC[3] earlier today. I think we should study the implementation of gvec and see how we could clean up the implementation of the clojurescript data structures. What do other people think? 

Jonas

[1] Search for e.g. "/EMPTY" at https://github.com/clojure/clojurescript/blob/master/src/cljs/cljs/core.cljs

[2] https://github.com/clojure/clojure/blob/master/src/clj/clojure/gvec.clj

[3] http://clojure-log.n01se.net/date/2012-06-20.html#12:30

[Herwig Hochleitner]

2012/6/20 Jonas Enlund <jonas....@gmail.com>

3. What then, does forms like these mean (i.e. What are the semantics?) [1]:
- cljs.core.PersistentVector/EMPTY
- cljs.core.PersistentVector/fromArray

In the clojure world of java interop the above would be static fields and methods but there are no such things for neither deftype nor javascript so I would have guessed that these forms would be considered invalid. In fact, I’m amazed that it works at all, since they confuse the analyzer into thinking that there is a namespace called cljs.core.PersistentVector with a var named EMPTY.

My guess: The implementor of PersistentVector mimicked the style of the closure library: http://closure-library.googlecode.com/svn/docs/closure_goog_events_eventtype.js.source.html

--

It's awkward there too, mostly because the statics are extremely hard to find in the closure lib doc, when you don't know to look in the file index. Namespaces of pure statics don't even show up in the class browser, even if camel cased.

Statics are truly the stepchild of OOP :o)

And everything because of the refusal to have single values considered 'equivalent' to whole classes of values.

Might help to think of classes as multi - entry functions ... with explicitely defined closed-overs ... therefore explicit instatiation ... how do they even ... well, it is what it is and that is what von Neumann can do for us. Fast punchcard machines.

Harvard has given up and coded Facebook in PHP ;-)

</rant>

 

What would be the alternative when implementing the clojure data structures using clojure itself? A good idea might be to look at the implementation of gvec[2]. In that implementation top level forms such as

    (def EMPTY-NODE (VecNode. nil (object-array 32)))

are defined instead of using static fields.

Agree!

(declare EMPTY)

(deftype UsingEmpty ...)

(def EMPTY ...)

 

Seems more clojury to me. It's definitely used everywhere else in cljs core.

kind regards

[Jonas Enlund]

It's interesting to see what Vars gets resolved to. In my fork[1] I've added the following code to 'confirm-var-exists':

  (when-not (or ((conj '#{cljs.core js goog} crnt-ns) prefix)

                (some #{prefix} (-> env :ns :requires vals)))

    (warning env

      (str "WARNING: Use of undeclared ns " prefix " with Var " prefix "/" suffix)))

With this check in place lots of warnings are generated (by i.e., running script/repljs). Here are a few:

1. WARNING: Use of undeclared ns cljs with Var cljs/core.-invoke at line 117

2. WARNING: Use of undeclared ns Math with Var Math/floor at line 1275

3. WARNING: Use of undeclared ns cljs.core.PersistentVector with Var cljs.core.PersistentVector/EMPTY at line 3013

4. WARNING: Use of undeclared ns cljs with Var cljs/core.PersistentTreeMap at line 5467

I'll quickly go through why these warnings are generated.

* There are many places in core.cljs that don't yet use the 'js' namespace when using javascript functions. This is where warning #2 comes from.

* When using the form cljs.core.PersistentVector/EMPTY the analyzer believes there is a namespace called cljs.core.PersistentVector (warning #3). I raised this issue in my previous message and I still don't understand why this kind of functionality is put on the deftype. Is this not an OO concept?

* Warning number #4 comes from the use of (.indexOf ...) instead of (.lastIndexOf ...) in 'resolve-existing-var'[2]

* The first warning was the trickiest to hunt down. The reason is at [3]. That line "resolves" the var '-invoke'  to 'cljs.core.-invoke' and later in 'resolve-existing-var' the same thing as in #4 happens. I see no reason to do var resolution at that point.

I have a local (experimental) branch[4] where I have fixed these issues. Note that I still put EMPTY/fromArray etc. on the deftypes which I don't think is a good idea.

I'd be happy to contribute these fixes to ClojureScript if you think I'm on the right track here.

Jonas

[1]: https://github.com/jonase/clojurescript/compare/warn-on-missing-ns

[2]: https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/analyzer.clj#L124

[3]: https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/core.clj#L690

[4]: https://github.com/jonase/clojurescript/compare/interop