The Future of ClojureScript Tools

[Thomas Heller]

Hey,

First of all, huge thanks to David Nolen for all your work on ClojureScript.

TL;DR: I want to help, but I'm sort of "stuck".

With all the recent CLJS commits going into tooling related issues (cljs.test, caching, ...) I find a lot of overlap to what I have done in shadow-build. I either find myself re-implementing things or see things going into CLJS that I had for some time. I don't mind all the work as it keeps me up-to-date with CLJS but it doesn't exactly contribute back to CLJS overall. I'm not even sure anything I have done needs to be in CLJS since it works quite well as a library.

Which brings me to my issue: What exactly needs to be in CLJS?

When looking at other CLJS tools I frequently see that they either copy "hacks" (eg. multiple source paths) between each other, monkey-patch each other or even bypassing CLJS completely and watching the :output-dir.

I believe most of it is due to the fact that everything has to go through cljs.closure/build. Since everything build related is hidden behind that function, CLJS has to do quite a lot instead of just worrying about compiling CLJS->JS. Recently a cljs.build.api namespace was added to allow querying the internal state, but it is still impossible to hook into the actual build process which you simply need for some features. Which is why I decided to not use cljs.closure at all.

There is a definite need for multiple build tools since build targets sometimes differ significantly (Node.JS vs Browser). Yet cljs.closure is sprinkled with a bit of everything, trying to do everything. I fear that this will hurt in the long run. I already see several things "bleeding" into other namespaces cause of decisions made there (eg. the reliance on files).

I'm sort of stuck. On one hand I want to help CLJS. On the other hand I don't know if anything I do would actually help since I have very specific targets and requirements that I certainly don't want to push on everyone.

I believe it would be beneficial to split up cljs.closure/build but I'm not even sure about that since technically certain targets (Node.JS) don't even need the whole Closure Compiler at all.

I think I'm too biased working with shadow-build for such a long time but I'd be interested to hear other opinions on the matter given that recently quite a few new build tools became popular.

I'm not sure there is a problem at all since so far I was able to do anything I wanted in shadow-build without too much hassle. Just feels off seeing basically duplicate work by either David or me.

I feel like an a**hole reporting (commenting) issues I have already fixed but can't contribute back due to the significant difference in architecture.

Should we try to refactor the build process overall? Or am I imagining things?

Cheers,
/thomas

[David Nolen]

On Tue, Dec 30, 2014 at 5:38 PM, Thomas Heller <th.h...@gmail.com> wrote:
> I believe most of it is due to the fact that everything has to go through cljs.closure/build. Since everything build related is hidden behind that function, CLJS has to do quite a lot instead of just worrying about compiling CLJS->JS. Recently a cljs.build.api namespace was added to allow querying the internal state, but it is still impossible to hook into the actual build process which you simply need for some features. Which is why I decided to not use cljs.closure at all.

I don't see how this is true in general. The Node.js REPL hooks into
the functions that cljs.closure/build uses without going through it.

> There is a definite need for multiple build tools since build targets sometimes differ significantly (Node.JS vs Browser). Yet cljs.closure is sprinkled with a bit of everything, trying to do everything. I fear that this will hurt in the long run. I already see several things "bleeding" into other namespaces cause of decisions made there (eg. the reliance on files).

Supporting both Node.js and the Browser has been a goal since the
initial release. There's nothing new going on here except more &
better.

As to the reliance on files this is directly result of the desire to
improve start time performance of REPLs and cold builds. I don't see
any better solutions when performance and sharing analysis information
across runs/processes is the goal.

> I believe it would be beneficial to split up cljs.closure/build but I'm not even sure about that since technically certain targets (Node.JS) don't even need the whole Closure Compiler at all.

I think it would better to invest in cljs.build.api.

> I'm not sure there is a problem at all since so far I was able to do anything I wanted in shadow-build without too much hassle. Just feels off seeing basically duplicate work by either David or me.

Agreed that I would rather see people put shareable work into
ClojureScript itself.

> Should we try to refactor the build process overall? Or am I imagining things?

Refactoring means needless churn. At this point it's preferable to do
things incrementally behind feature flags. Again I also don't see why
much of this work can't be done via patches to cljs.build.api.

David

[Thomas Heller]

On Wednesday, December 31, 2014 12:17:11 AM UTC+1, David Nolen wrote:
>
> Supporting both Node.js and the Browser has been a goal since the
> initial release. There's nothing new going on here except more &
> better.

What is tooling and what is CLJS? At which point does it make sense to address an issue in CLJS rather than a library. Given that probably no one will use the Node.JS REPL the way you described in your blog post other than for experiments. It will probably be mostly used via tools (eg. lein-cljsbuild, lein-figwheel, etc ...)

Does the Node REPL even need to be in CLJS itself? Clojure has a variety of "contrib" (eg. core.async, core.typed, data.json, ...), shouldn't we maybe split into contrib?

Why do we have clojure.browser.*? These things are dangerously out of date and incomplete. I tripped over that when I started.

>
> As to the reliance on files this is directly result of the desire to
> improve start time performance of REPLs and cold builds. I don't see
> any better solutions when performance and sharing analysis information
> across runs/processes is the goal.

Again, why is that the job of core CLJS? Granted we will probably always end up with files so it is a pretty safe bet.

>
> > Should we try to refactor the build process overall? Or am I imagining things?
>
> Refactoring means needless churn. At this point it's preferable to do
> things incrementally behind feature flags. Again I also don't see why
> much of this work can't be done via patches to cljs.build.api.

Well, easier said than done. A lot of stuff I'm looking at requires significant changes and some even break backwards compatibility. Obviously these things can (and should) take quite a long time and consideration which is why I think they should not be in CLJS core.

Take CLJS-851 [1] as a simple example. The fact that most people still are required to manually require goog/base.js + app.js MANUALLY in the HTML in development but not in production is completely ridiculous to me. A seemingly simple change of adding :main suddenly has very big implications overall as it ties into [2] since given a :main you can always compile in dependency order.


Which brings me back to my question: What exactly needs to be in CLJS?

Given that most tools will extend base functionality in one way or another anyways?


CLJS doesn't even support multiple source-paths and tools rely on a simple "hack" I wrote about 2 years ago [3]. Note that this hack does not work with :main since you may use namespaces from the another source-path which is not known to CLJS (given that we just map over them).

Sorry this is starting to turn into a rant, which isn't helpful.


I do believe refactoring is necessary.

/thomas


[1] http://dev.clojure.org/jira/browse/CLJS-851
[2] http://dev.clojure.org/jira/browse/CLJS-927
[3] https://github.com/emezeske/lein-cljsbuild/blob/3625516e82701ee89109875ea60d2d4035ef02e0/support/src/cljsbuild/compiler.clj#L64-L67

[David Nolen]

On Tue, Dec 30, 2014 at 10:46 PM, Thomas Heller <th.h...@gmail.com> wrote:
> What is tooling and what is CLJS? At which point does it make sense to address an issue in CLJS rather than a library. Given that probably no one will use the Node.JS REPL the way you described in your blog post other than for experiments. It will probably be mostly used via tools (eg. lein-cljsbuild, lein-figwheel, etc ...)

basic REPLs are not tooling.

As to why a Node.js REPL?

* Rhino is >100X slower than modern JS engines
* Many REPL interactions do not benefit from a browser

As far as users not running the Node.js REPL in the way that has been
described - to this day I still run raw Clojure REPLs sans Lein for a
good many reasons - I suspect the same will continue to be true for
the builtin ClojureScript REPLs.

> Does the Node REPL even need to be in CLJS itself? Clojure has a variety of "contrib" (eg. core.async, core.typed, data.json, ...), shouldn't we maybe split into contrib?

I'm having trouble seeing any tangible benefits here.

> Take CLJS-851 [1] as a simple example. The fact that most people still are required to manually require goog/base.js + app.js MANUALLY in the HTML in development but not in production is completely ridiculous to me. A seemingly simple change of adding :main suddenly has very big implications overall as it ties into [2] since given a :main you can always compile in dependency order.

CLJS-851 doesn't address the broader problem - i.e. foreign JS
dependencies. Fixing the problem only for CLJS namespaces isn't
actually dealing with the deeper issue.

> I do believe refactoring is necessary.

I don't see the value of this line of thought when cljs.build.api is a
legimate path forward for build tools.

David