Turning off optimization to speed compilation during development

[Matthew Gertner]

My colleague was complaining about waiting ~20 seconds for his ClojureScript to compile and discovered this post on Stack Overflow: http://stackoverflow.com/questions/20917202/auto-building-clojurescript-files-for-compojure-app. In a nutshell, compilation time goes _way_ down if `optimization` is set to `:none`, but some hackery is needed to get the resulting JS to run. (I didn't check yet whether `:none` is a supported value or whether any unsupported value would have the same effect.)

Is there some option for turning off optimization completely that we haven't been able to find? If not, is there some reason not to support this? Wouldn't it be possible to adapt the parser so that it emits runnable JS without any optimization at all during the dev process? The compilation time speedup is really significant.

[David Nolen]

:advanced is only for release builds. :none is a fully supported
option. To be honest I'm a bit surprised this is not clear after going
through the documentation of either ClojureScript or lein cljsbuild.
Would be happy to hear why you had difficulty finding this information
and how we can make this point clearer.

Thanks,
David

> --
> Note that posts from new members are moderated - please be patient with your first post.
> ---
> You received this message because you are subscribed to the Google Groups "ClojureScript" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojurescrip...@googlegroups.com.
> To post to this group, send email to clojur...@googlegroups.com.
> Visit this group at http://groups.google.com/group/clojurescript.

[Francis Avila]

On Wednesday, November 26, 2014 9:09:58 AM UTC-6, Matthew Gertner wrote:
> My colleague was complaining about waiting ~20 seconds for his ClojureScript to compile and discovered this post on Stack Overflow: http://stackoverflow.com/questions/20917202/auto-building-clojurescript-files-for-compojure-app. In a nutshell, compilation time goes _way_ down if `optimization` is set to `:none`, but some hackery is needed to get the resulting JS to run. (I didn't check yet whether `:none` is a supported value or whether any unsupported value would have the same effect.)
>
> Is there some option for turning off optimization completely that we haven't been able to find? If not, is there some reason not to support this? Wouldn't it be possible to adapt the parser so that it emits runnable JS without any optimization at all during the dev process? The compilation time speedup is really significant.

What you describe is exactly what ":optimization :none" does. It emits javascript generated by the clojurescript compiler *without* compiling that javascript with the google closure compiler.

It requires different script elements in the html because the clojurescript compiler outputs one js file per cljs file, but the google closure compiler squashes it into one js file. There is a ticket[1] that has a proposal to simplify this so you can use the same html in both cases.

[1]: http://dev.clojure.org/jira/browse/CLJS-851

[Gary Verhaegen]

:optimization :whitespace is only slightly slower than none in my experience, and requires no hackery.

The best explanation I found of this was in the Clojurescript Up and Running book, though I imagine it must be documented online too. Basically, optimization none completely bypasses Google Closure and leaves you with the output of the ClojureScript compiler.

One of the features of the Google compiler is that it takes all of your files and combines them into a single one. With optimization none, you get one file per namespace, so you need to place these files in a reachable place and manually tell Closure (the library, not the compiler) to load them.

Optimization whitespace basically just does the concatenation, so it is pretty fast, and does not require a different setup as it still retains the feature of producing a single file. It is slower than none though, so you'll probably have to balance the costs yourself.

@david: the :none value for :optimization does not seem to be mentioned at all on the lein cljsbuild github project. In particular, the sample.project.clj mentions explicitly "may be :whitespace, :simple or :advanced".

[David Nolen]

On Wed, Nov 26, 2014 at 11:12 AM, Gary Verhaegen
<gary.ve...@gmail.com> wrote:
> :optimization :whitespace is only slightly slower than none in my
> experience, and requires no hackery.

Not in my experience. Recompiles with :none can take as little as tens
of milliseconds.

More importantly *only* :none will supply you with accurate source
maps. Also while it may seem inconvenient, requiring that you manually
inline third party scripts under :none is valueable - you know you are
debugging precisely the released source of the third party library
with exact line numbers.

While admittedly more can be done from the ClojureScript compiler
standpoint, customizing your HTML generation to do the right thing is
very straightforward these days with tools like environ. See Chestnut
to see how this can be done in your own projects if you are looking
for a reference.

I think there are only two settings worth using: :none & :advanced.

David

[Kyle Cordes]

On Wednesday, November 26, 2014 at 10:12 AM, Gary Verhaegen wrote:
> :optimization :whitespace is only slightly slower than none in my experience, and requires no hackery.
>

Sadly, some tools (Figwheel maybe?) work with :none but not with :whitespace.


--
Kyle Cordes
http://kylecordes.com

[David Nolen]

I'll repeat.

Use :whitespace and :simple if you want corrupted source maps.

:whitespace was useful prior to source maps. :simple was useful for a
time due to goog.require not working under Node.js - no longer true.

IMO, there is absolutely no good reason to use these settings.

David

[Daniel Kersten]

To mirror what David is saying: :none compiles my code (with cljsbuild auto) in under 400ms in VirtualBox (yep, I've an odd dev environment), :whitespace can take a second or even two.

However, I have found :whitespace and :simple useful for debugging before when a bug manifested in :advanced but not in :none (I've had this happen once or twice and EVERY time it was due to a missing extern) - I find it useful to have these options (or, at least, :whitespace) for this purpose, but never use them in my day to day cljs development.

On Wed Nov 26 2014 at 4:55:23 PM David Nolen <dnolen...@gmail.com> wrote:

I'll repeat.

Use :whitespace and :simple if you want corrupted source maps.

:whitespace was useful prior to source maps. :simple was useful for a
time due to goog.require not working under Node.js - no longer true.

IMO, there is absolutely no good reason to use these settings.

David

On Wed, Nov 26, 2014 at 11:51 AM, Kyle Cordes <ky...@kylecordes.com> wrote:
> On Wednesday, November 26, 2014 at 10:12 AM, Gary Verhaegen wrote:
>> :optimization :whitespace is only slightly slower than none in my experience, and requires no hackery.
>>
>
>
> Sadly, some tools (Figwheel maybe?) work with :none but not with :whitespace.
>
>
> --
> Kyle Cordes
> http://kylecordes.com
>
>
>
>
> --
> Note that posts from new members are moderated - please be patient with your first post.
> ---
> You received this message because you are subscribed to the Google Groups "ClojureScript" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscribe@googlegroups.com.

> To post to this group, send email to clojur...@googlegroups.com.
> Visit this group at http://groups.google.com/group/clojurescript.

--
Note that posts from new members are moderated - please be patient with your first post.
---
You received this message because you are subscribed to the Google Groups "ClojureScript" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscribe@googlegroups.com.

[David Nolen]

Good points though I've found the :pseudo-names option under :advanced
to be about as useful in the missing externs case.

David

>> > an email to clojurescrip...@googlegroups.com.

>> > To post to this group, send email to clojur...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/clojurescript.
>>
>> --
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "ClojureScript" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to clojurescrip...@googlegroups.com.

>> To post to this group, send email to clojur...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/clojurescript.
>
> --
> Note that posts from new members are moderated - please be patient with your
> first post.
> ---
> You received this message because you are subscribed to the Google Groups
> "ClojureScript" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to clojurescrip...@googlegroups.com.

[Daniel Kersten]

Ah, that's true - I even used it last time I hit this too. Thanks for pointing it out again!

>> > an email to clojurescript+unsubscribe@googlegroups.com.

>> > To post to this group, send email to clojur...@googlegroups.com.
>> > Visit this group at http://groups.google.com/group/clojurescript.
>>
>> --
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "ClojureScript" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to clojurescript+unsubscribe@googlegroups.com.

>> To post to this group, send email to clojur...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/clojurescript.
>
> --
> Note that posts from new members are moderated - please be patient with your
> first post.
> ---
> You received this message because you are subscribed to the Google Groups
> "ClojureScript" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to clojurescript+unsubscribe@googlegroups.com.

> To post to this group, send email to clojur...@googlegroups.com.
> Visit this group at http://groups.google.com/group/clojurescript.

--
Note that posts from new members are moderated - please be patient with your first post.
---
You received this message because you are subscribed to the Google Groups "ClojureScript" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscribe@googlegroups.com.

[Kyle Cordes]

On Wednesday, November 26, 2014 at 10:55 AM, David Nolen wrote:
> I'll repeat.
>
> Use :whitespace and :simple if you want corrupted source maps.
>
> :whitespace was useful prior to source maps. :simple was useful for a
> time due to goog.require not working under Node.js - no longer true.
>
> IMO, there is absolutely no good reason to use these settings.
>

I wasn’t speaking in favor of those settings, just pointing out that there are various constraints. That is a problem with have a lot of settings available, we end up with different tools supporting different semi-overlapping subsets of the settings.

Personally I’d be happy if all except the most useful settings were removed. Maybe only :none and :advanced are actually still needed in 2014.

[David Nolen]

On Wed, Nov 26, 2014 at 3:22 PM, Kyle Cordes <ky...@kylecordes.com> wrote:
> I wasn’t speaking in favor of those settings, just pointing out that there are various constraints. That is a problem with have a lot of settings available, we end up with different tools supporting different semi-overlapping subsets of the settings.
>
> Personally I’d be happy if all except the most useful settings were removed. Maybe only :none and :advanced are actually still needed in 2014.

No plans to remove anything. If tools are passing bad settings to the
ClojureScript compiler or don't reliably support :none, I just would
not use these tools.

David

[Kyle Cordes]

On Wednesday, November 26, 2014 at 2:29 PM, David Nolen wrote:
> No plans to remove anything. If tools are passing bad settings to the
> ClojureScript compiler or don't reliably support :none, I just would
> not use these tools.
>

I think it is all a solved problem today. I have a process working well with :none, and the source map stuff is excellent. Some tool I used a while back did not support :none, and I dropped it, just as you described. I was just thinking historically the other settings were a bit of an “attractive nuisance”. Sorry to sound argumentative before, I am a huge fan of all this stuff.

[David Nolen]

On Wed, Nov 26, 2014 at 3:34 PM, Kyle Cordes <ky...@kylecordes.com> wrote:
> I think it is all a solved problem today. I have a process working well with :none, and the source map stuff is excellent. Some tool I used a while back did not support :none, and I dropped it, just as you described. I was just thinking historically the other settings were a bit of an “attractive nuisance”. Sorry to sound argumentative before, I am a huge fan of all this stuff.

We're not arguing :) Just clarifying the points for other possible
observers of this conversation who may be less knowledgeable about the
rationale for the various knobs.

It's unfortunate that we have so many knobs, but there's a few solid
combinations and we should clearly communicate that in docs,
libraries, tools, and discussions.

David

[danneu]

For one, lein-cljsbuild doesn't even list `:none` as an option in its sample config: https://github.com/emezeske/lein-cljsbuild/blob/1.0.3/sample.project.clj#L96-L98

And, unlike the other optimization options, `:none` requires some html boilerplate which seems to be the focus of this proposal: http://dev.clojure.org/jira/browse/CLJS-851

It doesn't take much code for `:whitespace` to start taking 2-3 seconds on my new Macbook Air. `:none` on the same codebase takes a couple hundred milliseconds.

One improvement would be to simply include a `:none` + `:output-to` + boilerplate example in lein-cljsbuild's README quickstart.

[Tomas Brambora]

I'm the aforementioned colleague and it takes up to 15s on my two year old Mac Air which, frankly, is quite unbearable.

Would be great to have a HOWTO on :none on the lein cljsbuild wiki.

[Mike Thompson]

On Thursday, November 27, 2014 2:09:58 AM UTC+11, Matthew Gertner wrote:
> My colleague was complaining about waiting ~20 seconds for his ClojureScript to compile and discovered this post on Stack Overflow: http://stackoverflow.com/questions/20917202/auto-building-clojurescript-files-for-compojure-app. In a nutshell, compilation time goes _way_ down if `optimization` is set to `:none`, but some hackery is needed to get the resulting JS to run. (I didn't check yet whether `:none` is a supported value or whether any unsupported value would have the same effect.)
>
> Is there some option for turning off optimization completely that we haven't been able to find? If not, is there some reason not to support this? Wouldn't it be possible to adapt the parser so that it emits runnable JS without any optimization at all during the dev process? The compilation time speedup is really significant.

If you want to mix `:none` with a unittest framework, then this may help:
https://github.com/mike-thompson-day8/cljsbuild-none-test-seed

--
Mike

[Colin Yates]

Mike, that's brilliant. Downloading it now.

[Kyle Cordes]

On Thu, Nov 27, 2014 at 7:08 AM, Colin Yates <colin...@gmail.com> wrote:
> Mike, that's brilliant. Downloading it now.

It is a great thing to have handy, and it is also points to another
example of the problem I was complaining about, which is that there
are N different optimization settings, of which different tools
support different subsets. Quoting the page linked, "At the time of
writing, cemerick/clojurescript.test doesn't work with :optimizations
:none".

This is a not a great situation. We have this wonderful programming
language, whose many great characteristics include a broad toolbox of
mostly composable tools. But up here at the tooling level, lots of
CLJS tools are not composable.

[Colin Yates]

I hear you. I definitely felt a little lost as to what "The Right Way"
forward was. It still feels very cutting edge.

Still, would much rather than the 100 paper cuts here than the 1000
paper cuts elsewhere ;).

> --
> Note that posts from new members are moderated - please be patient with your first post.
> ---

> You received this message because you are subscribed to a topic in the Google Groups "ClojureScript" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojurescript/0bSUNNKvmHE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to clojurescrip...@googlegroups.com.

[Mike Thompson]

Yes, tooling is definitely a problem. Compiler messages are poor and initially off putting. Docs are threadbare compared to other languages I've used. Debugging is a bit too hard. All these things are true. All these things stop wider adoption.

I'd expect with the current trajectory that things will be much better in about 12 months. There are greenshoots:
https://github.com/oakmac/cljsbuild-ui
https://github.com/adzerk/boot-cljs

To me, a crucial step for CLJS is to have a compiler that is liberated from Java. I've looked and it seems a number have tried to make it happen, but without official support each attempt seems to have withered. What we appear to need is a thin, official clojure I/O library over the top of Java, and all tools written in terms of that I/O library, which can then be implemented for different platforms. Then tools would be portable away from Java.

The trouble is that Java is portable in its own right, correct? Sorta. So there doesn't seem to be offical momentum to make clojurescript portable off Java itself.

On other matters, vote now:
https://github.com/cemerick/clojurescript.test/issues/75


--
Mike

[Matthew Gertner]

On Wednesday, November 26, 2014 10:48:52 AM UTC-5, David Nolen wrote:
> :advanced is only for release builds. :none is a fully supported
> option. To be honest I'm a bit surprised this is not clear after going
> through the documentation of either ClojureScript or lein cljsbuild.
> Would be happy to hear why you had difficulty finding this information
> and how we can make this point clearer.

Thanks for the clarification (and all the great information in this thread). My colleague Tomas was looking into the optimization stuff so he can provide more detail about what he felt was lacking. He mentions the lein cljsbuild docs later in this thread so it would probably be good to add it there.

I can't claim to fully grok all the implications of http://dev.clojure.org/jira/browse/CLJS-851 yet but it does seem like a good idea to enable users to change optimization settings without having to modify their source code.

[Andrew Rosa]

On Thursday, November 27, 2014 10:14:05 PM UTC-2, Mike Thompson wrote:
> On Friday, November 28, 2014 3:21:09 AM UTC+11, Kyle Cordes wrote:

Making CLJS compiler completely independent from Java seems infeasible, since Google Closure is written in Java itself, and Closure is one of the main advantages IMO over other languages that sit on top of Javascript.

I like the idea of IO abstraction. As I understand, internally the cljsc works as a two step process: first it emits the javascript, and after that kicks the closure compiler to do the dirt work. Maybe we could really split it into two separate "components", where the first one could be run without Java if desired. This component alone will not be able to do any optimizations other than :none, but this could be enough.

And there is a big gotcha: currently macros run on JVM, and I don't know if there is a way to decouple that.

[]'s
Andrew

[David Nolen]

On Sat, Nov 29, 2014 at 7:48 PM, Andrew Rosa <andr...@me.com> wrote:
>> To me, a crucial step for CLJS is to have a compiler that is liberated from Java. I've looked and it seems a number have tried to make it happen, but without official support each attempt seems to have withered.

It is not "crucial" for ClojureScript to be self-hostable, but yes it
would be a nice *option*.

A significant amount of progress has already been done to make this
possible. The only reason it hasn't been done yet is because no one
has yet has stepped up and pushed it past the finish line with a
series of patches. The remaining tasks:

* Portable tools.reader
* Macroexpander written in Clojure
* Conditional compilation tweaks (feature expressions) to pick the
right host type (clojure.core.PersistentVector vs.
cljs.core/PersistentVector)

> Making CLJS compiler completely independent from Java seems infeasible, since Google Closure is written in Java itself, and Closure is one of the main advantages IMO over other languages that sit on top of Javascript.

Closure is not an advantage in many desirable JS environments:
Node.js, node-webkit, Atom shell, iOS, etc.

David

[Colin Fleming]

Closure is not an advantage in many desirable JS environments:
Node.js, node-webkit, Atom shell, iOS, etc.

I assume you're talking about the Closure library here, not the compiler? Surely the compiler would be advantageous in all environments? I'd have thought that aspects of the library would be too (string/date handling etc) but I guess its value is diminished somewhat.

--
Note that posts from new members are moderated - please be patient with your first post.
---

You received this message because you are subscribed to the Google Groups "ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojurescrip...@googlegroups.com.

[Daniel Compton]

There are a few places where the recommended optimisation settings could be documented better which would save people this confusion in the future:

https://github.com/emezeske/lein-cljsbuild/blob/master/README.md - shows a project using whitespace optimisations, doesn’t really talk about the options available and when to use them.

https://github.com/emezeske/lein-cljsbuild/blob/1.0.3/example-projects/advanced/project.clj - shows whitespace and advanced optimisations. Also the comments suggest :whitespace is the lowest level of optimisations (I guess it is depending on how you look at it)

https://github.com/emezeske/lein-cljsbuild/blob/1.0.3/example-projects/simple/project.clj - shows whitespace optimisations

[Daniel Compton]

There are a few places where the recommended optimisation settings could be documented better which would save people this confusion in the future:

https://github.com/emezeske/lein-cljsbuild/blob/master/README.md - shows a project using whitespace optimisations, doesn’t really talk about the options available and when to use them.

https://github.com/emezeske/lein-cljsbuild/blob/1.0.3/example-projects/advanced/project.clj - shows whitespace and advanced optimisations. Also the comments suggest :whitespace is the lowest level of optimisations (I guess it is depending on how you look at it)

https://github.com/emezeske/lein-cljsbuild/blob/1.0.3/example-projects/simple/project.clj - shows whitespace optimisations

I assume that lowering the optim

On 1/12/2014, at 12:20 pm, Colin Fleming <colin.ma...@gmail.com> wrote:

[Daniel Compton]

There are a few places where the recommended optimisation settings could be documented better which would save people this confusion in the future:

https://github.com/emezeske/lein-cljsbuild/blob/master/README.md - shows a project using whitespace optimisations, doesn’t really talk about the options available and when to use them.

https://github.com/emezeske/lein-cljsbuild/blob/1.0.3/example-projects/advanced/project.clj - shows whitespace and advanced optimisations. Also the comments suggest :whitespace is the lowest level of optimisations (I guess it is depending on how you look at it)

https://github.com/emezeske/lein-cljsbuild/blob/1.0.3/example-projects/simple/project.clj - shows whitespace optimisations

I assume that lowering the optimisation level to :none would be safe in all of these examples, is that a correct assumption?

On 1/12/2014, at 12:20 pm, Colin Fleming <colin.ma...@gmail.com> wrote:

[David Nolen]

On Sun, Nov 30, 2014 at 6:20 PM, Colin Fleming
<colin.ma...@gmail.com> wrote:
>> Closure is not an advantage in many desirable JS environments:
>> Node.js, node-webkit, Atom shell, iOS, etc.
>
>
> I assume you're talking about the Closure library here, not the compiler?
> Surely the compiler would be advantageous in all environments? I'd have
> thought that aspects of the library would be too (string/date handling etc)
> but I guess its value is diminished somewhat.

The Closure library is useful in the various environments - it
supplies utilities and works around quirks.

The Closure Compiler is not that useful. Dead code elimination in
these environment don't matter. Code size similarly a non-issue.
Closure's performance optimizations in a modern high performance JS
engine are negligible - our own ClojureScript specific optimizations
are far more relevant.

David

[Chas Emerick]

If particular settings are considered "bad", then some kind of
deprecation warning is warranted. This would be a useful signal to
everyone (toolmakers included) that e.g. :simple is not useful anymore
and :none is strictly superior.

A couple of tools I maintain (clojurescript.test and others) do not
currently support :none only because, at least when I last tried it,
:none required an arrangement of resources on disk that didn't match up
well with the rest of my workflow. I will take a fresh look at :none in
the coming weeks.

I am conscious of the fact that lein-cljsbuild has by default become the
canonical compiler documentation, outside of various blog and mailing
list posts. I've just now merged a couple of pull requests that update
its docs to include mentions and examples using :none. At least until I
am again tracking ClojureScript compiler development on a regular basis,
I will rely upon community contributors to help keep lein-cljsbuild
documentation accurate and up to date.

Best,

- Chas

[David Nolen]

As stated earlier we're not going to deprecate any compiler settings,
there may still be some scenarios where these settings are useful.
That said, I think all development time tooling really needs to
support :none, that is the baseline.

Related, I've gone ahead and started an "official" port of
clojure.test to ClojureScript that is bundled in so that we can
provide the community with basic testing functionality that works for
all compiler settings out of the box. This thread has convinced me it
is a significant problem felt by the community that needs a standard
solution.

Along the way I've made changes to ClojureScript analysis and
compilation to ensure that anyone wanting to provide their own custom
testing tooling will now find considerable support to make such work
easier:

* compiler support for :test metadata on vars
* static vars - (var foo) will dump an object with metadata that
corresponds to Clojure var metadata (:file, :line, :column, :arglists,
:test, etc.)
* ns-interns macro - will dump a ClojureScript map from symbols to vars
* pure ClojureScript source map decoding - simplify rewriting
JavaScript stack traces for test reporting

I want to see people building off a common testing infrastructure to
guarantee compatibility with all compiler options moving forward.
Similarly for other tooling I'd rather see patches that make
ClojureScript easier to work with rather than see tool authors work
around the current limitations of the compiler and provide sub-optimal
experiences to end users.

David