Scripting with Clojure / "slow" boot time

[Alex Miller]

I'm doing some research on slow Clojure boot time and would be interested in collecting info about example use cases where it's been a problem for people.

http://goo.gl/forms/eSpn8M5FNB

I'm not expecting to release the results in any formal way, mostly looking to use it to pull out particular use cases and/or commonalities across anecdotes so that we can work on boot time problems that matter the most (and solutions likely to help the most). Any numbers you can provide would be great.

Alex

[James Elliott]

I don’t have any examples to share, I just wanted to thank you for doing this. It is really wonderful to see how responsive Clojure is to community input and concerns, and the launch of an effort like this so quickly in response to the survey feedback.

[Mikera]

Good initiative, I've filled in a response to the survey.

One thing that strikes me is that the main issue with loading time is the time required to transitively load and compile all the referred namespaces (which can be a lot in a big project...). This in turn may trigger class loading of Java libraries, IO as various resources are initialised etc. An observation is that a lot of this is often not required initially, so there *might* be a clever strategy to mitigate this through laziness.

This could be something like:

- When loading the current namespace, *don't* load referred namespaces (yet)

- Create lazy placeholders within vars in the current namespace for (for every def, defn etc.)

- Only when the placeholder is deref'd / invoked then compile the relevant function and pull in dependencies. 

- After first deref / invocation, replace the placeholder with the full compiled function / value, so that subsequent access has no overhead

This would be a pretty big change, and might break tools that make assumptions about order of loading of namespaces... but I think it would solve 90% of the boot time problems if implemented correctly.

The other potentially big win would be concurrent loading of namespaces. Guess you are looking at that already?

[Marc O'Morain]

Hi Alex,

I've love to offer any help/test data on this that I can. At CircleCI we run (among other processes) a large Clojure app that takes 2 minutes to load on a top of line Macbook Pro, with Java 1.8 (Hotspot) . From my best efforts at profiling, this time is all spend in the Clojure Compiler and the JVM class loader, loading all transitive dependencies. Mike's analysis above is totally in line with what we see.

One thing I have noticed is that all compilation runs in a single thread. In theory all namespaces required in an `ns` form can be loaded in parallel.

If there is any profiling you would like me to run on the code-base, please let me know.

Marc

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gary Verhaegen]

Please excuse the very naive question, but if the main problem seems to be compilation of loaded namespaces, what about aot? Does it help at all? If not, why? As far as I know, the JVM already does lazy loading with Java classes essentially in the way that Mike described.

[Herwig Hochleitner]

2016-02-10 13:06 GMT+01:00 Gary Verhaegen <gary.ve...@gmail.com>:

Please excuse the very naive question, but if the main problem seems to be compilation of loaded namespaces, what about aot? Does it help at all? If not, why?

As far as I understand the problem, it does help a little bit, but much less than expected. The main slowdown is not in generating the byte code, but in (statically) constructing all the bits and pieces, that constitute a namespace: var objects, fn objects, metadata. This is done on every start, because JVM lacks any kind of data literals (AFAIK, even arrays are constructed piecemeal on the JVM bytecode stack)

 

As far as I know, the JVM already does lazy loading with Java classes essentially in the way that Mike described.

It does, but when require'ing a namespace, every class it comprises is loaded, because the vars + their contents must be reconstructed. This is where Mike's proposal comes in.

BTW Mike: I think your proposal might just work, if we do that kind of lazy loading only for vars containing plain fn forms. It would have to be done very carefully indeed, since even a metadata expression can have side-effects.

Makes me wonder if that kind of elaborate effect-tracking, couldn't be used more generally, to do closed-world tree-shaking on clojure and clojurescript in addition to lazyfying side-effect free code-paths, killing multiple birds with one stone.

Clojurescript already has GClosure, I hear you say? Right, then why does the lower bound for meaningful CLJS programs seem to be at around 600K (minified, uncompressed)?

[Michał Marczyk]

FYI, you may want to have a look at the following commits:

commit 71930b6b6537a796cdf13c4ffa7cf93eb53b6235

Author: Rich Hickey <richh...@gmail.com>

Date:   Mon Feb 28 17:55:14 2011 -0500

    improve startup time via lazy defn loading

commit c5681382da775e898915b17f3ab18b49c65359ec

Author: Rich Hickey <richh...@gmail.com>

Date:   Tue Apr 19 07:41:04 2011 -0400

    temporarily disable lazy fn loading

Cheers,

Michał

--

[Alex Miller]

Lazy var loading is one of the things we're looking at. There is some work on this early in the "direct" branch (although at this point it's pretty out of sync with current code).

[Alex Miller]

Parallel loading is something I've looked at a little bit. Unfortunately theory is easier in theory than in practice. :) While in many cases, it probably works fine, it's possible for there to be problematic ambiguity with which namespace is loaded first, especially for things that update stateful things in the Clojure runtime (protocol extensions, multimethod cases, namespaces, the dynamic classloader, etc). Perhaps you've heard that mutable state and concurrency are problematic. :) I think it's been interesting to see the issues that ClojureScript has found as they've implemented parallel compilation as they are likely similar.

Anyhow, it's certainly worth further consideration. 

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

[Gary Trakhman]

Personally, I've recently fought machine-specific AOT breakage issues by forcing maven-clojure-plugin to only compile a single entry-point namespace, making compilation order deterministic again.  Arguably, it should have worked in any order but this specific issue was hard to track down.

In any case, I think there is existing code that relies on the determinism.

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Marc O'Morain]

On 10 February 2016 at 16:36, Alex Miller <al...@puredanger.com> wrote:
> Parallel loading is something I've looked at a little bit. Unfortunately
> theory is easier in theory than in practice. :) While in many cases, it
> probably works fine, it's possible for there to be problematic ambiguity
> with which namespace is loaded first, especially for things that update
> stateful things in the Clojure runtime (protocol extensions, multimethod
> cases, namespaces, the dynamic classloader, etc). Perhaps you've heard that
> mutable state and concurrency are problematic. :) I think it's been
> interesting to see the issues that ClojureScript has found as they've
> implemented parallel compilation as they are likely similar.

Yeah, I was thinking along these lines too – parallel loading of files
could leave to non-deterministic compilation.

[Alex Miller]

Based on the responses, many people are seeing slow times with "lein repl", which is generally *much* slower than just starting a normal Clojure repl with clojure.main.

If you want to compare, you can use lein to dump your classpath to a file (or the Windows equivalent of these):

  lein classpath >&1 my.cp

Then start the stock Clojure repl using that classpath:

  java -cp `cat my.cp` clojure.main

I think you'll find that the clojure.main repl startup is much faster, especially for bigger classpaths - for a new project with no deps it's about 5s vs 1s for me.

When you do "lein repl", Leiningen will launch an nrepl server, then launch a reply client that connects to that server via nrepl. reply integrates with jline to provide a lot of great command-line help, including completion (via clojure-complete). clojure-complete will scan your classpath to build up completions, incurring a fair amount of overhead. It seems like it does a lot of this work before you even get a live repl to work with - I have not been able to track down why that occurs, but it certainly seems unnecessary.

If someone wanted to work on improving that clojure-complete bit, it would help a lot of people (independent of anything we do in Clojure core). For example, it seems like the completion index could be created in the background and cached in an atom (with re-indexing if needed). Completions could just work from that atom - initially you'd have nothing while the index was being built, but at least you could start the REPL in the meantime.

Hard to say where to file an issue on this - it lies kind of across clojure-complete (which is not really set up this way) and reply which uses it. It would be cool if someone wanted to look into this further.

I didn't look into the nrepl server/client stuff, but I have to imagine there are probably things there that could be faster too.

On Tuesday, February 9, 2016 at 12:36:43 PM UTC-6, Alex Miller wrote:

[Daniel]

Aside from the survey results, is it fair to say that more developers would be interested in Clojure on Android if they could use the blessed compiler? Are there any plans to look into the changes in project Skummet or breaking up the core namespace (like dunaj experimented with)?

[Alex Miller]

> On Feb 10, 2016, at 7:00 PM, Daniel <double...@gmail.com> wrote:
>
> Aside from the survey results, is it fair to say that more developers would be interested in Clojure on Android if they could use the blessed compiler?

Not sure what this refers to? Android is not the focus of my current efforts, although there may be some benefits there.

> Are there any plans to look into the changes in project Skummet or breaking up the core namespace (like dunaj experimented with)?

Not right now.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---

> You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/MG8UTcgFhYc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Ryan Fowler]

On Tue, Feb 9, 2016 at 12:36 PM, Alex Miller <al...@puredanger.com> wrote:

I'm doing some research on slow Clojure boot time and would be interested in collecting info about example use cases where it's been a problem for people.

​The following snippet helps me visualize load times. It might be helpful to others.​ The output is a bit long, so I just added the code and usage/output in a gist at https://gist.github.com/ryfow/4283b64b4dd205d610e8

​​(def ^:dynamic *indent* 0)

(alter-var-root

 #'clojure.core/load

 (fn [orig]

   (fn [& paths]

     (let [t (System/nanoTime)

           r (binding [*indent* (inc *indent*)]

               (apply orig paths))]

       (binding [*out* *err*]

         (println (apply str (repeat *indent* " ")) (/ (- (System/nanoTime) t) 1000000.0)  paths)

         (flush))

       r))))

;; Require namespace in question here

(require 'clojure.core.async)

[Alex Miller]

Nice! I have some stuff similar to this I use, but this is nicely packaged.

[ZhX Chen]

Hi Alex,

    A couple of months ago I did an investigation on the slow boot time of `lein repl`. As you said, a lot of time (around 30% time on my computer) are spent on the features like completion. Also, I notice that there is a parsing library called `sjacket ` at https://github.com/cgrand/sjacket , which provides the functionality to generating the parsing tree on the fly. Unfortunately it seems that the parser itself is constructed every time during the startup time, although the parser itself is always the same every time. My idea the thoughts are put at https://github.com/cgrand/sjacket/issues/24 , and I guess it would be interesting to let somebody who is concerned to notice.


在 2016年2月11日星期四 UTC-5下午4:47:34，Alex Miller写道：