Re: Faster lein

[Ulises]

I've been using drip with quite some success (with the exception of midje tests which seem to launch their own jvm every time.)

On 20 February 2013 15:53, Buck Golemon <workit...@gmail.com> wrote:

I see that lein2 has factored out the 'interactive' command.

Can I use lein1 and expect the various clojure libraries and templates to work?

There's been several mentions of jark in relation to speeding up lein. From what I see, it doesn't seem battle tested. Do any of you use it on a daily basis?

Do you use something else?

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Softaddicts]

SSD + fastest laptop in your price range ;)
lein2 help takes 12 seconds from start to back at command prompt...

We have a huge build here with 9 distinct projects, AOT, ...
lein is fired up maybe a dozen times and it takes a bit
above 3 mns elapsed time and 2mns of cpu time.

Quite happy to live in a io-less world..

As far as lein 1 versus lein 2, the plugin framework has changed so beware of
the plugin versions you may refer to. Not all plugins have been migrated to lein 2.

Any reason why you are moving to lein 2 ?

Luc P.

> I see that lein2 has factored out the 'interactive' command.
>
> Can I use lein1 and expect the various clojure libraries and templates to
> work?
>
> There's been several mentions of jark in relation to speeding up lein. From
> what I see, it doesn't seem battle tested. Do any of you use it on a daily
> basis?
> Do you use something else?
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

--
Softaddicts<lprefo...@softaddicts.ca> sent by ibisMail from my ipad!

[Michael Klishin]

2013/2/20 Buck Golemon <workit...@gmail.com>

Can I use lein1 and expect the various clojure libraries and templates to work?

lein1 is no longer supported. It is a much better idea to move to lein2 and 
use drip or nREPL-based tools such as nREPL.el.

--
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

[Softaddicts]

Drip does not give me any significant advantage on my hardware...
I think that most of this 12 seconds is spent compiling and initializing all the plugins
on top of lein and clojure itself.

One day I will have some time to spit out a tool to create a local AOT version of
all these nice tools to at least skip the compilation phase.

Luc P.

[Phillip Lord]

I've got very variable performance from drip. In some cases, it's
slower.

My guess is that it runs the application in the last JVM, and spins up
the new one to replace it at the same time. On slow machines, this is
problematic.

Personally, I use patience; I find this solves all my problems.

Phil

> --

--
Phillip Lord, Phone: +44 (0) 191 222 7827
Lecturer in Bioinformatics, Email: philli...@newcastle.ac.uk
School of Computing Science, http://homepages.cs.ncl.ac.uk/phillip.lord
Room 914 Claremont Tower, skype: russet_apples
Newcastle University, twitter: phillord
NE1 7RU

[Softaddicts]

You should have worked on a vax-725 with a removable 10mb disk the size if a
large pizza 4 inches thick .

It would have torned your patience to snowflakes which are presently
falling on my head, almost as white as my hairs which have been whitening over
years of waiting after the then really too slow computers.

Technology helps, a lot, even if it cannot solve all the problems.

BTWY, out of 12 seconds elapsed time to start lein, it chews up nearly the same
amount of cpu time.

Aside of getting a marginal increase by over clocking the i7, there ain't much I
can do in terms of external optimizations... drip effectively slows down things
on my hardware having to spin up a replacement JVM.

Luc

[Phil Hagelberg]

Buck Golemon writes:

> Can I use lein1 and expect the various clojure libraries and templates to
> work?

Not really. You could use it on your own projects if you stick to a
subset of project.clj that's supported on both, but you would be
hampered contributing to other projects.

Most templates should work with the lein-newnew plugin though.

> There's been several mentions of jark in relation to speeding up lein. From
> what I see, it doesn't seem battle tested. Do any of you use it on a daily
> basis?

From what I can tell, Jark is not under active development. There are a
number of other efforts to address this problem, including a pure-elisp
replacement for `bin/lein` that communicates over nREPL. I've documented
them here:

https://github.com/technomancy/leiningen/wiki/Faster

Hope that helps.

-Phil

[Phil Hagelberg]

Softaddicts writes:

> SSD + fastest laptop in your price range ;)
> lein2 help takes 12 seconds from start to back at command prompt...

FWIW the help task is basically the worst case scenario for measuring
startup time since it has to load every single task in order to get
docstrings for them. If you just want to measure startup time, use `lein
version` for Leiningen itself and `lein run -m clojure.main -e nil` for
measuring Leiningen plus project boot.

I get about 1.5s for Leiningen alone and just under 4s with a simple
project on my 4-year-old laptop on Leiningen 2.0.0 with no additional
tweaks. Of course, larger projects will take longer, but at that point
it's orthogonal to Leiningen.

-Phil

[AtKaaZ]

on win7 64bit
Microsoft Windows [Version 6.1.7601]

C:\Users\user>cd \1

C:\1>powershell
Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. All rights reserved.

PS C:\1> Measure-Command {lein version}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 3
Milliseconds      : 562
Ticks             : 35624113
TotalDays         : 4.12316122685185E-05
TotalHours        : 0.000989558694444444
TotalMinutes      : 0.0593735216666667
TotalSeconds      : 3.5624113
TotalMilliseconds : 3562.4113



PS C:\1> Measure-Command {lein new someproj1}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 4
Milliseconds      : 362
Ticks             : 43622473
TotalDays         : 5.04889733796296E-05
TotalHours        : 0.00121173536111111
TotalMinutes      : 0.0727041216666667
TotalSeconds      : 4.3622473
TotalMilliseconds : 4362.2473



PS C:\1> cd .\someproj1
PS C:\1\someproj1> Measure-Command {lein uberjar}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 8
Milliseconds      : 461
Ticks             : 84613500
TotalDays         : 9.79322916666667E-05
TotalHours        : 0.002350375
TotalMinutes      : 0.1410225
TotalSeconds      : 8.46135
TotalMilliseconds : 8461.35



PS C:\1\someproj1> Measure-Command {lein help}


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 12
Milliseconds      : 220
Ticks             : 122208485
TotalDays         : 0.000141445005787037
TotalHours        : 0.00339468013888889
TotalMinutes      : 0.203680808333333
TotalSeconds      : 12.2208485
TotalMilliseconds : 12220.8485

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Please correct me if I'm wrong or incomplete,
even if you think I'll subconsciously hate it.

[Softaddicts]

I am around the same figures, mostly cpu time.

If I take our software which is AOT compiled, I can eval nil (or simple values)
through clojure.man in 0.9 seconds. I see this as a significant improvement,
we use about two dozen Clojure libs, most of which get at least partially compiled
prior to launch.

I still wonder how better lein would fare with an AOT version for my specific vm.

I leave for a business trip in a couple weeks. Looks like the kind of project
I could find sone time working on.

I understand not delivering AOT libs through maven not knowing the target
environment but having a tuning tool to speed up loading by avoiding compilation
would help with all the tooling starting to appear.

I suspect that most people are not switching JVMs every hour or so.

Thoughts ?

Luc

[Herwig Hochleitner]

> I understand not delivering AOT libs through maven not knowing the target
> environment but having a tuning tool to speed up loading by avoiding compilation
> would help with all the tooling starting to appear.

How about a lein-bytecache plugin, that can AOT compile and manage the .class files of released jars in the ~/.cache directory.

When that's up and running, leiningen could make use of it for the leiningen jar itself.

2013/2/20 Softaddicts <lprefo...@softaddicts.ca>

[Phil Hagelberg]

Herwig Hochleitner writes:

>> I understand not delivering AOT libs through maven not knowing the target
>> environment but having a tuning tool to speed up loading by avoiding
> compilation
>> would help with all the tooling starting to appear.
>
> How about a lein-bytecache plugin, that can AOT compile and manage the
> .class files of released jars in the ~/.cache directory.
> When that's up and running, leiningen could make use of it for the
> leiningen jar itself.

Unless you want to clear the cache manually every time your dependencies
change, you'd have to make a parallel tree of jars for each version of
Clojure you plan on using.

I think the simplest way to do this would be with a proxying repository
that also cross-compiled jars to new artifacts with added classifiers as
they came through. You'd also need a bit of magic on the Leiningen side
with some middleware in a plugin to add classifiers to every Clojure
dependency. But I have no idea how you would easily tell apart Clojure
jars from Java jars without opening them up to inspect their contents,
which could add a startup time penalty.

Anyway, it can certainly be done. I suspect you'd get more bang for your
buck by just compiling it once and not doing a clean, but protocols kind
of throw a wrench in that plan when it comes to AOT.

-Phil

[Softaddicts]

The byte cache avenue is one, allowing for AOT compilation of artifacts in the local
repo on disk is another one.

Here we use archiva so reloading dependencies does not even require internet.
Still need to see how can this come to play with lein's dependency checking
an avoid resyncing the local repo after changes to the artifacts.

Phil, I can understand why you think it might be an issue, you probably
play with different clojure versions on a daily basis.

My work flow here is kind of different. Moving to a different Clojure version
requires planning and a significant effort, it must be staged in time when
we see a smooth spot in the planning were this can be done.

In prod, a full AOT version would speed up startups. I have this issue with
reply, just cranking up the engine is slow when doing maintenance.

Luc

[Herwig Hochleitner]

2013/2/21 Phil Hagelberg <ph...@hagelb.org>

Unless you want to clear the cache manually every time your dependencies
change, you'd have to make a parallel tree of jars for each version of
Clojure you plan on using.

Ah yes, I forgot that clojure code may compile differently, depending on the clojure version it is used with.

 

I think the simplest way to do this would be with a proxying repository
that also cross-compiled jars to new artifacts with added classifiers as
they came through. You'd also need a bit of magic on the Leiningen side
with some middleware in a plugin to add classifiers to every Clojure
dependency. But I have no idea how you would easily tell apart Clojure
jars from Java jars without opening them up to inspect their contents,
which could add a startup time penalty.

I would imagine that as an opt in solution, where either the library writers and/or users specify that the byte cache should be used for a library.

Then the middleware would only need to look if there is ~/.cache/bytecache/org/example/lib/0.2/clj-1.4/ and add it to the classpath, if available.

 

[Buck Golemon]

So to summarize it seems that one of you uses drip, a couple think it's a non-issue, and the rest want to design a new system.

I take this to mean that there's no widely accepted solution.

I don't/won't use emacs so nREPL.el is out for me. I use vim, so it's most natural for me to have some kind of separate command-line tool.

Really, I just want `lein run` to be faster. Can someone explain where all this time is spent?

I hear a lot of talk of compiling, but why would we re-compile things where none of the dependencies have changed?

[Buck Golemon]

 My own times:

10.2s -- lein help

2.8s -- lein version

4.0s -- lein run nil, in the default template project.

[Buck Golemon]

Thanks!

I'm quite interested in the "interactive session" option, but none of the mechanics are described.

How would I do the equivalent of `lein run` or `line cljs autobuild` in the repl?

Did I miss this in the docs somewhere? It's also quite possible that it's an obvious feature of lisp/clojure that I don't know as a newbie. 

[Softaddicts]

If you look at the dependencies you are using, the Clojure libs
are delivered as source code.

This makes sense, the lib creator/maintainer does not have the slightest idea
of your target runtime (which JVM implementation, which version,...).
There a single version available to all possible target environments.

Everytime you start a new JVM, the name spaces you require directly or
indirectly are recompiled before being used.

If you can avoid restarting your REPL, you save on this overhead, you pay it
only for the source code you change and reload.

This is why I want to see if some plugin could be created to pre-compile
your dependencies. The startup times should improve.

I want also to investigate if lein itself could be pre-compiled by the same
plugin.

Luc P.

> So to summarize it seems that one of you uses drip, a couple think it's a
> non-issue, and the rest want to design a new system.
>
> I take this to mean that there's no widely accepted solution.
>
> I don't/won't use emacs so nREPL.el is out for me. I use vim, so it's most
> natural for me to have some kind of separate command-line tool.
> Really, I just want `lein run` to be faster. Can someone explain where all
> this time is spent?
> I hear a lot of talk of compiling, but why would we re-compile things where
> none of the dependencies have changed?
>
> On Wednesday, February 20, 2013 8:38:10 AM UTC-8, Michael Klishin wrote:
> >
> >

> > 2013/2/20 Buck Golemon <workit...@gmail.com <javascript:>>

> >
> >> Can I use lein1 and expect the various clojure libraries and templates to
> >> work?
> >
> >
> > lein1 is no longer supported. It is a much better idea to move to lein2
> > and
> > use drip or nREPL-based tools such as nREPL.el.
> >
> > --
> > MK
> >
> > http://github.com/michaelklishin
> > http://twitter.com/michaelklishin
> >
>

[Michael Klishin]

2013/3/3 Softaddicts <lprefo...@softaddicts.ca>

I want also to investigate if lein itself could be pre-compiled by the same
plugin.

Lein is AOT compiled. You will find compiled versions of Clojure, REPLy, clj-http

and other dependencies in the standalone jar.

[Buck Golemon]

Thanks Luc.

In summary, the current compile system has no smart way to cache compilation steps, even when it (could) know that the dependencies are unchanged?

I can see that this might be hard, as the jvm itself, and the version of closure are implicit global dependencies. A fully reliable system wouldn't use any of the pre-compiled cache if either of these changed.

[Marko Topolnik]

If you specify AOT compilation for your main namespace (which presumably depends on everything else), the compilation resutls will be saved and reused.

[Phil Hagelberg]

Buck Golemon writes:

> I take this to mean that there's no widely accepted solution.

The widely-accepted solution is to leave a single process running. It
certainly has limitations, but it's the way most people deal with the
problem.

> Really, I just want `lein run` to be faster. Can someone explain where all
> this time is spent?

Basically it comes from having to load two JVMs, one for Leiningen
itself and one for the project. Leiningen itself is fairly optimized for
this (fully AOTed, bytecode verification is turned off, fancy warm-up
JIT techniques disabled) which is why it's possible to get `lein
version` to return in under a second in some cases. But there are
various compatibility issues that prevent us from being able to perform
the same optimizations on project JVMs. These are documented on the
"Faster" page of the Leiningen wiki, and you can do some testing to
determine whether or not they affect your project in particular; if not
then they should provide a good boost.

But nothing will ever come close to the speed of keeping the JVM
resident, which is why I'm working on `:eval-in :nrepl` and lein.el. For
people who don't use Emacs, Jark is the only tool I'm aware of that is
working towards this in a way that's decoupled from the editor. They
could probably use some help both testing and implementing it.

> I hear a lot of talk of compiling, but why would we re-compile things where
> none of the dependencies have changed?

Performing a full AOT of all your dependencies will help if you have a
large project with lots of dependencies that get loaded at application
boot. But that effect would be something along the lines of bringing
boot down from 20s to 12s rather than bringing it from 5s to <1s.

-Phil

[Buck Golemon]

On Mon, Mar 4, 2013 at 10:30 AM, Phil Hagelberg <ph...@hagelb.org> wrote:

Buck Golemon writes:

> I take this to mean that there's no widely accepted solution.

The widely-accepted solution is to leave a single process running. It
certainly has limitations, but it's the way most people deal with the
problem.

Thanks Phil! I posted this question earlier, but it may have gotten lost.

[Phil Hagelberg]

Buck Golemon writes:

> I'm quite interested in the "interactive session" option, but none of
> the mechanics are described. How would I do the equivalent of `lein
> run` or `line cljs autobuild` in the repl? Did I miss this in the docs
> somewhere? It's also quite possible that it's an obvious feature of
> lisp/clojure that I don't know as a newbie.

All that `lein run` does is invoke a function call with the given args.
Which function it invokes is determined by :main in project.clj or
optionally the command-line argument, but once you have a repl open
there's practically nothing more to do to get that functionality.

I've never used lein-cljsbuild, so I don't know what autobuild does. Not
sure if it translates into an interactive context nicely.

-Phil