Leiningen, Clojure and libraries: what am I missing?

4,390 views
Skip to first unread message

Jan Rychter

unread,
Mar 4, 2010, 8:33:12 AM3/4/10
to clo...@googlegroups.com
I haven't hacked on new Clojure stuff for the past two months or
so. Now, having updated my repositories, I find that everybody just
dropped ant and moved to leiningen.

I tried to make sense of things, but can't. I must be missing something
big here.

It seems that leiningen assumes that you are building an application. It
downloads all dependencies and can build an uberjar for you. That's
great, but what if what I have is a bunch of libraries, all being
developed and modified?

Also, I now have at least six clojure jars in ~/.m2 (huh?!), along with
a collection of other assorted stuff. I don't want to use any of these
clojure jars, I have a checked out git repo with clojure and this is the
only code I want to run. Also, swank-clojure insists on having its own
clojure jar in ~/.swank-clojure (why?).

I used to have a bunch of directories with git repos. I would update
those, run ant to gather code into jars, fiddle with the classpath in my
custom-crafted clj script, symlink things around. It was about as bad as
with Common Lisp. But now things got worse -- I no longer feel in
control of anything. Some things won't build, lein tells me about
"artifacts" (huh?!), things get installed in weird places. My clj script
that gets run by SLIME form Emacs can't find libraries that leiningen
downloaded.

How do people deal with this?

As a more general observation, I think that a large part of Perl's and
Python's success was a unified way of dealing with libraries. There
are certain directories where you can drop libraries and expect them to
work (be found). There is ONE way of running the VM (one script, blessed
by the language creator). Then on top of that there is CPAN, which plugs
into that, downloading dependencies and dropping libraries into those
well-known directories. It isn't perfect, but it works, and is
predictable.

Shouldn't we have a well-known Single Setup for Clojure as well?

I am very worried that Clojure is rapidly going the Common Lisp way,
with each developer having his own precious carefully crafted
setup. Every system is different, everyone uses a different script to
run clojure programs, there is no single place to drop your libraries
into, and libraries have no conventions on where their code is (what do
I add to classpath once I have a library checked out?). I thought
Leiningen was supposed to solve these issues, but either I am missing
something, or it is really a tool to manage dependencies for a single
application.

--J.

Stuart Halloway

unread,
Mar 4, 2010, 8:52:34 AM3/4/10
to clo...@googlegroups.com
Hi Jan,

My 2c: Leiningen is an important step, but there is still plenty to do.

My workflow: I am developing multiple Clojure libraries and
applications, with some of the dependencies changing daily. Whenever I
want to make a change, I:

In the project being changed:
git tag a new version number
update the project.clj to a new version number
run "lein pom" and "lein jar"
copy the new files to a repos (clojars)

In projects that depend on changed project
update the dependency in project.clj (if it is hard coded and not >=)
run lein deps

This seems adequate to me (although there could be a bit more
automation). The problem of snapshot dependencies remains. The
solution that springs to mind here is to get projects off snapshots
(presumably by releasing more incremental versions). Library authors:
please release lots of versions! (or tell me why this is a bad idea).

Some needs I see:

(0) More point releases of key libs (or some other solution to the
snapshots problem)

(1) Unify around the project structure imposed by lein, and/or

(2) Improve the project structure imposed by lein, if it is broken in
some specific way. (Maybe add a well-known config file for adding
elements to the classpath?)

(3) "lein deps" (or some other task) to clean up old jars. Right now I
am deleting them by hand from the lib directory. (Is this already
solved?)

Stu

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient
> with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

Stuart Halloway

unread,
Mar 4, 2010, 8:54:14 AM3/4/10
to clo...@googlegroups.com
> "artifacts" (huh?!), things get installed in weird places. My clj
> script
> that gets run by SLIME form Emacs can't find libraries that leiningen
> downloaded.

"lein deps" copies dependencies into the project's lib directory.
Various other lein commands assume lib/*.jar is on the classpath, and
if you are writing scripts you should assume that classpath as well.

Is this good enough?

Stu

mac

unread,
Mar 4, 2010, 9:39:48 AM3/4/10
to Clojure
The .m2 folder is a Maven 2 convention. Leiningen uses Maven
internally.
.m2 contains your "local repository" - the single standard place where
stuff gets downloaded to.
When doing lein deps in a project folder any dependencies are
downloaded to your local repo first (unless you already had the
library in your .m2 folder) and then copied to the projects local lib
folder.
Maven is largely convention based, if anything about leiningen seems
weird and you can't find it in the lein docs try browsing the Maven 2
docs.

Stuart Sierra

unread,
Mar 4, 2010, 10:19:31 AM3/4/10
to Clojure
On Mar 4, 8:52 am, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> My 2c: Leiningen is an important step, but there is still plenty to do.

Oh yeah.

> Some needs I see:
>
> (0) More point releases of key libs (or some other solution to the  
> snapshots problem)

Yes!

> (1) Unify around the project structure imposed by lein, and/or

As many of you know, I've always been tepid on lein. I'd rather go
with Maven whole-hog, because that offers the most robust model for
incorporating Java libraries.

The more we can leverage the Java tool set, the better. I'd rather
lein were just a thin POM generator for those who can't stand writing
XML.

-SS

Konrad Hinsen

unread,
Mar 4, 2010, 10:59:15 AM3/4/10
to clo...@googlegroups.com
On 04.03.2010, at 14:33, Jan Rychter wrote:

> It seems that leiningen assumes that you are building an application. It
> downloads all dependencies and can build an uberjar for you. That's
> great, but what if what I have is a bunch of libraries, all being
> developed and modified?

That's my situation as well. I did start to use Leiningen, but only for producing releases for the outside world. On my development machine, it's the source code directories of all libraries that are on my classpath. That way I am sure which version of the code I am running.

> with Common Lisp. But now things got worse -- I no longer feel in
> control of anything. Some things won't build, lein tells me about
> "artifacts" (huh?!), things get installed in weird places. My clj script
> that gets run by SLIME form Emacs can't find libraries that leiningen
> downloaded.
>
> How do people deal with this?

Here's what works for me.

Each of my libraries is one "project" in the sense of swank-clojure. I use swank-clojure-project to start Clojure inside slime, and that takes care of my classpath. All I need to do is have the right stuff in the lib subdirectory of each project, so that's where are all the configuration is done.

Instead of doing something like "lein deps" to fill the lib subdirectories, I decided to manage them manually and have only soft links in there that point to the real jars and source directories. There's one soft link to the clojure jar (the one I rebuild after each "git pull" from github), one for clojure-contrib (same principle), and one for swank-clojure (same again). All other dependencies on Clojure libraries are soft links to the source code directory inside the respective project directories. Dependencies on external Java libs are soft links to the jar files, which reside in a single place on my hard disk.

> As a more general observation, I think that a large part of Perl's and
> Python's success was a unified way of dealing with libraries. There
> are certain directories where you can drop libraries and expect them to
> work (be found). There is ONE way of running the VM (one script, blessed
> by the language creator). Then on top of that there is CPAN, which plugs
> into that, downloading dependencies and dropping libraries into those
> well-known directories. It isn't perfect, but it works, and is
> predictable.

Right, but its imperfections have bothered me often enough. For once, if you have different applications that require different versions of the same library, you are in for a major headache. Python setuptools and its eggs are a partial solution, but add a lot of complexity of their own. Another source of endless frustration is maintaining a personal library installation on a machine where I don't have administrator rights and need to work with multiple Python versions. It's possible, but it's a big mess and I need to be very careful to be sure that I really use the version that I need.

I am not too unhappy with the JVM approach to configurability via classpath, but it requires a layer of management tools and/or conventions to be productive. What doesn't fit with my JVM world-view is Maven, and thus leiningen. Their dependency handling model is clearly geared towards application deployment rather than library development. Moreover, I think their model is too complicated, making it difficult to predict which jars will end up being used where. But perhaps that's just my lack of experience.

> Shouldn't we have a well-known Single Setup for Clojure as well?

It would perhaps be a good idea to have one default setup, with the option for expert users to roll their own. For newcomers the current situation is definitely too messy.

> I add to classpath once I have a library checked out?). I thought
> Leiningen was supposed to solve these issues, but either I am missing
> something, or it is really a tool to manage dependencies for a single
> application.

That's my view of what it is, but I can't claim much experience with it.

Konrad.

Praki Prakash

unread,
Mar 4, 2010, 10:11:59 AM3/4/10
to clo...@googlegroups.com
Lein could be a step in the right direction, but I wonder how it will
manage to evolve given the complex tools it uses internally.

I have also seen a weird issue in my code base which has two classes
derived from java.lang.Exception. If I do a clean and try compile, I
get an error saying that my.package.MyException class not found. My
workaround is, set :namespaces to only those Exception derived
classes, compile and then set :namespaces to :all.

Maybe I am missing some cool lein thing here or lein needs quite a bit
of work in dependency handling. I am tempted to just stick to mvn
which is quite mature and well-documented. Any clojure-specific needs
could be addressed through a plug-in.

Comments, thoughts?

Thanks,
Praki

--
http://www.google.com/profiles/praki.prakash

Meikel Brandmeyer

unread,
Mar 4, 2010, 12:55:22 PM3/4/10
to clo...@googlegroups.com
Hi,

On Thu, Mar 04, 2010 at 07:11:59AM -0800, Praki Prakash wrote:

> I have also seen a weird issue in my code base which has two classes
> derived from java.lang.Exception. If I do a clean and try compile, I
> get an error saying that my.package.MyException class not found. My
> workaround is, set :namespaces to only those Exception derived
> classes, compile and then set :namespaces to :all.

You have to :require the namespaces implementing the exceptions in the
files were you use the Exceptions. This will resolve the dependency
problem. See here: http://tr.im/FNmt

Sincerely
Meikel

James Reeves

unread,
Mar 4, 2010, 1:00:07 PM3/4/10
to clo...@googlegroups.com
On 4 March 2010 10:19, Stuart Sierra <the.stua...@gmail.com> wrote:
> As many of you know, I've always been tepid on lein.  I'd rather go
> with Maven whole-hog, because that offers the most robust model for
> incorporating Java libraries.

Out of interest, what scenarios does Maven account for that Leiningen
currently does not?

> The more we can leverage the Java tool set, the better.  I'd rather
> lein were just a thin POM generator for those who can't stand writing
> XML.

I personally favour the opposite approach :). Whilst Java provides a
lot of useful functionality, I've never been particularly impressed
with the way most Java libraries are designed.

- James

David Nolen

unread,
Mar 4, 2010, 1:38:25 PM3/4/10
to clo...@googlegroups.com
On Thu, Mar 4, 2010 at 10:59 AM, Konrad Hinsen <konrad...@fastmail.net> wrote:
Right, but its imperfections have bothered me often enough. For once, if you have different applications that require different versions of the same library, you are in for a major headache. Python setuptools and its eggs are a partial solution, but add a lot of complexity of their own. Another source of endless frustration is maintaining a personal library installation on a machine where I don't have administrator rights and need to work with multiple Python versions. It's possible, but it's a big mess and I need to be very careful to be sure that I really use the version that I need.

Imperfections meaning it's perfectly broken :) That's why tools like virtualenv exist for Python.

David 

David Nolen

unread,
Mar 4, 2010, 1:57:17 PM3/4/10
to clo...@googlegroups.com
On Thu, Mar 4, 2010 at 8:33 AM, Jan Rychter <j...@rychter.com> wrote:
Also, I now have at least six clojure jars in ~/.m2 (huh?!), along with
a collection of other assorted stuff. I don't want to use any of these
clojure jars, I have a checked out git repo with clojure and this is the
only code I want to run. Also, swank-clojure insists on having its own
clojure jar in ~/.swank-clojure (why?).

Except the libraries you're consuming with Lein might require specific versions of Clojure and not the bleeding edge.
 
I used to have a bunch of directories with git repos. I would update
those, run ant to gather code into jars, fiddle with the classpath in my
custom-crafted clj script, symlink things around. It was about as bad as
with Common Lisp. But now things got worse -- I no longer feel in
control of anything. Some things won't build, lein tells me about
"artifacts" (huh?!), things get installed in weird places. My clj script
that gets run by SLIME form Emacs can't find libraries that leiningen
downloaded.

In my experience haven gone through everything you've described Lein has made things at least an order of magnitude simpler. You no longer have to do things by hand. Even better you can use other people's libraries in a much more straightforward manner.

Case in point Penumbra, an OpenGL library for Clojure. To get Penumbra to work w/o Lein you need to:

a) get the Penumbra library from source
b) get the LWJGL 2.2.2 jars and native libraries
c) know how to set your class path to point to not only the jars but to the correct native libraries for you particular platform/architecture.

This is nightmarish to say the least. Now with lein to use Penumbra you only have to remember 3 commands

lein deps
lein native-deps (I added this)
lein swank (just slime-connect from Emacs)

For most libraries it's just, lein deps, lein swank.

As a more general observation, I think that a large part of Perl's and
Python's success was a unified way of dealing with libraries. There
are certain directories where you can drop libraries and expect them to
work (be found). There is ONE way of running the VM (one script, blessed
by the language creator). Then on top of that there is CPAN, which plugs
into that, downloading dependencies and dropping libraries into those
well-known directories. It isn't perfect, but it works, and is
predictable.

As has been said "one place" presents serious problem for anything beyond the simplest libraries with only trivial dependencies. I'm not saying lein is the end all be all. But it removes a fairly serious pain point for Clojure - managing your classpath and using other people's libraries.

David

David Nolen

unread,
Mar 4, 2010, 1:59:31 PM3/4/10
to clo...@googlegroups.com
On Thu, Mar 4, 2010 at 10:19 AM, Stuart Sierra <the.stua...@gmail.com> wrote:
As many of you know, I've always been tepid on lein.  I'd rather go
with Maven whole-hog, because that offers the most robust model for
incorporating Java libraries.

If Lein evolves to to handle dependencies of dependencies and intelligently generates the classpath based on these dependencies (instead of copying files around) what advantage does Maven really have?

David

Konrad Hinsen

unread,
Mar 4, 2010, 3:02:22 PM3/4/10
to clo...@googlegroups.com
On 4 Mar 2010, at 19:38, David Nolen wrote:

> Imperfections meaning it's perfectly broken :) That's why tools like
> virtualenv exist for Python.

I'd say it's "New Jersey style": good enough for 90% of the users, but
indeed fundamentally broken.

Konrad.

Chris Perkins

unread,
Mar 4, 2010, 12:39:40 PM3/4/10
to Clojure
On Mar 4, 8:33 am, Jan Rychter <j...@rychter.com> wrote:
> I haven't hacked on new Clojure stuff for the past two months or
> so. Now, having updated my repositories, I find that everybody just
> dropped ant and moved to leiningen.

> How do people deal with this?

I don't have any good answers for you; I just want to say "I feel your
pain".

The situation is worse on Windows, where I can't even get leiningen to
work at all. Luckily I'm only working on one clojure project, so I
just copy clojure.jar and clojure-contrib.jar into my project's lib
directory, as well as a faked version of swank-clojure.jar (just a
zipfile of swank source, renamed to ".jar"). That works well enough
for me. Then I have a tiny little Rakefile that builds a classpath
string from the contents of lib/, and has two tasks: run tests, and
start a repl (which I rarely use anymore, since I got swank-clojure
working).

> As a more general observation, I think that a large part of Perl's and
> Python's success was a unified way of dealing with libraries. There
> are certain directories where you can drop libraries and expect them to
> work (be found). There is ONE way of running the VM (one script, blessed
> by the language creator). Then on top of that there is CPAN, which plugs
> into that, downloading dependencies and dropping libraries into those
> well-known directories. It isn't perfect, but it works, and is
> predictable.

Actually, I have noticed a move towards using tools like virtualenv
for Python projects, precisely because depending on system-wide
libraries makes it too non-obvious exactly what your external
dependencies are, as well as creating potential version conficts.

-- Chris

Glen Stampoultzis

unread,
Mar 4, 2010, 6:32:09 PM3/4/10
to clo...@googlegroups.com
On 5 March 2010 04:39, Chris Perkins <chrispe...@gmail.com> wrote:

The situation is worse on Windows, where I can't even get leiningen to
work at all.  Luckily I'm only working on one clojure project, so I
just copy clojure.jar and clojure-contrib.jar into my project's lib
directory, as well as a faked version of swank-clojure.jar (just a
zipfile of swank source, renamed to ".jar").  That works well enough
for me. Then I have a tiny little Rakefile that builds a classpath
string from the contents of lib/, and has two tasks: run tests, and
start a repl (which I rarely use anymore, since I got swank-clojure
working).

 
Windows seems to be a second class citizen as far as clojure tools go.  I wasn't able to figure out how to get it going on Windows either which makes things much harder considering so many clojure projects are now using it.

Getting swank-clojure going in Windows is also a pain.  To install it the recommended way via ELPA you need to patch the ELPA source code first.  This isn't documented at the swank-clojure site either.

I'm hopeful this situation will improve going forward but right now things are still very bleeding edge.


Antony Blakey

unread,
Mar 4, 2010, 9:23:45 PM3/4/10
to clo...@googlegroups.com, David Nolen

The tools (e.g. Archiva, Nexus et al), documentation, IDE support, integration with other languages e.g. mixed Java/Clojure/Scala/Groovy projects. The more clojure integrates with the existing ecosystem, the more likely it is to succeed, especially in a viral sense. This is the essence of clojure being a JVM language with great integration with Java.

Personally I wish that leiningen effort was instead put towards polyglot maven http://polyglot.sonatype.org/

Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787

Nothing is really work unless you would rather be doing something else.
-- J. M. Barre


Brian Schlining

unread,
Mar 4, 2010, 9:41:17 PM3/4/10
to clo...@googlegroups.com
>
> If Lein evolves to to handle dependencies of dependencies and intelligently generates the classpath based on these dependencies (instead of copying files around) what advantage does Maven really have?

The tools (e.g. Archiva, Nexus et al), documentation, IDE support, integration with other languages e.g. mixed Java/Clojure/Scala/Groovy projects. The more clojure integrates with the existing ecosystem, the more likely it is to succeed, especially in a viral sense. This is the essence of clojure being a JVM language with great integration with Java.

Well said! Here's a pair of helpful links or anyone who wants to Maven + Clojure for a spin:

 
And for those who haven't tried it, you can run  'mvn clojure:repl'  in your project to get a REPL running with all the dependencies (including transitive ones) defined in your pom.xml

--
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Brian Schlining
bschl...@gmail.com

Felix Breuer

unread,
Mar 4, 2010, 7:03:03 PM3/4/10
to Clojure
I am also on Windows and have struggled a good deal with Leiningen (I
would like to avoid wrangling classpaths myself as far as possible). I
finally got Leiningen to run, but some features like lein swank still
elude me (though this is a problem of swank-clojure and not of
Leiningen). I agree that Windows is a second class citizen as far as
clojure tools go. For most of my clojure work I run Ubuntu inside
VirtualBox - but I would rather use clojure + tools from Windows
directly.

Felix

On 5 Mrz., 00:32, Glen Stampoultzis <gst...@gmail.com> wrote:

Meikel Brandmeyer

unread,
Mar 5, 2010, 1:22:10 AM3/5/10
to Clojure
Hi,

On Mar 5, 1:03 am, Felix Breuer <fe...@fbreuer.de> wrote:

> I agree that Windows is a second class citizen as far as clojure
> tools go.

Oh please stop that. I have a stable setup of Gradle + Clojuresque +
VimClojure on Windows. Granted setting up VimClojure on Windows is
tricky due to Vim and Windows specifics, but it's doable. However the
Gradle + Clojuresque part is as easy and trivial as it can get. And
this setup survived already a few updates of the different components
without blowing up a single time. I type "gradle runServer" in the
morning and get a running Jetty/Nailgun combination for dynamic
development with VimClojure - without thinking a single time about the
classpath. Which actually consists of several checked out other
projects (enlive, ring, fleetdb, ...). They were modified non-
intrusively to use clojuresque instead of maven/ant/lein/whatever as
build system. It checks the interdependencies and rebuilds the
required jars if necessary. A change in a dependency is immediately
picked up. If necessary, I can fix a known-to-work version in my local
repository with a few commands.

All without virtualbox, VMWare or whatever. So if one tool doesn't
fill your need, then choose another.

I apologise for sounding a little harsh.

Sincerely
Meikel

Martin DeMello

unread,
Mar 5, 2010, 1:47:37 AM3/5/10
to clo...@googlegroups.com
On Thu, Mar 4, 2010 at 7:03 PM, Jan Rychter <j...@rychter.com> wrote:
>
> As a more general observation, I think that a large part of Perl's and
> Python's success was a unified way of dealing with libraries. There
> are certain directories where you can drop libraries and expect them to
> work (be found). There is ONE way of running the VM (one script, blessed
> by the language creator). Then on top of that there is CPAN, which plugs
> into that, downloading dependencies and dropping libraries into those
> well-known directories. It isn't perfect, but it works, and is
> predictable.

One huge drawback I've found with clojure (which it doubtless
inherited from Java) is that you need an actual jarfile in your
classpath, not just the directory containing the jarfile.

martin

Meikel Brandmeyer

unread,
Mar 5, 2010, 2:04:28 AM3/5/10
to Clojure
Hi,

On Mar 5, 7:47 am, Martin DeMello <martindeme...@gmail.com> wrote:

> One huge drawback I've found with clojure (which it doubtless
> inherited from Java) is that you need an actual jarfile in your
> classpath, not just the directory containing the jarfile.

With newer Java versions you can specify wildcards. java -cp lib/
\* ... will pick up everything in lib for the classpath.

Sincerely
Meikel

Rob Wolfe

unread,
Mar 5, 2010, 2:53:36 AM3/5/10
to Clojure

On 5 Mar, 01:03, Felix Breuer <fe...@fbreuer.de> wrote:
> I am also on Windows and have struggled a good deal with Leiningen (I

[...]

> On 5 Mrz., 00:32, Glen Stampoultzis <gst...@gmail.com> wrote:
>
>
>
> > On 5 March 2010 04:39, Chris Perkins <chrisperkin...@gmail.com> wrote:
>
> > > The situation is worse on Windows, where I can't even get leiningen to
> > > work at all.  Luckily I'm only working on one clojure project, so I

[...]

> > Windows seems to be a second class citizen as far as clojure tools go.  I
> > wasn't able to figure out how to get it going on Windows either which makes
> > things much harder considering so many clojure projects are now using it.

[...]


First of all have you ever tried this script?
http://github.com/technomancy/leiningen/blob/master/bin/lein.bat

Secondly have you seen this thread?
http://groups.google.com/group/leiningen/browse_thread/thread/2474d2e9019ee29c
There was almost no feedback at all.

Have you reported any issue with Leiningen on Windows?
http://github.com/technomancy/leiningen/issues

Leiningen is not perfect and I'd like to improve this tool (including
Windows version),
but so far I can see only some general complaining. Please report
concrete
issues and we will be able to make concrete improvements.

Br,
Rob

Martin DeMello

unread,
Mar 5, 2010, 4:27:00 AM3/5/10
to clo...@googlegroups.com

Neat! Never knew that. So now it *is* possible to have a standard
clojure lib directory the way ruby has its gems directory or python
its site-libs.

martin

Laurent PETIT

unread,
Mar 5, 2010, 4:27:27 AM3/5/10
to clo...@googlegroups.com
2010/3/5 Meikel Brandmeyer <m...@kotka.de>:

> Hi,
>
> On Mar 5, 1:03 am, Felix Breuer <fe...@fbreuer.de> wrote:
>
>> I agree that Windows is a second class citizen as far as clojure
>> tools go.
>
> Oh please stop that. I have a stable setup of Gradle + Clojuresque +
> VimClojure on Windows.

Furthermore, all java tools play well on windows, so using ant, maven,
will work just as well on windows and on linux, mac, etc.

And you also have the Eclipse plugin for Clojure, the netbeans plugin
for clojure, the IntelliJ IDEA plugin for clojure, which work on
windows too !

> Granted setting up VimClojure on Windows is
> tricky due to Vim and Windows specifics, but it's doable. However the
> Gradle + Clojuresque part is as easy and trivial as it can get. And
> this setup survived already a few updates of the different components
> without blowing up a single time. I type "gradle runServer" in the
> morning and get a running Jetty/Nailgun combination for dynamic
> development with VimClojure - without thinking a single time about the
> classpath. Which actually consists of several checked out other
> projects (enlive, ring, fleetdb, ...). They were modified non-
> intrusively to use clojuresque instead of maven/ant/lein/whatever as
> build system. It checks the interdependencies and rebuilds the
> required jars if necessary. A change in a dependency is immediately
> picked up. If necessary, I can fix a known-to-work version in my local
> repository with a few commands.
>
> All without virtualbox, VMWare or whatever. So if one tool doesn't
> fill your need, then choose another.
>
> I apologise for sounding a little harsh.
>
> Sincerely
> Meikel
>

Stuart Sierra

unread,
Mar 5, 2010, 11:17:49 AM3/5/10
to Clojure
On Mar 4, 1:59 pm, David Nolen <dnolen.li...@gmail.com> wrote:
> If Lein evolves to to handle dependencies of dependencies and intelligently
> generates the classpath based on these dependencies (instead of copying
> files around) what advantage does Maven really have?

Maven supports a wide range of development/deployment models already.
Whatever you want to do (including compile/test/run Clojure code)
there's a plugin for that. Why recreate all this functionality in
Leiningen?

Yes, the Maven configuration model isn't pretty, but if you follow the
conventions it works. Polyglot Maven will obviate the need to write
XML. But frankly, I don't mind a little XML if it gets the job done.

Basically, I'm skeptical that any tool developed exclusively by/for
the Clojure community could ever achieve the scope and utility of a
tool used by thousands of people and companies. We're a small
community, we can't do it all alone.

-SS

Ramakrishnan Muthukrishnan

unread,
Mar 5, 2010, 11:43:21 AM3/5/10
to clo...@googlegroups.com
On Fri, Mar 5, 2010 at 9:47 PM, Stuart Sierra
<the.stua...@gmail.com> wrote:
>
> Yes, the Maven configuration model isn't pretty, but if you follow the
> conventions it works.  Polyglot Maven will obviate the need to write

I didn't know about Polyglot. Thanks. Looks like a very nice tool.

<http://polyglot.sonatype.org/clojure.html>

--
Ramakrishnan

James Reeves

unread,
Mar 5, 2010, 1:01:35 PM3/5/10
to Clojure
On Mar 5, 11:17 am, Stuart Sierra <the.stuart.sie...@gmail.com> wrote:
> Maven supports a wide range of development/deployment models already.
> Whatever you want to do (including compile/test/run Clojure code)
> there's a plugin for that.  Why recreate all this functionality in
> Leiningen?

Most of the time, I won't need the wide range of functionality that
Maven supports. For the most common tasks, such as compiling and
packaging, Leiningen is the better build tool. It's conceptually
simpler, more understandable, and easier to extend. I'd rather use a
build tool that covers 80% of tasks well, than a tool that covers 95%
poorly.

- James

Phil Hagelberg

unread,
Mar 5, 2010, 1:27:08 PM3/5/10
to clo...@googlegroups.com
On Thu, Mar 4, 2010 at 5:52 AM, Stuart Halloway
<stuart....@gmail.com> wrote:
> Some needs I see:
> (0) More point releases of key libs (or some other solution to the snapshots
> problem)

Agreed.

> (1) Unify around the project structure imposed by lein, and/or

Yes, but it should be noted this isn't really the "leiningen" project
structure; it has been the conventional Clojure layout in the
community from long before leiningen was released.

> (2) Improve the project structure imposed by lein, if it is broken in some
> specific way. (Maybe add a well-known config file for adding elements to the
> classpath?)

People have asked for this, but I have yet to hear a good use case for
it. Mostly it's been requested by people who don't realize how easy it
is to push their own jars to clojars. But if someone wants to
implement it I will not refuse the patch.

> (3) "lein deps" (or some other task) to clean up old jars. Right now I am
> deleting them by hand from the lib directory. (Is this already solved?)

Should be able to "lein clean". =)

-Phil

Stuart Halloway

unread,
Mar 5, 2010, 1:49:18 PM3/5/10
to clo...@googlegroups.com
>> (2) Improve the project structure imposed by lein, if it is broken
>> in some
>> specific way. (Maybe add a well-known config file for adding
>> elements to the
>> classpath?)
>
> People have asked for this, but I have yet to hear a good use case for
> it. Mostly it's been requested by people who don't realize how easy it
> is to push their own jars to clojars. But if someone wants to
> implement it I will not refuse the patch.

I agree that lein+clojars is good enough for lib dependencies. My two
use cases so far are places where the classpath needs to point to
something other than outside libs:

(1) getting ./config on the classpath so I can keep properties files,
json config, whatever, separate from code and lib

(2) having more than one kind of tests, in different dirs (e.g. specs,
features) and having their classpaths work through lein.

Stu

Neill Alexander

unread,
Mar 5, 2010, 2:53:06 PM3/5/10
to Clojure
I'll add my 2 cents worth. I've come from a Java development
background to Clojure. A couple of years ago our company migrated from
Ant to Maven. At first I really really hated it. I found it difficult
to make it do what I wanted it to do. Nowadays, though, it is my build
tool of choice.

The turning point was when I realized that the key to using Maven is
understanding the Maven model - understanding how Maven wants to
build. You don't tell Maven what to do. You just put the files in the
places Maven expects, and Maven then _knows_ what to do. My rule of
thumb is: if I'm spending a lot of time tweaking configuration and
overriding defaults in the pom file then I'm probably doing it wrong.

When I started programming in Clojure I immediately looked for a Maven
plugin to build Clojure projects. The first project I worked on I did
using the plugin. For my next Clojure project I decided to give
Leiningen a spin. Initially I was very impressed. It was easy to get a
project up and running, easy to build, easy to run tests, easy to run
a swank server etc. For a couple of weeks I was very happy. But.....

I finally hit a point where Leiningen was no longer working for me. My
current project has a Swing GUI, built using Matisse (the Netbeans GUI
builder). I couldn't find a way to build the Swing portion and the
Clojure portion together using Leiningen. I also couldn't find a way
to divide my project up into different sub-modules with one build
(maybe I'm guilty of trying to use Leiningen like I use Maven). In any
case, I ended up converting the project to use Maven.

Maven has a steep learning curve, but once you get it, it's incredibly
powerful.

"The philosophy of Tai Chi Chuan is that if one uses hardness to
resist violent force, then both sides are certain to be injured at
least to some degree. Such injury, according to tai chi theory, is a
natural consequence of meeting brute force with brute force. Instead,
students are taught not to directly fight or resist an incoming force,
but to meet it in softness and follow its motion while remaining in
physical contact until the incoming force of attack exhausts itself or
can be safely redirected, meeting yang with yin." (http://
en.wikipedia.org/wiki/Tai_chi_chuan)

If you fight with Maven, you will lose. If you meet it in softness and
follow its motion you will attain build enlightenment...

;-)

FInally, I agree with Stuart that polyglot Maven could / should be the
way forward. I don't really wrestle with pom files much manually
anyway since Netbeans does a very good job of automatically updating
them. But Leiningen has given me a taste of writing build files in
Clojure, so I will definitely be investigating the polyglot option
soon.

To learn more about Maven I would recommend
http://www.sonatype.com/books/mvnref-book/reference/public-book.html
and http://www.sonatype.com/books/mvnex-book/reference/public-book.html

Rick Moynihan

unread,
Mar 5, 2010, 5:38:27 PM3/5/10
to clo...@googlegroups.com

As I'm one of the people that's requested this feature... I'll add my
reason too:

(3) Managing proprietary libraries and API's that you are developing
or using... Yes you can throw these into your own maven repo, but
then you also need to manage, configure and secure a server (when
working in a team).

R.

Brian Schlining

unread,
Mar 5, 2010, 6:56:39 PM3/5/10
to clo...@googlegroups.com

As I'm one of the people that's requested this feature... I'll add my
reason too:

(3) Managing proprietary libraries and API's that you are developing
or using...  Yes you can throw these into your own maven repo, but
then you also need to manage, configure and secure a server (when
working in a team).

Actually, you can create a project local directory (e.g. myproject/lib or the more maven-ish myproject/src/lib) that acts as a maven repository and stick your dependencies in that directory. Then you can check it in to source control with your project. See http://stubbisms.wordpress.com/2008/08/28/maven-is-to-ant-as-a-nail-gun-is-to-hammer-and-nails-you-need-to-move-on/ and http://techdiary.peterbecker.de/2009/03/letting-maven-use-local-libraries.html for more details.

Cheers

Phil Hagelberg

unread,
Mar 6, 2010, 12:58:29 AM3/6/10
to clo...@googlegroups.com
On Fri, Mar 5, 2010 at 2:38 PM, Rick Moynihan <rick.m...@gmail.com> wrote:
> (1) getting ./config on the classpath so I can keep properties files, json
> config, whatever, separate from code and lib

This is actually what the resources/ directory is for.

> (2) having more than one kind of tests, in different dirs (e.g. specs,
> features) and having their classpaths work through lein.

It would be best if this could be handled by plugins rather than
specified in every project.clj. Currently the plugin framework doesn't
expose the hooks that would be needed for this, but it's definitely on
the short list for version 1.2.0.

> (3) Managing proprietary libraries and API's that you are developing
> or using...  Yes you can throw these into your own maven repo, but
> then you also need to manage, configure and secure a server (when
> working in a team).

I think it would be great if it were easy to run your own clojars
server. Not only for mirroring purposes (poor ato isn't going to be
able to eat the bandwidth bills forever) but also to deal with
situations like this.

But there may be other valid reasons that nobody has even encountered
yet, so I favour having total flexibility anyway.

-Phil

Phil Hagelberg

unread,
Mar 6, 2010, 12:58:49 AM3/6/10
to clo...@googlegroups.com
On Fri, Mar 5, 2010 at 3:56 PM, Brian Schlining <bschl...@gmail.com> wrote:
>> (3) Managing proprietary libraries and API's that you are developing
>> or using...  Yes you can throw these into your own maven repo, but
>> then you also need to manage, configure and secure a server (when
>> working in a team).
>
> Actually, you can create a project local directory (e.g. myproject/lib or
> the more maven-ish myproject/src/lib) that acts as a maven repository and
> stick your dependencies in that directory. Then you can check it in to
> source control with your project.

If you're using git, this should be avoided. git is remarkably bad at
storing binary data. A number of projects I've been watching have made
this mistake, and they've all had to correct it by rewriting history
down the line, which is very awkward.

-Phil

James Reeves

unread,
Mar 6, 2010, 12:11:01 PM3/6/10
to Clojure
On Mar 6, 5:58 am, Phil Hagelberg <p...@hagelb.org> wrote:
> If you're using git, this should be avoided. git is remarkably bad at
> storing binary data.

I seem to recall that recent versions of Git have improved
significantly in this area. However, there's still the fundamental
problem that changing the dependencies increases the size of the
project. New developers will not only have to download your source
files, but the binary diff of every dependency change.

So yep, it's a bad idea :)

- James

Brian Schlining

unread,
Mar 6, 2010, 2:00:30 PM3/6/10
to clo...@googlegroups.com
> If you're using git, this should be avoided. git is remarkably bad at
> storing binary data.

I seem to recall that recent versions of Git have improved
significantly in this area. However, there's still the fundamental
problem that changing the dependencies increases the size of the
project. New developers will not only have to download your source
files, but the binary diff of every dependency change.

So yep, it's a bad idea :)

I just want to point out that a maven repo manages versions as separate copies. e.g. Using our hypothetical project local repository, the paths to 2 versions would be something like:

--myproject/src/lib/com/myapi/foo/1.0/foo-1.0.jar
--myproject/src/lib/com/myapi/foo/2.0/foo-2.0.jar

So yes, if you check it into version control, it increases the size of your project, but you aren't necessarily storing any binary diffs.

Glen Stampoultzis

unread,
Mar 7, 2010, 9:44:44 AM3/7/10
to clo...@googlegroups.com
Hi Meikel, sorry if my email came off as being a bit critical but I'm not sure what I said was necessarily inaccurate - perhaps a little unhelpful.  It seems (from my limited observations) that most of the Clojure developers are Linux/Mac people so it's understandable that Windows isn't currently as well supported by all tools.  That is meant more as a general observation than a criticism.

My own problems getting emacs set up were more of an issue with ELPA than Clojure-Swank in particular.  The Lein setup is a known shortcoming currently that hopefully will be corrected at some point.  I hope so because it seems to be becoming a standard in the community.  If my Clojure improves maybe Windows support is something I can take a look at myself.


Meikel Brandmeyer

unread,
Mar 7, 2010, 10:40:55 AM3/7/10
to clo...@googlegroups.com
Hi,

On Mon, Mar 08, 2010 at 01:44:44AM +1100, Glen Stampoultzis wrote:

> It seems (from my limited observations) that most of the Clojure developers
> are Linux/Mac people so it's understandable that Windows isn't currently as
> well supported by all tools.

Windows has a fairly big share, I think. There are a lot of posts (blog,
group, pm, IRC, ...) where the classpath starts with C:\Clojure.

My setup was just an example. I gave it because I use it actively, so I
could also answer specific questions. Laurent gave a broader view on the
tool landscape.

So your point boils down to emacs and lein don't work on Windows. I
don't know about emacs, but at least for leiningen there is some
activity at that front (cf. Rob's response).

So I still think: if you have trouble with clojure and Windows, you are
probably using the wrong tools.

Sincerely
Meikel

Mike K

unread,
Mar 7, 2010, 10:57:13 AM3/7/10
to Clojure
On Mar 4, 4:32 pm, Glen Stampoultzis <gst...@gmail.com> wrote:
>
> Getting swank-clojure going in Windows is also a pain.  To install it the
> recommended way via ELPA you need to patch the ELPA source code first.  This
> isn't documented at the swank-clojure site either.

For others who are struggling with this issue, see the following
thread.

http://groups.google.com/group/clojure/browse_frm/thread/c4d00ba0f1614c49/3b4e04ef9fb0c8bf?lnk=gst&q=elpa+windows#3b4e04ef9fb0c8bf

I ended the thread with a plea to Phil update the docs wrt this issue,
but no movement on that front yet.

Mike

Mike K

unread,
Mar 7, 2010, 11:27:16 AM3/7/10
to Clojure
Is anyone familiar with running Maven (and possibly Polygot Maven) on
Windows? Does it pretty much work as expected? The learning curve
for Maven looks steeper than for Leiningen, but if it makes up for it
with better documentation and being better-behaved, then it may be
worth it.

Mike

Mark Nutter

unread,
Mar 7, 2010, 1:14:17 PM3/7/10
to clo...@googlegroups.com
On Sun, Mar 7, 2010 at 10:40 AM, Meikel Brandmeyer <m...@kotka.de> wrote:

> So your point boils down to emacs and lein don't work on Windows. I
> don't know about emacs, but at least for leiningen there is some
> activity at that front (cf. Rob's response).
>
> So I still think: if you have trouble with clojure and Windows, you are
> probably using the wrong tools.

I wrestled with trying to set up emacs and clojure under Windows, but
quickly gave up on trying to do it all by hand and just downloaded
Clojure Box (http://clojure.bighugh.com/), which was about as painless
as I could hope for. I'm still getting my toes wet in Clojure, though,
and have only used it for trying out quick ideas and doing little
quickie tutorial type problems, so I can't say how well it would work
for more industrial-sized projects.

Mark

Phil Hagelberg

unread,
Mar 7, 2010, 6:34:45 PM3/7/10
to clo...@googlegroups.com

Sorry, I lost that message when I changed email clients recently. I've
applied the patch along with several others in a branch of my
package.el repository that is slated to be included in Emacs soon:

http://github.com/technomancy/package.el/tree/hax

Could you try the latest from this branch to ensure that it fixes the
issue on Windows?

thanks,
Phil

evins...@gmail.com

unread,
Mar 10, 2010, 12:35:14 PM3/10/10
to Clojure

On Mar 4, 7:33 am, Jan Rychter <j...@rychter.com> wrote:
> I haven't hacked on new Clojure stuff for the past two months or
> so. Now, having updated my repositories, I find that everybody just
> dropped ant and moved to leiningen.
>
> I tried to make sense of things, but can't. I must be missing something
> big here.

[...]

> How do people deal with this?

Here's how I do it.

We've got five people in different places with different systems
writing code for a dozen projects, of which five are Clojure projects.
We use a common git server for coordination.

Our standard project layout places all five Clojure projects as
siblings in a common directory, along with a directory called "xglib"
that contains common libraries.

We tried leiningen, and it's appealing in some ways, but it makes too
many assumptions about the way projects are organized, and its habit
of downloading scads of jars quickly became annoying. We have five
projects with partially-overlapping dependencies, some of which have
others as dependencies. It was tiresome always to have multiple
redundant copies of the same jars all over the place, so instead we
created the common xglib directory, which is a sibling of the
projects, and which contains all the libraries used by them.

We use NetBeans as our standard IDE, not because I like it (I don't
like any current-generation IDE) but because it works, and telling new
contributors how to set it up to work with our projects is easy. So
each project is, in fact, a NetBeans project.

We also use Emacs, because a couple of use hate it slightly less than
we hate NetBeans. We have a custom variant of swank-clojure-project
(just a couple of function definitions, really) that knows how to find
our common libraries directory and add all the jars in it to the
CLASSPATH before running the Clojure REPL.

It's also possible to use other editors (TextMate, for example) with
this setup. The standard way we do builds is with NetBeans, but we can
also do them with ant.

It's not great, but it works, and it stays out of our way better than
anything else we've tried so far. A couple of us are old Lisp hackers
with memories of things like SK8 and Leibniz and MCL and SPE and
Genera, so grumbling is ongoing, but we must adapt to the times we
live in.

Brent Millare

unread,
Mar 10, 2010, 11:07:17 PM3/10/10
to Clojure
Since leiningen downloads everything to a local repo, can't we do away
with copies and use symlinks if they are supported by the filesystem?
I feel there should be an option for this.

-Brent

On Mar 4, 1:59 pm, David Nolen <dnolen.li...@gmail.com> wrote:

> On Thu, Mar 4, 2010 at 10:19 AM, Stuart Sierra
> <the.stuart.sie...@gmail.com>wrote:
>
> > As many of you know, I've always been tepid on lein.  I'd rather go
> > with Maven whole-hog, because that offers the most robust model for
> > incorporating Java libraries.


>
> If Lein evolves to to handle dependencies of dependencies and intelligently
> generates the classpath based on these dependencies (instead of copying
> files around) what advantage does Maven really have?
>

> David

Meikel Brandmeyer

unread,
Mar 11, 2010, 1:24:51 AM3/11/10
to Clojure
Hi,

On Mar 11, 5:07 am, Brent Millare <brent.mill...@gmail.com> wrote:
> Since leiningen downloads everything to a local repo, can't we do away
> with copies and use symlinks if they are supported by the filesystem?
> I feel there should be an option for this.

Why not adding the files from the repo directly to the classpath?
Works everywhere and doesn't copy...

Sincerely
Meikel

Alex Osborne

unread,
Mar 11, 2010, 1:45:03 AM3/11/10
to clo...@googlegroups.com
Brent Millare <brent....@gmail.com> writes:

> Since leiningen downloads everything to a local repo, can't we do away
> with copies and use symlinks if they are supported by the filesystem?
> I feel there should be an option for this.

What benefit does this have aside from a tiny saving in disk space?

The way it currently works has a number of benefits:

* really simple and easy to understand!

* you can tarball your project directory, rsync it another machine or
whatever and everything you need will be right there. No need to
worry about redownloading anything or copying your local maven cache
around.

* you can run stuff manually just with:

java -cp 'src:classes:lib/*' clojure.main

* really simple and easy to understand! (worth repeating)

Doing tricks with symlinks or having lein set your classpath magically
like Maven does just seems to be adding complexity for the sake of
complexity.

Cheers,

Alex

Richard Newman

unread,
Mar 11, 2010, 1:53:43 AM3/11/10
to clo...@googlegroups.com
> What benefit does this have aside from a tiny saving in disk space?

Not that tiny when you multiply it across the dozens of projects on
your hard drive.

repos $ du -hc $(ls */lib/*.jar) | fgrep total
291M total

Add to that the size of the Maven repo itself.

Symlinks are nice.


> * you can tarball your project directory, rsync it another machine or
> whatever and everything you need will be right there. No need to
> worry about redownloading anything or copying your local maven cache
> around.

Unix is great.

man tar:

-h, --dereference
don't dump symlinks; dump the files they point to

Alex Osborne

unread,
Mar 11, 2010, 2:26:47 AM3/11/10
to clo...@googlegroups.com
Richard Newman <holy...@gmail.com> writes:

>> What benefit does this have aside from a tiny saving in disk space?
>
> Not that tiny when you multiply it across the dozens of projects on
> your hard drive.
>
> repos $ du -hc $(ls */lib/*.jar) | fgrep total
> 291M total

Cost (on standard disks): < 5 cents.

Sorry, that's tiny. It's even less than 0.5% of the small SSD I have in
my laptop.

Seriously, this is just premature optimization.

Richard Newman

unread,
Mar 11, 2010, 2:58:15 AM3/11/10
to clo...@googlegroups.com
>> repos $ du -hc $(ls */lib/*.jar) | fgrep total
>> 291M total
>
> Cost (on standard disks): < 5 cents.
>
> Sorry, that's tiny. It's even less than 0.5% of the small SSD I have
> in
> my laptop.
>
> Seriously, this is just premature optimization.

You're seriously fine with every single Leiningen-based project
spitting a redundant copy of Clojure, contrib, and whatever other
dependencies it needs onto disk, and doing so every time it picks up a
new snapshot? That every tiny library you pull off GitHub grabs
another few dozen MB, rather than a few dozen KB?

290MB is only tiny if you have an empty disk. 290MB here, 290MB there,
and suddenly the 13GB free on my 180GB laptop disk starts looking
mighty cramped. I have a lot of stuff I'd rather store on that disk
than 290MB of redundant jar copies. Those hundreds of megs get backed
up, scanned, indexed, defragmented, yadda yadda. What a waste.

You might also be overlooking disk caching, paging, JVM memory
sharing, and other performance effects from having multiple instances
of the same jars in memory. I know that the Mac JVM does some tricks
to share memory in this way.

Furthermore, it's slower to copy 28MB of jars than it is to symlink
them:

$ time cp -R lib /tmp/foo

real 0m2.981s
user 0m0.007s
sys 0m0.152s

... surprise disk slowdown! All the nonsense of looking up snapshots,
checking deps against the Maven repo, etc. already takes so much time
that I dislike using Leiningen. Anything to bring that down by a few
hundred milliseconds and a few fewer disk head slaps would be nice.

There is no engineering reason to blindly copy jars into lib/. Hard
links, symlinks, or computed classpaths are all better solutions in
every single way.

"Premature optimization" is the act of expending significant time
addressing a performance problem prior to measurement, typically
allowing unmeasured performance concerns to affect system design.
Using symlinks instead of copying would not be a significant amount of
work, doesn't affect the overall design, and we're hardly flying
blind. You're using the wrong phrase. Here's a different Knuth quote:

"In established engineering disciplines a 12 % improvement, easily
obtained, is never considered marginal and I believe the same
viewpoint should prevail in software engineering"

I'd call this a decent improvement, and it's certainly easily obtained.

Whatever happened to engineers building a decent solution out of a
sense of pride, or even making the most basic half-assed attempt to
conserve resources?

Sometimes I despair.

Alex Osborne

unread,
Mar 11, 2010, 3:50:29 AM3/11/10
to clo...@googlegroups.com
Richard Newman <holy...@gmail.com> writes:

> Whatever happened to engineers building a decent solution out of a
> sense of pride, or even making the most basic half-assed attempt to
> conserve resources?

I take your points but I think we may just have different priorities of
what to conserve, so we could probably argue about this all day. To me
disk space is cheap and simplicity precious. I'd rather waste 300mb of
space then have to muck around sorting out link problems when I run the
code on a different server and suddenly all the symlinks are broken
because I forgot an obscure flag on my copy/tar/rsync/whatever.

Hard links are better, there's much less that could go wrong (but
obviously they have the limitation that they don't work across different
filesystems).

The implementation is also not so trivial as you make out. How does one
even create a symlink (or hardlink) on the JVM (prior to the unreleased
JDK7)? There's no way I know of to do it via the standard library.
You'd have to resort to JNI or shelling out to "ln". What happens if
you're on a platform or even just a filesystem that doesn't support
links? Do you start littering the code with per operating system
conditionals? If so, what happens if you're technically running on Linux
but with a different userspace (perhaps Android or something) and there
happens to be no "ln" binary? Another special case? Very quickly we're
spiralling down into a mess of over-engineering.

Per Vognsen

unread,
Mar 10, 2010, 11:19:05 PM3/10/10
to clo...@googlegroups.com
The way to go is definitely symlinks or hard links, not classpath
hijinks. The only problem is that Java's filesystem API has no support
for links, so you'll probably need to use OS-specific command line
calls. Not too bad though.

-Per

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

Per Vognsen

unread,
Mar 11, 2010, 1:54:08 AM3/11/10
to clo...@googlegroups.com
Instead of symlinks you can use hard links like every decent DVCS does
for clone-style branches. Hard links are supported on NTFS, HFS, and
all the Unix file systems on wide use. They are transparent to users
of the file system, so you can easily make tarballs of hardlinked
jars, etc, without any problems.

-Per

Per Vognsen

unread,
Mar 11, 2010, 2:37:36 AM3/11/10
to clo...@googlegroups.com
If you use a DVCS like Mercury or Bazaar that mainly uses clone-style
branching then it can become significant. The jars are build artifacts
and so ignored by the repository and not subject to the DCVS's
hard-link-based sharing. Penumbra is a small project and it already
has 12 megabytes of jars in its lib/ directory. A bigger project can
easily have well over a hundred megabytes of lib/ jars. Now multiply
that by the number of active clone branches (let's say a few dozen)
and you're looking at a few gigabytes of waste.

-Per

Laurent PETIT

unread,
Mar 11, 2010, 7:20:18 AM3/11/10
to clo...@googlegroups.com
Hello,

2010/3/11 Per Vognsen <per.v...@gmail.com>:


> The way to go is definitely symlinks or hard links, not classpath
> hijinks.

Why so ?
If you use 'mvn repl' or 'lein repl' to start your repl, why not just
let mvn / lein manage the dependencies from beginning to end ?

Phil Hagelberg

unread,
Mar 11, 2010, 4:32:36 PM3/11/10
to clo...@googlegroups.com
On Wed, Mar 10, 2010 at 10:24 PM, Meikel Brandmeyer wrote: > On Mar
11, 5:07 am, Brent Millare wrote:

>> Since leiningen downloads everything to a local repo, can't we do away
>> with copies and use symlinks if they are supported by the filesystem?
>> I feel there should be an option for this.
>
> Why not adding the files from the repo directly to the classpath?
> Works everywhere and doesn't copy...

Historically this has been because calculating the classpath couldn't
be done in Clojure itself since it needed to be done before the JVM
booted. Having to figure this kind of thing out in a shell script or
in elisp is a headache if the rules are not simple. But now with the
subclassloader approach that lein and mvn use it's less of an issue.
But it would mean deprecating the pure-elisp swank launcher, and I
don't know how it would affect other tools (IDEs, etc), so I'm still
hesitant.

I would be happy with a (simple-ish) patch to enable optionally
switching to using symlinks or hard-links as this would be the
lowest-impact fix, but I can't bring myself to care enough about the
number of megabytes currently being wasted to code it myself; sorry.

-Phil

Cyrus Harmon

unread,
Mar 12, 2010, 1:39:10 PM3/12/10
to clo...@googlegroups.com
I'm confused. Why do we need symlinks or copies at all? Why can't we just tell clojure where it's supposed to find a given projects dependencies? I'm sure the answer involves some mumbo-jumbo about classpath and what not, but surely there has to be a better alternative than whatever maven/leiningen are currently doing.

Granted, I'm coming at this from a common lisp background, but I think many lisp developers are used to not just working on a single project, but working on a bunch of related projects simultaneously, including the the language/runtime environment itself. I'd like a single instance of clojure, a single clojure-contrib (I still don't understand why I had to (admittedly seamlessly) download a bunch of jars to clean the build directory), a single incanter instance, etc... Perhaps I'm missing the obvious here, but I continue to fail to understand why the clojure world is doing things this way.

On a related note, why is the build system the cool kids seem to be using (leiningen) controlled by a bunch of shell scripts? Surely things like compiling .clj source files and making jars should be operations one can drive from the clojure repl, no?

Oh, and on a related note, I hate being forced into the src, test, lib heirarchy... Let me put my files where I want them to go. Whatever happened to the lancet build system from Stuart Holloway's book? That seemed to make sense to me.

thanks,

Cyrus

Stuart Halloway

unread,
Mar 12, 2010, 2:17:38 PM3/12/10
to clo...@googlegroups.com
> Oh, and on a related note, I hate being forced into the src, test,
> lib heirarchy... Let me put my files where I want them to go.
> Whatever happened to the lancet build system from Stuart Holloway's
> book? That seemed to make sense to me.

I think there is a big benefit in consistent directory structure
across projects, but agree that it should still be *possible* to
configure things a different way.

I am not actively developing Lancet. Anyone who wants is certainly
welcome to fork it and continue. It is used under the covers in
Leiningen IIRC.

Stu

Richard Newman

unread,
Mar 12, 2010, 3:22:06 PM3/12/10
to clo...@googlegroups.com
> I'm confused. Why do we need symlinks or copies at all? Why can't we
> just tell clojure where it's supposed to find a given projects
> dependencies? I'm sure the answer involves some mumbo-jumbo about
> classpath and what not, but surely there has to be a better
> alternative than whatever maven/leiningen are currently doing.

Not if we want to precompile code and create an überjar, or otherwise
integrate with Java. Ultimately, tools expect a pile of jars listed in
the classpath. Management of those jars is easier if they're (actually
or virtually) in one place.

I'm from a CL background too, and I do long for the days of the
continuum from a load script through to ASDF and friends, but at some
point it makes sense to give up and do things The Java Way.

Even simple things like log4j.properties are looked up on… you guessed
it: the classpath. It's not a battle that's worth fighting.


> On a related note, why is the build system the cool kids seem to be
> using (leiningen) controlled by a bunch of shell scripts? Surely
> things like compiling .clj source files and making jars should be
> operations one can drive from the clojure repl, no?

It's a shell script wrapping a bunch of Clojure code. I'm sure you
could use Leiningen as a library: each task is simply a function which
calls out to Maven or Lancet.


> Oh, and on a related note, I hate being forced into the src, test,
> lib heirarchy... Let me put my files where I want them to go.
> Whatever happened to the lancet build system from Stuart Holloway's
> book? That seemed to make sense to me.

That's convention over configuration (though you can put alternatives
in the Leiningen project.clj file).

It's one of those things where it's easier to go with the flow, I'm
afraid.

Meikel Brandmeyer

unread,
Mar 13, 2010, 5:14:33 PM3/13/10
to clo...@googlegroups.com
Hi,

On Thu, Mar 11, 2010 at 01:32:36PM -0800, Phil Hagelberg wrote:

> Historically this has been because calculating the classpath couldn't
> be done in Clojure itself since it needed to be done before the JVM
> booted. Having to figure this kind of thing out in a shell script or
> in elisp is a headache if the rules are not simple. But now with the
> subclassloader approach that lein and mvn use it's less of an issue.
> But it would mean deprecating the pure-elisp swank launcher, and I
> don't know how it would affect other tools (IDEs, etc), so I'm still
> hesitant.

I think this is a typical emacs misconception. Vim is an editor. Why
should it have to know how a JVM classpath works? Setting this up is the
responsibility of the developer. Put everything in lib/? No problem.
Let maven/gradle/lein handle the classpath? No Problem. Have an esoteric
shell script doing some voodoo incatations? No Problem.

Putting this logic into the editor was the failure in the first place.

Sincerely
Meikel

Brent Millare

unread,
Mar 29, 2010, 12:07:07 PM3/29/10
to Clojure
Can someone please elaborate on this subclassloader process? Although
the design for leiningen is architecturally simple and understandable
and thus easy to extend, I'm still quite unfamiliar with the "java
way". I believe a little documentation on this aspect can make a
significant improvement in allowing contributions to leiningen.

Also, I noticed that symlink support was added to the master branch. I
read the documentation for maven's ant symlink support and didn't see
any mention of hardlinking. Does this imply that hardlinking support
would need to be implemented separately? I feel hardlinking is a more
elegant solution as ~/.m2/repository can act as a true cache with size
bounds but without the fear that projects that depend on deleted
artifacts would be affected.

Alex Osborne

unread,
Mar 29, 2010, 7:42:48 PM3/29/10
to clo...@googlegroups.com
Brent Millare <brent....@gmail.com> writes:

> Can someone please elaborate on this subclassloader process? Although
> the design for leiningen is architecturally simple and understandable
> and thus easy to extend, I'm still quite unfamiliar with the "java
> way". I believe a little documentation on this aspect can make a
> significant improvement in allowing contributions to leiningen.

Given I implemented the initial support for it, I'll give it a go. In
earlier versions of Leiningen you weren't able to compile code against a
different version of Clojure to the one that Lein was running on top
of, as you could only have one or the other in the classpath. One way to
get around this is to fork off a new java process and run the compile in
a separate JVM. Unfortunately we'd then be incurring the JVM's startup
overhead twice (and lein is already annoyingly slow to start). Wouldn't
it be nice if we could just run the compile in the original JVM but
isolated in a container somehow?

Well we can actually do this because the JVM supports custom
classloaders. A classloader is an object that when given a classname,
locates the appropriate class files (in jars, classpath etc) and loads it
into memory. There's a default builtin classloader which searches the
classpath you supply on the command-line. Custom classloaders can be
used for a number of things, for example loading jars over the network
or out of a new archive format. They also play a strong role in the
JVM's security module (for sandboxing etc). When a class is loaded
(manually) into a custom classloader, any classes it in turn
tries to load (by normal means) are also loaded via that same
classloader.

So basically what Lein does is creates a new classloader (it actually
reuses Apache Ant's implementation) with a different classpath which
points to the version of clojure and other libraries that you want to
compile against.

Take a look at the eval-in-project function in leiningen/compile.clj.
You pass in a form to be evaluated as an argument. It's converted to a
string and then clojure.main is invoked with the '-e' argument to
evaluate the string 'inside' the new classloader. The reason its done
this way is so that its easy to fall back to forking and also because
objects in the new classloader aren't compatible with those in the
original (as they're potentially different versions of Clojure).

You can see that it doesn't work perfectly for all cases, so for example
if you're using native libs or OS X it'll fork instead. (OS X injects
some GUI stuff into the classpath of all the apps it runs, so anything
using using Swing or AWT doesn't work properly in the new classloader).

You may have wondered how Java servlet containers like Jetty and Tomcat
are able to load different versions of the same class into the same JVM
without them conflicting. As I understand it they use basically the
same mechanism.

Hope that helps. If you're having trouble following the classloader
stuff just think of the eval-in-project function as starting a new
"virtual" JVM with a different classpath and evaling some code in it.
Its really not all that complex, its just the terminology makes it sound
somewhat arcane.

Alex

Brent Millare

unread,
Mar 30, 2010, 2:24:51 PM3/30/10
to Clojure
Thanks, that was insightful. To recap, subclassloader is a
optimization to reduce the startup time of launching a new clojure
instance from leiningen. In addition, I learned that computing the
classpath was conducted in clojure code. I see now that the bash
script is just a bootstrapping tool to get an instance of clojure
running. This means that we have the full power of clojure at our
disposal to compute the desired classpath. Many people make the more
complex compute classpath feature request but may not realize the
reasonable tractability of writing such a plugin.

On another note, I'm currently concerned with utilizing a global
configuration of leiningen (e.g. installing plugins and define
additional repositories globally). My current idea is to create a git
fork/branch to manage personal configuration files within the checkout
folder and then implement a bootstrapping script to facilitate
automating the deployment of alternative branches of leiningen.

Brent

Reply all
Reply to author
Forward
0 new messages