I tried to make sense of things, but can't. I must be missing something
big here.
It seems that leiningen assumes that you are building an application. It
downloads all dependencies and can build an uberjar for you. That's
great, but what if what I have is a bunch of libraries, all being
developed and modified?
Also, I now have at least six clojure jars in ~/.m2 (huh?!), along with
a collection of other assorted stuff. I don't want to use any of these
clojure jars, I have a checked out git repo with clojure and this is the
only code I want to run. Also, swank-clojure insists on having its own
clojure jar in ~/.swank-clojure (why?).
I used to have a bunch of directories with git repos. I would update
those, run ant to gather code into jars, fiddle with the classpath in my
custom-crafted clj script, symlink things around. It was about as bad as
with Common Lisp. But now things got worse -- I no longer feel in
control of anything. Some things won't build, lein tells me about
"artifacts" (huh?!), things get installed in weird places. My clj script
that gets run by SLIME form Emacs can't find libraries that leiningen
downloaded.
How do people deal with this?
As a more general observation, I think that a large part of Perl's and
Python's success was a unified way of dealing with libraries. There
are certain directories where you can drop libraries and expect them to
work (be found). There is ONE way of running the VM (one script, blessed
by the language creator). Then on top of that there is CPAN, which plugs
into that, downloading dependencies and dropping libraries into those
well-known directories. It isn't perfect, but it works, and is
predictable.
Shouldn't we have a well-known Single Setup for Clojure as well?
I am very worried that Clojure is rapidly going the Common Lisp way,
with each developer having his own precious carefully crafted
setup. Every system is different, everyone uses a different script to
run clojure programs, there is no single place to drop your libraries
into, and libraries have no conventions on where their code is (what do
I add to classpath once I have a library checked out?). I thought
Leiningen was supposed to solve these issues, but either I am missing
something, or it is really a tool to manage dependencies for a single
application.
--J.
My 2c: Leiningen is an important step, but there is still plenty to do.
My workflow: I am developing multiple Clojure libraries and
applications, with some of the dependencies changing daily. Whenever I
want to make a change, I:
In the project being changed:
git tag a new version number
update the project.clj to a new version number
run "lein pom" and "lein jar"
copy the new files to a repos (clojars)
In projects that depend on changed project
update the dependency in project.clj (if it is hard coded and not >=)
run lein deps
This seems adequate to me (although there could be a bit more
automation). The problem of snapshot dependencies remains. The
solution that springs to mind here is to get projects off snapshots
(presumably by releasing more incremental versions). Library authors:
please release lots of versions! (or tell me why this is a bad idea).
Some needs I see:
(0) More point releases of key libs (or some other solution to the
snapshots problem)
(1) Unify around the project structure imposed by lein, and/or
(2) Improve the project structure imposed by lein, if it is broken in
some specific way. (Maybe add a well-known config file for adding
elements to the classpath?)
(3) "lein deps" (or some other task) to clean up old jars. Right now I
am deleting them by hand from the lib directory. (Is this already
solved?)
Stu
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient
> with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
"lein deps" copies dependencies into the project's lib directory.
Various other lein commands assume lib/*.jar is on the classpath, and
if you are writing scripts you should assume that classpath as well.
Is this good enough?
Stu
Oh yeah.
> Some needs I see:
>
> (0) More point releases of key libs (or some other solution to the
> snapshots problem)
Yes!
> (1) Unify around the project structure imposed by lein, and/or
As many of you know, I've always been tepid on lein. I'd rather go
with Maven whole-hog, because that offers the most robust model for
incorporating Java libraries.
The more we can leverage the Java tool set, the better. I'd rather
lein were just a thin POM generator for those who can't stand writing
XML.
-SS
> It seems that leiningen assumes that you are building an application. It
> downloads all dependencies and can build an uberjar for you. That's
> great, but what if what I have is a bunch of libraries, all being
> developed and modified?
That's my situation as well. I did start to use Leiningen, but only for producing releases for the outside world. On my development machine, it's the source code directories of all libraries that are on my classpath. That way I am sure which version of the code I am running.
> with Common Lisp. But now things got worse -- I no longer feel in
> control of anything. Some things won't build, lein tells me about
> "artifacts" (huh?!), things get installed in weird places. My clj script
> that gets run by SLIME form Emacs can't find libraries that leiningen
> downloaded.
>
> How do people deal with this?
Here's what works for me.
Each of my libraries is one "project" in the sense of swank-clojure. I use swank-clojure-project to start Clojure inside slime, and that takes care of my classpath. All I need to do is have the right stuff in the lib subdirectory of each project, so that's where are all the configuration is done.
Instead of doing something like "lein deps" to fill the lib subdirectories, I decided to manage them manually and have only soft links in there that point to the real jars and source directories. There's one soft link to the clojure jar (the one I rebuild after each "git pull" from github), one for clojure-contrib (same principle), and one for swank-clojure (same again). All other dependencies on Clojure libraries are soft links to the source code directory inside the respective project directories. Dependencies on external Java libs are soft links to the jar files, which reside in a single place on my hard disk.
> As a more general observation, I think that a large part of Perl's and
> Python's success was a unified way of dealing with libraries. There
> are certain directories where you can drop libraries and expect them to
> work (be found). There is ONE way of running the VM (one script, blessed
> by the language creator). Then on top of that there is CPAN, which plugs
> into that, downloading dependencies and dropping libraries into those
> well-known directories. It isn't perfect, but it works, and is
> predictable.
Right, but its imperfections have bothered me often enough. For once, if you have different applications that require different versions of the same library, you are in for a major headache. Python setuptools and its eggs are a partial solution, but add a lot of complexity of their own. Another source of endless frustration is maintaining a personal library installation on a machine where I don't have administrator rights and need to work with multiple Python versions. It's possible, but it's a big mess and I need to be very careful to be sure that I really use the version that I need.
I am not too unhappy with the JVM approach to configurability via classpath, but it requires a layer of management tools and/or conventions to be productive. What doesn't fit with my JVM world-view is Maven, and thus leiningen. Their dependency handling model is clearly geared towards application deployment rather than library development. Moreover, I think their model is too complicated, making it difficult to predict which jars will end up being used where. But perhaps that's just my lack of experience.
> Shouldn't we have a well-known Single Setup for Clojure as well?
It would perhaps be a good idea to have one default setup, with the option for expert users to roll their own. For newcomers the current situation is definitely too messy.
> I add to classpath once I have a library checked out?). I thought
> Leiningen was supposed to solve these issues, but either I am missing
> something, or it is really a tool to manage dependencies for a single
> application.
That's my view of what it is, but I can't claim much experience with it.
Konrad.
I have also seen a weird issue in my code base which has two classes
derived from java.lang.Exception. If I do a clean and try compile, I
get an error saying that my.package.MyException class not found. My
workaround is, set :namespaces to only those Exception derived
classes, compile and then set :namespaces to :all.
Maybe I am missing some cool lein thing here or lein needs quite a bit
of work in dependency handling. I am tempted to just stick to mvn
which is quite mature and well-documented. Any clojure-specific needs
could be addressed through a plug-in.
Comments, thoughts?
Thanks,
Praki
On Thu, Mar 04, 2010 at 07:11:59AM -0800, Praki Prakash wrote:
> I have also seen a weird issue in my code base which has two classes
> derived from java.lang.Exception. If I do a clean and try compile, I
> get an error saying that my.package.MyException class not found. My
> workaround is, set :namespaces to only those Exception derived
> classes, compile and then set :namespaces to :all.
You have to :require the namespaces implementing the exceptions in the
files were you use the Exceptions. This will resolve the dependency
problem. See here: http://tr.im/FNmt
Sincerely
Meikel
Out of interest, what scenarios does Maven account for that Leiningen
currently does not?
> The more we can leverage the Java tool set, the better. I'd rather
> lein were just a thin POM generator for those who can't stand writing
> XML.
I personally favour the opposite approach :). Whilst Java provides a
lot of useful functionality, I've never been particularly impressed
with the way most Java libraries are designed.
- James
Right, but its imperfections have bothered me often enough. For once, if you have different applications that require different versions of the same library, you are in for a major headache. Python setuptools and its eggs are a partial solution, but add a lot of complexity of their own. Another source of endless frustration is maintaining a personal library installation on a machine where I don't have administrator rights and need to work with multiple Python versions. It's possible, but it's a big mess and I need to be very careful to be sure that I really use the version that I need.
Also, I now have at least six clojure jars in ~/.m2 (huh?!), along with
a collection of other assorted stuff. I don't want to use any of these
clojure jars, I have a checked out git repo with clojure and this is the
only code I want to run. Also, swank-clojure insists on having its own
clojure jar in ~/.swank-clojure (why?).
I used to have a bunch of directories with git repos. I would update
those, run ant to gather code into jars, fiddle with the classpath in my
custom-crafted clj script, symlink things around. It was about as bad as
with Common Lisp. But now things got worse -- I no longer feel in
control of anything. Some things won't build, lein tells me about
"artifacts" (huh?!), things get installed in weird places. My clj script
that gets run by SLIME form Emacs can't find libraries that leiningen
downloaded.
As a more general observation, I think that a large part of Perl's and
Python's success was a unified way of dealing with libraries. There
are certain directories where you can drop libraries and expect them to
work (be found). There is ONE way of running the VM (one script, blessed
by the language creator). Then on top of that there is CPAN, which plugs
into that, downloading dependencies and dropping libraries into those
well-known directories. It isn't perfect, but it works, and is
predictable.
As many of you know, I've always been tepid on lein. I'd rather go
with Maven whole-hog, because that offers the most robust model for
incorporating Java libraries.
> Imperfections meaning it's perfectly broken :) That's why tools like
> virtualenv exist for Python.
I'd say it's "New Jersey style": good enough for 90% of the users, but
indeed fundamentally broken.
Konrad.
> How do people deal with this?
I don't have any good answers for you; I just want to say "I feel your
pain".
The situation is worse on Windows, where I can't even get leiningen to
work at all. Luckily I'm only working on one clojure project, so I
just copy clojure.jar and clojure-contrib.jar into my project's lib
directory, as well as a faked version of swank-clojure.jar (just a
zipfile of swank source, renamed to ".jar"). That works well enough
for me. Then I have a tiny little Rakefile that builds a classpath
string from the contents of lib/, and has two tasks: run tests, and
start a repl (which I rarely use anymore, since I got swank-clojure
working).
> As a more general observation, I think that a large part of Perl's and
> Python's success was a unified way of dealing with libraries. There
> are certain directories where you can drop libraries and expect them to
> work (be found). There is ONE way of running the VM (one script, blessed
> by the language creator). Then on top of that there is CPAN, which plugs
> into that, downloading dependencies and dropping libraries into those
> well-known directories. It isn't perfect, but it works, and is
> predictable.
Actually, I have noticed a move towards using tools like virtualenv
for Python projects, precisely because depending on system-wide
libraries makes it too non-obvious exactly what your external
dependencies are, as well as creating potential version conficts.
-- Chris
The situation is worse on Windows, where I can't even get leiningen to
work at all. Luckily I'm only working on one clojure project, so I
just copy clojure.jar and clojure-contrib.jar into my project's lib
directory, as well as a faked version of swank-clojure.jar (just a
zipfile of swank source, renamed to ".jar"). That works well enough
for me. Then I have a tiny little Rakefile that builds a classpath
string from the contents of lib/, and has two tasks: run tests, and
start a repl (which I rarely use anymore, since I got swank-clojure
working).
The tools (e.g. Archiva, Nexus et al), documentation, IDE support, integration with other languages e.g. mixed Java/Clojure/Scala/Groovy projects. The more clojure integrates with the existing ecosystem, the more likely it is to succeed, especially in a viral sense. This is the essence of clojure being a JVM language with great integration with Java.
Personally I wish that leiningen effort was instead put towards polyglot maven http://polyglot.sonatype.org/
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Nothing is really work unless you would rather be doing something else.
-- J. M. Barre
>
> If Lein evolves to to handle dependencies of dependencies and intelligently generates the classpath based on these dependencies (instead of copying files around) what advantage does Maven really have?
The tools (e.g. Archiva, Nexus et al), documentation, IDE support, integration with other languages e.g. mixed Java/Clojure/Scala/Groovy projects. The more clojure integrates with the existing ecosystem, the more likely it is to succeed, especially in a viral sense. This is the essence of clojure being a JVM language with great integration with Java.
Felix
On 5 Mrz., 00:32, Glen Stampoultzis <gst...@gmail.com> wrote:
On Mar 5, 1:03 am, Felix Breuer <fe...@fbreuer.de> wrote:
> I agree that Windows is a second class citizen as far as clojure
> tools go.
Oh please stop that. I have a stable setup of Gradle + Clojuresque +
VimClojure on Windows. Granted setting up VimClojure on Windows is
tricky due to Vim and Windows specifics, but it's doable. However the
Gradle + Clojuresque part is as easy and trivial as it can get. And
this setup survived already a few updates of the different components
without blowing up a single time. I type "gradle runServer" in the
morning and get a running Jetty/Nailgun combination for dynamic
development with VimClojure - without thinking a single time about the
classpath. Which actually consists of several checked out other
projects (enlive, ring, fleetdb, ...). They were modified non-
intrusively to use clojuresque instead of maven/ant/lein/whatever as
build system. It checks the interdependencies and rebuilds the
required jars if necessary. A change in a dependency is immediately
picked up. If necessary, I can fix a known-to-work version in my local
repository with a few commands.
All without virtualbox, VMWare or whatever. So if one tool doesn't
fill your need, then choose another.
I apologise for sounding a little harsh.
Sincerely
Meikel
One huge drawback I've found with clojure (which it doubtless
inherited from Java) is that you need an actual jarfile in your
classpath, not just the directory containing the jarfile.
martin
On Mar 5, 7:47 am, Martin DeMello <martindeme...@gmail.com> wrote:
> One huge drawback I've found with clojure (which it doubtless
> inherited from Java) is that you need an actual jarfile in your
> classpath, not just the directory containing the jarfile.
With newer Java versions you can specify wildcards. java -cp lib/
\* ... will pick up everything in lib for the classpath.
Sincerely
Meikel
On 5 Mar, 01:03, Felix Breuer <fe...@fbreuer.de> wrote:
> I am also on Windows and have struggled a good deal with Leiningen (I
[...]
> On 5 Mrz., 00:32, Glen Stampoultzis <gst...@gmail.com> wrote:
>
>
>
> > On 5 March 2010 04:39, Chris Perkins <chrisperkin...@gmail.com> wrote:
>
> > > The situation is worse on Windows, where I can't even get leiningen to
> > > work at all. Luckily I'm only working on one clojure project, so I
[...]
> > Windows seems to be a second class citizen as far as clojure tools go. I
> > wasn't able to figure out how to get it going on Windows either which makes
> > things much harder considering so many clojure projects are now using it.
[...]
First of all have you ever tried this script?
http://github.com/technomancy/leiningen/blob/master/bin/lein.bat
Secondly have you seen this thread?
http://groups.google.com/group/leiningen/browse_thread/thread/2474d2e9019ee29c
There was almost no feedback at all.
Have you reported any issue with Leiningen on Windows?
http://github.com/technomancy/leiningen/issues
Leiningen is not perfect and I'd like to improve this tool (including
Windows version),
but so far I can see only some general complaining. Please report
concrete
issues and we will be able to make concrete improvements.
Br,
Rob
Neat! Never knew that. So now it *is* possible to have a standard
clojure lib directory the way ruby has its gems directory or python
its site-libs.
martin
Furthermore, all java tools play well on windows, so using ant, maven,
will work just as well on windows and on linux, mac, etc.
And you also have the Eclipse plugin for Clojure, the netbeans plugin
for clojure, the IntelliJ IDEA plugin for clojure, which work on
windows too !
> Granted setting up VimClojure on Windows is
> tricky due to Vim and Windows specifics, but it's doable. However the
> Gradle + Clojuresque part is as easy and trivial as it can get. And
> this setup survived already a few updates of the different components
> without blowing up a single time. I type "gradle runServer" in the
> morning and get a running Jetty/Nailgun combination for dynamic
> development with VimClojure - without thinking a single time about the
> classpath. Which actually consists of several checked out other
> projects (enlive, ring, fleetdb, ...). They were modified non-
> intrusively to use clojuresque instead of maven/ant/lein/whatever as
> build system. It checks the interdependencies and rebuilds the
> required jars if necessary. A change in a dependency is immediately
> picked up. If necessary, I can fix a known-to-work version in my local
> repository with a few commands.
>
> All without virtualbox, VMWare or whatever. So if one tool doesn't
> fill your need, then choose another.
>
> I apologise for sounding a little harsh.
>
> Sincerely
> Meikel
>
Maven supports a wide range of development/deployment models already.
Whatever you want to do (including compile/test/run Clojure code)
there's a plugin for that. Why recreate all this functionality in
Leiningen?
Yes, the Maven configuration model isn't pretty, but if you follow the
conventions it works. Polyglot Maven will obviate the need to write
XML. But frankly, I don't mind a little XML if it gets the job done.
Basically, I'm skeptical that any tool developed exclusively by/for
the Clojure community could ever achieve the scope and utility of a
tool used by thousands of people and companies. We're a small
community, we can't do it all alone.
-SS
I didn't know about Polyglot. Thanks. Looks like a very nice tool.
<http://polyglot.sonatype.org/clojure.html>
--
Ramakrishnan
Most of the time, I won't need the wide range of functionality that
Maven supports. For the most common tasks, such as compiling and
packaging, Leiningen is the better build tool. It's conceptually
simpler, more understandable, and easier to extend. I'd rather use a
build tool that covers 80% of tasks well, than a tool that covers 95%
poorly.
- James
Agreed.
> (1) Unify around the project structure imposed by lein, and/or
Yes, but it should be noted this isn't really the "leiningen" project
structure; it has been the conventional Clojure layout in the
community from long before leiningen was released.
> (2) Improve the project structure imposed by lein, if it is broken in some
> specific way. (Maybe add a well-known config file for adding elements to the
> classpath?)
People have asked for this, but I have yet to hear a good use case for
it. Mostly it's been requested by people who don't realize how easy it
is to push their own jars to clojars. But if someone wants to
implement it I will not refuse the patch.
> (3) "lein deps" (or some other task) to clean up old jars. Right now I am
> deleting them by hand from the lib directory. (Is this already solved?)
Should be able to "lein clean". =)
-Phil
I agree that lein+clojars is good enough for lib dependencies. My two
use cases so far are places where the classpath needs to point to
something other than outside libs:
(1) getting ./config on the classpath so I can keep properties files,
json config, whatever, separate from code and lib
(2) having more than one kind of tests, in different dirs (e.g. specs,
features) and having their classpaths work through lein.
Stu
The turning point was when I realized that the key to using Maven is
understanding the Maven model - understanding how Maven wants to
build. You don't tell Maven what to do. You just put the files in the
places Maven expects, and Maven then _knows_ what to do. My rule of
thumb is: if I'm spending a lot of time tweaking configuration and
overriding defaults in the pom file then I'm probably doing it wrong.
When I started programming in Clojure I immediately looked for a Maven
plugin to build Clojure projects. The first project I worked on I did
using the plugin. For my next Clojure project I decided to give
Leiningen a spin. Initially I was very impressed. It was easy to get a
project up and running, easy to build, easy to run tests, easy to run
a swank server etc. For a couple of weeks I was very happy. But.....
I finally hit a point where Leiningen was no longer working for me. My
current project has a Swing GUI, built using Matisse (the Netbeans GUI
builder). I couldn't find a way to build the Swing portion and the
Clojure portion together using Leiningen. I also couldn't find a way
to divide my project up into different sub-modules with one build
(maybe I'm guilty of trying to use Leiningen like I use Maven). In any
case, I ended up converting the project to use Maven.
Maven has a steep learning curve, but once you get it, it's incredibly
powerful.
"The philosophy of Tai Chi Chuan is that if one uses hardness to
resist violent force, then both sides are certain to be injured at
least to some degree. Such injury, according to tai chi theory, is a
natural consequence of meeting brute force with brute force. Instead,
students are taught not to directly fight or resist an incoming force,
but to meet it in softness and follow its motion while remaining in
physical contact until the incoming force of attack exhausts itself or
can be safely redirected, meeting yang with yin." (http://
en.wikipedia.org/wiki/Tai_chi_chuan)
If you fight with Maven, you will lose. If you meet it in softness and
follow its motion you will attain build enlightenment...
;-)
FInally, I agree with Stuart that polyglot Maven could / should be the
way forward. I don't really wrestle with pom files much manually
anyway since Netbeans does a very good job of automatically updating
them. But Leiningen has given me a taste of writing build files in
Clojure, so I will definitely be investigating the polyglot option
soon.
To learn more about Maven I would recommend
http://www.sonatype.com/books/mvnref-book/reference/public-book.html
and http://www.sonatype.com/books/mvnex-book/reference/public-book.html
As I'm one of the people that's requested this feature... I'll add my
reason too:
(3) Managing proprietary libraries and API's that you are developing
or using... Yes you can throw these into your own maven repo, but
then you also need to manage, configure and secure a server (when
working in a team).
R.
As I'm one of the people that's requested this feature... I'll add my
reason too:
(3) Managing proprietary libraries and API's that you are developing
or using... Yes you can throw these into your own maven repo, but
then you also need to manage, configure and secure a server (when
working in a team).
This is actually what the resources/ directory is for.
> (2) having more than one kind of tests, in different dirs (e.g. specs,
> features) and having their classpaths work through lein.
It would be best if this could be handled by plugins rather than
specified in every project.clj. Currently the plugin framework doesn't
expose the hooks that would be needed for this, but it's definitely on
the short list for version 1.2.0.
> (3) Managing proprietary libraries and API's that you are developing
> or using... Yes you can throw these into your own maven repo, but
> then you also need to manage, configure and secure a server (when
> working in a team).
I think it would be great if it were easy to run your own clojars
server. Not only for mirroring purposes (poor ato isn't going to be
able to eat the bandwidth bills forever) but also to deal with
situations like this.
But there may be other valid reasons that nobody has even encountered
yet, so I favour having total flexibility anyway.
-Phil
If you're using git, this should be avoided. git is remarkably bad at
storing binary data. A number of projects I've been watching have made
this mistake, and they've all had to correct it by rewriting history
down the line, which is very awkward.
-Phil
I seem to recall that recent versions of Git have improved
significantly in this area. However, there's still the fundamental
problem that changing the dependencies increases the size of the
project. New developers will not only have to download your source
files, but the binary diff of every dependency change.
So yep, it's a bad idea :)
- James
> If you're using git, this should be avoided. git is remarkably bad atI seem to recall that recent versions of Git have improved
> storing binary data.
significantly in this area. However, there's still the fundamental
problem that changing the dependencies increases the size of the
project. New developers will not only have to download your source
files, but the binary diff of every dependency change.
So yep, it's a bad idea :)
On Mon, Mar 08, 2010 at 01:44:44AM +1100, Glen Stampoultzis wrote:
> It seems (from my limited observations) that most of the Clojure developers
> are Linux/Mac people so it's understandable that Windows isn't currently as
> well supported by all tools.
Windows has a fairly big share, I think. There are a lot of posts (blog,
group, pm, IRC, ...) where the classpath starts with C:\Clojure.
My setup was just an example. I gave it because I use it actively, so I
could also answer specific questions. Laurent gave a broader view on the
tool landscape.
So your point boils down to emacs and lein don't work on Windows. I
don't know about emacs, but at least for leiningen there is some
activity at that front (cf. Rob's response).
So I still think: if you have trouble with clojure and Windows, you are
probably using the wrong tools.
Sincerely
Meikel
For others who are struggling with this issue, see the following
thread.
I ended the thread with a plea to Phil update the docs wrt this issue,
but no movement on that front yet.
Mike
Mike
> So your point boils down to emacs and lein don't work on Windows. I
> don't know about emacs, but at least for leiningen there is some
> activity at that front (cf. Rob's response).
>
> So I still think: if you have trouble with clojure and Windows, you are
> probably using the wrong tools.
I wrestled with trying to set up emacs and clojure under Windows, but
quickly gave up on trying to do it all by hand and just downloaded
Clojure Box (http://clojure.bighugh.com/), which was about as painless
as I could hope for. I'm still getting my toes wet in Clojure, though,
and have only used it for trying out quick ideas and doing little
quickie tutorial type problems, so I can't say how well it would work
for more industrial-sized projects.
Mark
Sorry, I lost that message when I changed email clients recently. I've
applied the patch along with several others in a branch of my
package.el repository that is slated to be included in Emacs soon:
http://github.com/technomancy/package.el/tree/hax
Could you try the latest from this branch to ensure that it fixes the
issue on Windows?
thanks,
Phil
On Mar 4, 7:33 am, Jan Rychter <j...@rychter.com> wrote:
> I haven't hacked on new Clojure stuff for the past two months or
> so. Now, having updated my repositories, I find that everybody just
> dropped ant and moved to leiningen.
>
> I tried to make sense of things, but can't. I must be missing something
> big here.
[...]
> How do people deal with this?
Here's how I do it.
We've got five people in different places with different systems
writing code for a dozen projects, of which five are Clojure projects.
We use a common git server for coordination.
Our standard project layout places all five Clojure projects as
siblings in a common directory, along with a directory called "xglib"
that contains common libraries.
We tried leiningen, and it's appealing in some ways, but it makes too
many assumptions about the way projects are organized, and its habit
of downloading scads of jars quickly became annoying. We have five
projects with partially-overlapping dependencies, some of which have
others as dependencies. It was tiresome always to have multiple
redundant copies of the same jars all over the place, so instead we
created the common xglib directory, which is a sibling of the
projects, and which contains all the libraries used by them.
We use NetBeans as our standard IDE, not because I like it (I don't
like any current-generation IDE) but because it works, and telling new
contributors how to set it up to work with our projects is easy. So
each project is, in fact, a NetBeans project.
We also use Emacs, because a couple of use hate it slightly less than
we hate NetBeans. We have a custom variant of swank-clojure-project
(just a couple of function definitions, really) that knows how to find
our common libraries directory and add all the jars in it to the
CLASSPATH before running the Clojure REPL.
It's also possible to use other editors (TextMate, for example) with
this setup. The standard way we do builds is with NetBeans, but we can
also do them with ant.
It's not great, but it works, and it stays out of our way better than
anything else we've tried so far. A couple of us are old Lisp hackers
with memories of things like SK8 and Leibniz and MCL and SPE and
Genera, so grumbling is ongoing, but we must adapt to the times we
live in.
-Brent
On Mar 4, 1:59 pm, David Nolen <dnolen.li...@gmail.com> wrote:
> On Thu, Mar 4, 2010 at 10:19 AM, Stuart Sierra
> <the.stuart.sie...@gmail.com>wrote:
>
> > As many of you know, I've always been tepid on lein. I'd rather go
> > with Maven whole-hog, because that offers the most robust model for
> > incorporating Java libraries.
>
> If Lein evolves to to handle dependencies of dependencies and intelligently
> generates the classpath based on these dependencies (instead of copying
> files around) what advantage does Maven really have?
>
> David
On Mar 11, 5:07 am, Brent Millare <brent.mill...@gmail.com> wrote:
> Since leiningen downloads everything to a local repo, can't we do away
> with copies and use symlinks if they are supported by the filesystem?
> I feel there should be an option for this.
Why not adding the files from the repo directly to the classpath?
Works everywhere and doesn't copy...
Sincerely
Meikel
> Since leiningen downloads everything to a local repo, can't we do away
> with copies and use symlinks if they are supported by the filesystem?
> I feel there should be an option for this.
What benefit does this have aside from a tiny saving in disk space?
The way it currently works has a number of benefits:
* really simple and easy to understand!
* you can tarball your project directory, rsync it another machine or
whatever and everything you need will be right there. No need to
worry about redownloading anything or copying your local maven cache
around.
* you can run stuff manually just with:
java -cp 'src:classes:lib/*' clojure.main
* really simple and easy to understand! (worth repeating)
Doing tricks with symlinks or having lein set your classpath magically
like Maven does just seems to be adding complexity for the sake of
complexity.
Cheers,
Alex
Not that tiny when you multiply it across the dozens of projects on
your hard drive.
repos $ du -hc $(ls */lib/*.jar) | fgrep total
291M total
Add to that the size of the Maven repo itself.
Symlinks are nice.
> * you can tarball your project directory, rsync it another machine or
> whatever and everything you need will be right there. No need to
> worry about redownloading anything or copying your local maven cache
> around.
Unix is great.
man tar:
-h, --dereference
don't dump symlinks; dump the files they point to
>> What benefit does this have aside from a tiny saving in disk space?
>
> Not that tiny when you multiply it across the dozens of projects on
> your hard drive.
>
> repos $ du -hc $(ls */lib/*.jar) | fgrep total
> 291M total
Cost (on standard disks): < 5 cents.
Sorry, that's tiny. It's even less than 0.5% of the small SSD I have in
my laptop.
Seriously, this is just premature optimization.
You're seriously fine with every single Leiningen-based project
spitting a redundant copy of Clojure, contrib, and whatever other
dependencies it needs onto disk, and doing so every time it picks up a
new snapshot? That every tiny library you pull off GitHub grabs
another few dozen MB, rather than a few dozen KB?
290MB is only tiny if you have an empty disk. 290MB here, 290MB there,
and suddenly the 13GB free on my 180GB laptop disk starts looking
mighty cramped. I have a lot of stuff I'd rather store on that disk
than 290MB of redundant jar copies. Those hundreds of megs get backed
up, scanned, indexed, defragmented, yadda yadda. What a waste.
You might also be overlooking disk caching, paging, JVM memory
sharing, and other performance effects from having multiple instances
of the same jars in memory. I know that the Mac JVM does some tricks
to share memory in this way.
Furthermore, it's slower to copy 28MB of jars than it is to symlink
them:
$ time cp -R lib /tmp/foo
real 0m2.981s
user 0m0.007s
sys 0m0.152s
... surprise disk slowdown! All the nonsense of looking up snapshots,
checking deps against the Maven repo, etc. already takes so much time
that I dislike using Leiningen. Anything to bring that down by a few
hundred milliseconds and a few fewer disk head slaps would be nice.
There is no engineering reason to blindly copy jars into lib/. Hard
links, symlinks, or computed classpaths are all better solutions in
every single way.
"Premature optimization" is the act of expending significant time
addressing a performance problem prior to measurement, typically
allowing unmeasured performance concerns to affect system design.
Using symlinks instead of copying would not be a significant amount of
work, doesn't affect the overall design, and we're hardly flying
blind. You're using the wrong phrase. Here's a different Knuth quote:
"In established engineering disciplines a 12 % improvement, easily
obtained, is never considered marginal and I believe the same
viewpoint should prevail in software engineering"
I'd call this a decent improvement, and it's certainly easily obtained.
Whatever happened to engineers building a decent solution out of a
sense of pride, or even making the most basic half-assed attempt to
conserve resources?
Sometimes I despair.
> Whatever happened to engineers building a decent solution out of a
> sense of pride, or even making the most basic half-assed attempt to
> conserve resources?
I take your points but I think we may just have different priorities of
what to conserve, so we could probably argue about this all day. To me
disk space is cheap and simplicity precious. I'd rather waste 300mb of
space then have to muck around sorting out link problems when I run the
code on a different server and suddenly all the symlinks are broken
because I forgot an obscure flag on my copy/tar/rsync/whatever.
Hard links are better, there's much less that could go wrong (but
obviously they have the limitation that they don't work across different
filesystems).
The implementation is also not so trivial as you make out. How does one
even create a symlink (or hardlink) on the JVM (prior to the unreleased
JDK7)? There's no way I know of to do it via the standard library.
You'd have to resort to JNI or shelling out to "ln". What happens if
you're on a platform or even just a filesystem that doesn't support
links? Do you start littering the code with per operating system
conditionals? If so, what happens if you're technically running on Linux
but with a different userspace (perhaps Android or something) and there
happens to be no "ln" binary? Another special case? Very quickly we're
spiralling down into a mess of over-engineering.
-Per
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
-Per
-Per
2010/3/11 Per Vognsen <per.v...@gmail.com>:
> The way to go is definitely symlinks or hard links, not classpath
> hijinks.
Why so ?
If you use 'mvn repl' or 'lein repl' to start your repl, why not just
let mvn / lein manage the dependencies from beginning to end ?
>> Since leiningen downloads everything to a local repo, can't we do away
>> with copies and use symlinks if they are supported by the filesystem?
>> I feel there should be an option for this.
>
> Why not adding the files from the repo directly to the classpath?
> Works everywhere and doesn't copy...
Historically this has been because calculating the classpath couldn't
be done in Clojure itself since it needed to be done before the JVM
booted. Having to figure this kind of thing out in a shell script or
in elisp is a headache if the rules are not simple. But now with the
subclassloader approach that lein and mvn use it's less of an issue.
But it would mean deprecating the pure-elisp swank launcher, and I
don't know how it would affect other tools (IDEs, etc), so I'm still
hesitant.
I would be happy with a (simple-ish) patch to enable optionally
switching to using symlinks or hard-links as this would be the
lowest-impact fix, but I can't bring myself to care enough about the
number of megabytes currently being wasted to code it myself; sorry.
-Phil
Granted, I'm coming at this from a common lisp background, but I think many lisp developers are used to not just working on a single project, but working on a bunch of related projects simultaneously, including the the language/runtime environment itself. I'd like a single instance of clojure, a single clojure-contrib (I still don't understand why I had to (admittedly seamlessly) download a bunch of jars to clean the build directory), a single incanter instance, etc... Perhaps I'm missing the obvious here, but I continue to fail to understand why the clojure world is doing things this way.
On a related note, why is the build system the cool kids seem to be using (leiningen) controlled by a bunch of shell scripts? Surely things like compiling .clj source files and making jars should be operations one can drive from the clojure repl, no?
Oh, and on a related note, I hate being forced into the src, test, lib heirarchy... Let me put my files where I want them to go. Whatever happened to the lancet build system from Stuart Holloway's book? That seemed to make sense to me.
thanks,
Cyrus
I think there is a big benefit in consistent directory structure
across projects, but agree that it should still be *possible* to
configure things a different way.
I am not actively developing Lancet. Anyone who wants is certainly
welcome to fork it and continue. It is used under the covers in
Leiningen IIRC.
Stu
Not if we want to precompile code and create an überjar, or otherwise
integrate with Java. Ultimately, tools expect a pile of jars listed in
the classpath. Management of those jars is easier if they're (actually
or virtually) in one place.
I'm from a CL background too, and I do long for the days of the
continuum from a load script through to ASDF and friends, but at some
point it makes sense to give up and do things The Java Way.
Even simple things like log4j.properties are looked up on… you guessed
it: the classpath. It's not a battle that's worth fighting.
> On a related note, why is the build system the cool kids seem to be
> using (leiningen) controlled by a bunch of shell scripts? Surely
> things like compiling .clj source files and making jars should be
> operations one can drive from the clojure repl, no?
It's a shell script wrapping a bunch of Clojure code. I'm sure you
could use Leiningen as a library: each task is simply a function which
calls out to Maven or Lancet.
> Oh, and on a related note, I hate being forced into the src, test,
> lib heirarchy... Let me put my files where I want them to go.
> Whatever happened to the lancet build system from Stuart Holloway's
> book? That seemed to make sense to me.
That's convention over configuration (though you can put alternatives
in the Leiningen project.clj file).
It's one of those things where it's easier to go with the flow, I'm
afraid.
On Thu, Mar 11, 2010 at 01:32:36PM -0800, Phil Hagelberg wrote:
> Historically this has been because calculating the classpath couldn't
> be done in Clojure itself since it needed to be done before the JVM
> booted. Having to figure this kind of thing out in a shell script or
> in elisp is a headache if the rules are not simple. But now with the
> subclassloader approach that lein and mvn use it's less of an issue.
> But it would mean deprecating the pure-elisp swank launcher, and I
> don't know how it would affect other tools (IDEs, etc), so I'm still
> hesitant.
I think this is a typical emacs misconception. Vim is an editor. Why
should it have to know how a JVM classpath works? Setting this up is the
responsibility of the developer. Put everything in lib/? No problem.
Let maven/gradle/lein handle the classpath? No Problem. Have an esoteric
shell script doing some voodoo incatations? No Problem.
Putting this logic into the editor was the failure in the first place.
Sincerely
Meikel
Also, I noticed that symlink support was added to the master branch. I
read the documentation for maven's ant symlink support and didn't see
any mention of hardlinking. Does this imply that hardlinking support
would need to be implemented separately? I feel hardlinking is a more
elegant solution as ~/.m2/repository can act as a true cache with size
bounds but without the fear that projects that depend on deleted
artifacts would be affected.
> Can someone please elaborate on this subclassloader process? Although
> the design for leiningen is architecturally simple and understandable
> and thus easy to extend, I'm still quite unfamiliar with the "java
> way". I believe a little documentation on this aspect can make a
> significant improvement in allowing contributions to leiningen.
Given I implemented the initial support for it, I'll give it a go. In
earlier versions of Leiningen you weren't able to compile code against a
different version of Clojure to the one that Lein was running on top
of, as you could only have one or the other in the classpath. One way to
get around this is to fork off a new java process and run the compile in
a separate JVM. Unfortunately we'd then be incurring the JVM's startup
overhead twice (and lein is already annoyingly slow to start). Wouldn't
it be nice if we could just run the compile in the original JVM but
isolated in a container somehow?
Well we can actually do this because the JVM supports custom
classloaders. A classloader is an object that when given a classname,
locates the appropriate class files (in jars, classpath etc) and loads it
into memory. There's a default builtin classloader which searches the
classpath you supply on the command-line. Custom classloaders can be
used for a number of things, for example loading jars over the network
or out of a new archive format. They also play a strong role in the
JVM's security module (for sandboxing etc). When a class is loaded
(manually) into a custom classloader, any classes it in turn
tries to load (by normal means) are also loaded via that same
classloader.
So basically what Lein does is creates a new classloader (it actually
reuses Apache Ant's implementation) with a different classpath which
points to the version of clojure and other libraries that you want to
compile against.
Take a look at the eval-in-project function in leiningen/compile.clj.
You pass in a form to be evaluated as an argument. It's converted to a
string and then clojure.main is invoked with the '-e' argument to
evaluate the string 'inside' the new classloader. The reason its done
this way is so that its easy to fall back to forking and also because
objects in the new classloader aren't compatible with those in the
original (as they're potentially different versions of Clojure).
You can see that it doesn't work perfectly for all cases, so for example
if you're using native libs or OS X it'll fork instead. (OS X injects
some GUI stuff into the classpath of all the apps it runs, so anything
using using Swing or AWT doesn't work properly in the new classloader).
You may have wondered how Java servlet containers like Jetty and Tomcat
are able to load different versions of the same class into the same JVM
without them conflicting. As I understand it they use basically the
same mechanism.
Hope that helps. If you're having trouble following the classloader
stuff just think of the eval-in-project function as starting a new
"virtual" JVM with a different classpath and evaling some code in it.
Its really not all that complex, its just the terminology makes it sound
somewhat arcane.
Alex
On another note, I'm currently concerned with utilizing a global
configuration of leiningen (e.g. installing plugins and define
additional repositories globally). My current idea is to create a git
fork/branch to manage personal configuration files within the checkout
folder and then implement a bootstrapping script to facilitate
automating the deployment of alternative branches of leiningen.
Brent