Prepping clojure for packaging (was: Re: Clojure for system administration)
Peter Schuller
> major thorn in the side of the JVM, especially for Clojure and other
It seems to be that there are two problems here.
One problem is that there needs to be a convention for a clojure
"binary" that works consistently across platforms. I maintain the
clojure port in the FreeBSD port collection, and I install an
appropriate wrapper script such that you can in fact do:
clojure myscript.clj
And have it "just work" without further ado.
This is something that I think needs to be made more consistent;
probably by having the clojure distribution ship with such a script
that works on all POSIX shells, and making it clear to packagers that
it is intended to be the standard way of invoking clojure.
The other issue is that of picking up libraries. Here scripting
languages like ruby/python/etc have a well-defined convention for
doing this, where you have certain system paths that are intended to
contain modules. The exact path will vary with platform (for example
as a function of whether you use pycentral, are on debian, freebsd,
redhat, etc).
I would suggest that clojure could have a concept of "site library
path" that would be the responsibility of packagers to set
appropriately (just like 'site-packages' for python) where .jar:s
would be automatically picked up. Alternatively there could be two
such paths; one for a path that gets automatically added to CLASSPATH
and one that gets automatically scanned for .jar files.
As a practical example, I envision that what for Python might be:
/usr/local/python2.5/site-packages/X/Y/Z
Would for clojure be:
/usr/local/clojure/site-jars
and
/usr/local/clojure/site-classpath
What do people think about this? I think it's all about providing a
documented default method of finding libraries; packagers for
respective platforms will then follow suit, I think.
This also makes it much easier to start packaging clojure libraries in
respective platform's packaging systems, where they can install .jar:s
in the appropriate place using common infrastructure shared across all
clojure packages.
So in short: Defining some semantics and interfaces that make sense
from the perspective of debian/freebsd/redhat/whatever package
management, I think is likely to result in a more consistent end-user
experience once package maintainers start picking this up, just like
the case is with Python and Ruby.
I'd be interested in looking into getting some kind of draft
implementation of this, if people think it's a good idea (though I
can't make time commitments - busy busy busy).
--
/ Peter Schuller
Stuart Sierra
> One problem is that there needs to be a convention for a clojure
> "binary" that works consistently across platforms.
My extremely biased opinion: Clojure is not a scripting language.
The "binary" is java. System-wide classpaths are a bad idea.
-SS
Peter Schuller
One can spent a lot of time debating the definition of 'scripting
language', but sure. The bigger issue though:
> The "binary" is java. System-wide classpaths are a bad idea.
In theory I agree with you. The concept of site-local (system wide)
python paths, ruby library paths, and linker paths for C, are horribly
and completely broken. (It is somewhat acceptable in the case of
native libraries since they map to the underlying platform and
performance optimizations such as read-only mmap:ing apply, and there
is a system in place for versioning.)
Unfortunately, because of the practical reality that the vast majority
of the software that needs packaging for a platform are written in
languages and using tools that make prohibitively problematic to try
to coerce this set of arbitrary software into a world view where the
system-wide stuff does not exist, all major packaging systems (debian
packages, rpms AFAIK, freebsd ports, pkgsrc, etc) operate under the
assumption that this is how you package software.
What this means is that there tends to be a pre-existing
infrastructure to handle packages that install files in some
particular system-global location. Coercing a packaging system to
handle a situation where the same software can be installed in
multiple versions and appropriately selected by some particular
version of some software that depends on other software, is
problematic.
I suspect that a key feature of clojure that has contributed to it's
very promising level of popularity and use, is that it is
comparatively easy to get going writing real software with it, and the
fact that things that "should" be available, *are* available (in the
form of Java libraries to some extent, but also largely due to being
based on the JVM).
I submit that adopting something similar to python's/ruby's system of
loading code may be conducive to lowering the barrier of entry and
increasing acceptance of Clojure as a platform to build on.
As a developer, I have no problem with using leinigen for example
(it's great!). But while an executable uberjar is extremely
convenient, it does not tie into the packaging system of a platform
(and there may be licensing issues with e.g. GPL libraries and
re-distribution of uberjars).
As a system administrator, you want to maintain some number of systems
in some consistent fashion. This includes having a set of software
that is supported, upgraded and maintained using a consistent tool
set. Python and Ruby applications are well-behaved from this
perspective. Clojure is not. This is why, even though Ruby has gems
for example, there are still ports of specific ruby gems at specific
versions in the FreeBSD ports tree - because that means they are
suddenly integrated into the platform's packaging system. The fact
that a random user can "gem install XXX" does not obviate the need for
such integration. All forms of change management tend to be difficult,
and software installation is a form of (particularly complicated)
change management, assuming you need to keep stuff up-to-date.
As a user, you want to use some piece of software without spending
time figuring details out that should be left the programmer and the
packagers for your platform.
A project I started a few months ago, I wanted to write in Clojure.
There were two major reasons why I ended up going with Python, and
this is one of them (the other was POSIX Integration issues). I knew
that I wanted something which the user could easily install and use
using the tools that are integrated with his or her platform, without
knowing anything about Java or Clojure or mucking about with things
manually. Something which could just *be there* by default, as part of
the regular experience of the operating system.
In summary, I think one should not underestimate the importance of
making sure users can just "aptitude install your-application"
(substitute for whatever with freebsd//redhat/etc). Making it as easy
as possible for applications writers to in turn make it as easy as
possible for packagers to make that happen, is, I think, important.
Having said that, this does not necessarily mean that one has to adopt
the the Python/Ruby style of system-wide library paths. But I think it
does mean that there needs to be some sensible way for an operating
system's native packaging infrastructure to interface with the
infrastructure of the language. And any random blackbox doesn't cut
it; certain demands are imposed on the blackbox in order for it to
play nice with the packaging infrastructure.
At this time I haven't thought this through enough in the case of
Clojure to offer a practical suggestion that does not involve
system-wide library paths.
--
/ Peter Schuller
Michał Marczyk
script to launch the app with a minimal classpath, Leiningen style,
based on some assumption regarding the whereabouts of versioned jars
on the system? (E.g. jline-0.9.94.jar rather than jline.jar, say.)
That seems to me to be preferrable by far to just dropping the whole
of /usr/share/java & /usr/lib/java (for Ubuntu, or the equivalent for
other platforms) on the classpath and then hoping that there are no
incompatible versions of the same jars on the monster classpath this
produces.
Note that it will never do to have dependencies handled by a system
which isn't capable of installing several versions of the same jar
side-by-side... I don't know what apt / rpm / ports do about Java lib
versioning, so maybe there's no problem here?
Sincerely,
Michał
Peter Schuller
> script to launch the app with a minimal classpath, Leiningen style,
> based on some assumption regarding the whereabouts of versioned jars
> on the system? (E.g. jline-0.9.94.jar rather than jline.jar, say.)
Something along those lines was what I had in mind (rather than, as
you say, just drop everything onto the CLASSPATH). It would not be
something you want to fuss with necessarily during development, but
some standard way of "building" your application in a packaging
friendly manner.
> Note that it will never do to have dependencies handled by a system
> which isn't capable of installing several versions of the same jar
> side-by-side... I don't know what apt / rpm / ports do about Java lib
> versioning, so maybe there's no problem here?
As far as I can tell, few Java things are packaged. I suspect because
of this reason, in part. Things that are packaged tend to be those
things that depend only on the JDK itself, or on very simple
dependencies, or else major stuff like 'eclipse' which release
engineering and huge dependency trees are not your concern.
But as far as I have encountered, there is no magic solution to the
multiple-versions-of-the-same-library problem that integrate with
native OS packaging systems (that doesn't mean there are none).
Maybe for Clojure it would indeed be more desirable to have some
well-defined interface for packaging systems to use when dealing with
clojure applications. It may not be entirely trivial though. For
example with FreeBSD ports, the build step is expected to only depend
on downloading certain well-defined and checksum controlled files from
remote locations. You do not want something that arbitrarily downloads
stuff during building. I'm pretty sure similar conventions exist for
e.g. Debian.
It's a difficult problem, and I'm not blaming Clojure. Especially
since I think the Java/Clojure/Maven way of dealing with dependencies
is vastly superior to the traditional way. But the practical issue
remains that if I want to write some software that I want sysadmins in
various situations to want to use effortlessly (in my case, a backup
tool), problems like these do get in the way of choosing Clojure.
Maybe uberjars are the pragmatic approach after all. But then what
about GPL libraries and similar license hassles?
Actually, how is this dealt with in the Maven community?
--
/ Peter Schuller
Constantine Vetoshev
> But the practical issue
> remains that if I want to write some software that I want sysadmins in
> various situations to want to use effortlessly (in my case, a backup
> tool), problems like these do get in the way of choosing Clojure.
>
> Maybe uberjars are the pragmatic approach after all. But then what
> about GPL libraries and similar license hassles?
I can't comment on the licensing problem, but could you expand on what
exactly you don't like about packaging jars (as uberjars or separates)
and distributing them as part of your application? If you distribute
your app with precisely the jars you tested with, you don't have to
worry about subtle incompatibilities sneaking in. Except for obviously
bloated software (e.g., JBoss), Java jar files are relatively small.
When you package for Debian, put the jars in /usr/share/my-awesome-app/
jars, put in a dependency on sun-java-1.x, and provide a wrapper shell
script to start the whole thing (your classpath just includes the
stuff you provided). For the installing admin, your package is just an
"aptitude install my-awesome-app" away, and it just pulls down a JVM
dependency. Simple, and fairly lightweight.
I stopped using Python and Ruby and Perl partly because the packaging
situation for all those languages is a horrible mess. For example, if
you write a Ruby app, you are highly likely to run into trouble
packaging it even for a recent distribution because your dependencies
may not have been packaged or made available as rb-*.debs in the
official repository. Or, rb-*.debs exist, but provide an incompatible
version of the library you need. Ubuntu and Fedora may ship different
versions of common gems, so you can't even code against an obvious
lowest common denominator. (To make matters worse, "gem install",
"easy_install", and "cpan -i" all create weird parallel universes of
packages which may cause conflicts with system-provided packages.
Python's virtualenv helps mitigate the problem to some extent.)
C programs have it easier, because the overwhelming majority of
commonly used libraries are packaged and available in recent
distributions, and I have rarely had trouble with just writing against
a slightly older version of a library. It's perfectly reasonable to
dynamically link against libjpeg and make a note of a libjpeg
dependency in the package descriptor file.
In short, I think that the Java and Clojure way of packaging software
make life much easier for programmers, package maintainers, and
administrators, not harder. Making applications self-contained helps
avoid dependency hell at packaging and deployment time, not to mention
during development. It's a bit wasteful to have multiple copies of
log4j or whatever floating around, but I'd rather waste a few
gigabytes of disk space on duplicates than deal with system-wide
classpaths.
Mike Meyer
Constantine Vetoshev <gepa...@gmail.com> wrote:
> On Feb 6, 1:06 pm, Peter Schuller <peter.schul...@infidyne.com> wrote:
> > But the practical issue
> > remains that if I want to write some software that I want sysadmins in
> > various situations to want to use effortlessly (in my case, a backup
> > tool), problems like these do get in the way of choosing Clojure.
> >
> > Maybe uberjars are the pragmatic approach after all. But then what
> > about GPL libraries and similar license hassles?
>
> I can't comment on the licensing problem, but could you expand on what
> exactly you don't like about packaging jars (as uberjars or separates)
> and distributing them as part of your application? If you distribute
> your app with precisely the jars you tested with, you don't have to
> worry about subtle incompatibilities sneaking in. Except for obviously
> bloated software (e.g., JBoss), Java jar files are relatively small.
> When you package for Debian, put the jars in /usr/share/my-awesome-app/
> jars, put in a dependency on sun-java-1.x, and provide a wrapper shell
> script to start the whole thing (your classpath just includes the
> stuff you provided). For the installing admin, your package is just an
> "aptitude install my-awesome-app" away, and it just pulls down a JVM
> dependency. Simple, and fairly lightweight.
It means that your software isn't playing well in the system. You
install a duplicate copy of everything - or nearly everything - you
require, basically creating a "weird parallel universe" just for your
application.
> I stopped using Python and Ruby and Perl partly because the packaging
> situation for all those languages is a horrible mess. For example, if
> you write a Ruby app, you are highly likely to run into trouble
> packaging it even for a recent distribution because your dependencies
> may not have been packaged or made available as rb-*.debs in the
> official repository. Or, rb-*.debs exist, but provide an incompatible
> version of the library you need. Ubuntu and Fedora may ship different
> versions of common gems, so you can't even code against an obvious
> lowest common denominator. (To make matters worse, "gem install",
> "easy_install", and "cpan -i" all create weird parallel universes of
> packages which may cause conflicts with system-provided packages.
> Python's virtualenv helps mitigate the problem to some extent.)
The first part isn't an issue with the python/ruby/perl packaging
software, it's an issue with the OS packaging systems providing
out-of-date or no versions of things you need. That's pretty much why
I've punted on Linux as a development environment - I couldn't find a
distribution that had up-to-date versions of the tools I wanted
available.
This second part also sounds like a problem with the OS packaging
system, not python, ruby, cpan, etc. FreeBSD solves these problems by
making the packaging system cpan/easy_install/etc. aware, so that when
you install an OS package, it goes through those systems, and when you
install something with those systems, the OS will notice them and let
them satisfy the dependency on things that need it. I suspect that
pkgsrc does the same, given how modular it is.
> C programs have it easier, because the overwhelming majority of
> commonly used libraries are packaged and available in recent
> distributions, and I have rarely had trouble with just writing against
> a slightly older version of a library. It's perfectly reasonable to
> dynamically link against libjpeg and make a note of a libjpeg
> dependency in the package descriptor file.
You've had better luck than I've had. Creating a truly static binary
using a modern GCC is nearly impossible, given that it wants to
dynamically link part of it's support code. I inevitably find that
critical shared libraries either aren't in the packaging system, or
are so far out of date the author lists them as "DO NOT USE". I
frequently find that even when the available tools are only a little
bit out of date, critical bug fixes or features aren't available in
that "slightly older library". Again, why I don't use Linux as a
development platform.
> In short, I think that the Java and Clojure way of packaging software
> make life much easier for programmers, package maintainers, and
> administrators, not harder. Making applications self-contained helps
> avoid dependency hell at packaging and deployment time, not to mention
> during development. It's a bit wasteful to have multiple copies of
> log4j or whatever floating around, but I'd rather waste a few
> gigabytes of disk space on duplicates than deal with system-wide
> classpaths.
I think of this as the "Windows" way of packaging, though calling it
the "proprietary" way may be more accurate. That solution works for
Python, Ruby, C - pretty much anything. You just bundle up every
dependency you can get a license for, cram it all into the application
directory, and distribute that along with a binary that arranges to
use all those things. It's essentually required if you're building a
distribution for an OS that has no packaging system, and you don't
want to put your users through "install X, Y, Z, W ...." just to use
your software. It also works if you want to distribute applications
to run on lots of different Linux distros without having to have
someone make the effort building a package for that distro - because
if you don't include everything, you'll inevitably run into a distro
that built things just different enough that your binary won't work on
their distro.
Maybe disks are big and cheap enough that bundling every third party
library with every application is an acceptable solution, but it still
makes me feel dirty. On the other hand, the only alternative I know of
is to get someone who actually knows the target packaging system to
add your application as a package, properly integrated into that
packaging systems environment.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Konrad Hinsen
> I stopped using Python and Ruby and Perl partly because the packaging
> situation for all those languages is a horrible mess. For example, if
I agree. It's not just packaging, even providing sufficiently general installation scripts is a difficult task. Even with the help of tools like autoconf, figuring out what is installed and what needs to be added is far from trivial. Add to this the unfortunate but undeniable existence of incorrect or incomplete software installations on many machines. It's a real pain. I distribute a couple of scientific computing software packages, mostly written in Python but all with some C code in them and some external dependencies. A large part of technical support questions is about installation problems.
> In short, I think that the Java and Clojure way of packaging software
> make life much easier for programmers, package maintainers, and
> administrators, not harder. Making applications self-contained helps
I have no experience with this yet, but one reason for looking into the JVM has been the hope for a simpler deployment scheme.
Konrad.
Greg
>> make life much easier for programmers, package maintainers, and
>> administrators, not harder. Making applications self-contained helps
>
> I have no experience with this yet, but one reason for looking into the JVM has been the hope for a simpler deployment scheme.
Back in the day I used to write Java code, and indeed the concept of an "uber-jar" file made things nice, but if I remember correctly jar files cannot contain native code dependencies (I could be mistaken, let me know).
That was the only area where distribution is a pain with Java, damned native libraries.
- Greg
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
Rick Moynihan
>> I've been wondering about this. The classpath issue seems like a
>> major thorn in the side of the JVM, especially for Clojure and other
>
> It seems to be that there are two problems here.
>
> One problem is that there needs to be a convention for a clojure
> "binary" that works consistently across platforms. I maintain the
> clojure port in the FreeBSD port collection, and I install an
> appropriate wrapper script such that you can in fact do:
>
> clojure myscript.clj
>
> And have it "just work" without further ado.
>
> This is something that I think needs to be made more consistent;
> probably by having the clojure distribution ship with such a script
> that works on all POSIX shells, and making it clear to packagers that
> it is intended to be the standard way of invoking clojure.
This is my biggest issue with Clojure, and something I spend far more
time than I should handling. Clojure as a language is beautiful,
terse and simple, yet I find managing the VM arguments and classpath
tedious, and a real barrier to hacking in the language. Especially as
Clojure adds the REPL as an additional environment to run code in.
I have high hopes for Leiningen helping to tackle this issue, but it's
by not quite there yet.
I'd love for the clojure community to settle on a standard method of
starting Clojure programs, repl's and swank-servers (with classpath
and VM settings configured by a file).
IMHO Ruby (and probably python) do this better than Clojure, though
I'm not sure if we'll ever be able to find a solution we can all agree
on.
> The other issue is that of picking up libraries. Here scripting
> languages like ruby/python/etc have a well-defined convention for
> doing this, where you have certain system paths that are intended to
> contain modules. The exact path will vary with platform (for example
> as a function of whether you use pycentral, are on debian, freebsd,
> redhat, etc).
I personally don't put so much weight on this issue. In my experience
when it comes to things like ruby, people only tend to install
ruby,irb and rubygems from their package manager, and rely on using
rubygems to get their packages. As the distro's tend to be very out
of date when it comes to keeping these packages up to date, so I've
found it's almost always better to gem install them.
I also second the fears about having a system wide classpath. They're
brittle and very error prone.
R.
Brian Schlining
IMHO Ruby (and probably python) do this better than Clojure, though
I'm not sure if we'll ever be able to find a solution we can all agree
on.
I also second the fears about having a system wide classpath. They're
brittle and very error prone.
--
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Brian Schlining
Meikel Brandmeyer
maybe I don't understand the problem. Why can't the system provide
some kind of local repository? The package system (deb, rpm, ports,
whatever) just installs the dependencies there. A wrapper script reads
in the dependencies and adds them to the classpath on program start.
Nothing is downloaded. There might be several versions of a library
installed. No global classpath. I think that's what maven/ivy do right
now. Why wouldn't this work together with a packaging system? (I think
FreeBSD shows the way to go: cooperation between the system and the
language.)
Sincerely
Meikel
Ramakrishnan Muthukrishnan
That's exactly what Debian does. For every Java package also provide
the maven xml file and the jar is discoverable from maven. The
installed packages on the local system acts as a local maven repo.
<http://wiki.debian.org/Java/MavenRepoSpec>
--
Ramakrishnan
Peter Schuller
> the maven xml file and the jar is discoverable from maven. The
> installed packages on the local system acts as a local maven repo.
>
> <http://wiki.debian.org/Java/MavenRepoSpec>
I see they also solved the problem of not downloading during build.
One should probably look into doing something similar for other
packaging systems then.
Guess I need to bite the bullet and learn Maven ;)
--
/ Peter Schuller
Brian Schlining
> That's exactly what Debian does. For every Java package also provideI see they also solved the problem of not downloading during build.
> the maven xml file and the jar is discoverable from maven. The
> installed packages on the local system acts as a local maven repo.
>
> <http://wiki.debian.org/Java/MavenRepoSpec>
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Brian Schlining
Meikel Brandmeyer
On Feb 10, 12:31 am, Brian Schlining <bschlin...@gmail.com> wrote:
> > > That's exactly what Debian does. For every Java package also provide
> > > the maven xml file and the jar is discoverable from maven. The
> > > installed packages on the local system acts as a local maven repo.
>
> > > <http://wiki.debian.org/Java/MavenRepoSpec>
>
> > I see they also solved the problem of not downloading during build.
>
> Just FYI, with Maven, the first time you run a build it downloads all the
> needed dependencies. If you've specified version numbers for ALL the
> dependencies it will never try to download them again and just use the ones
> in your local repo (i.e. ~/.m2/repositories). You can also disable
> dependency updates with the '-o' flag (o stands for offline); for example:
> 'maven clean install -o'
I would also think, that you can construct a local maven repository
via the packaging system (deb in this case) before the build. Then
tell maven just to use this repository. Then no download at package
build time would be necessary, no?
Sincerely
Meikel