I need help distributing the new contrib

[Stuart Halloway]

The "simply awesome" new contrib libs do not really exist until they can be made visible in maven repositories. Since there will soon be a lot of libs, there should be some common ground in the project setup to accomplish this. There is a design page at http://dev.clojure.org/display/design/Common+Contrib+Build.

Help greatly wanted. I would appreciate discussion and feedback, but even more than that would like for somebody to take ownership of this, leading the discussion and carrying it all the way through to implementation.

Thanks,
Stu

[Alex Miller]

This is very similar in some ways to a forge we built at Terracotta.  We had a bunch of open source projects, all in svn with Maven poms.  Our ci was somewhat manual in per-project setup but ran initially in CruiseControl and eventually in Hudson, which seems to work reasonably well for this stuff, especially when you have networks of dependencies between the projects.  

Our build maestro built a bunch of ruby scripts to handle the release process for individual projects.  Maven has release support that will move from SNAPSHOT to release, produce the artifacts, publish them to a repository, then bump the version numbers to the next -SNAPSHOT.  However, we didn't find that the Maven release plugins worked very well (and I've heard this independently from others as well), so we relied on custom scripts.  We also generated a web site off the project and release repository web site so you could browse docs and see releases, dependencies, etc.

I can verify by an existence proof that this kind of infrastructure will work.  It was somewhat fragile in the case of lots of fine-grained version dependencies.  If you have lots of deps, it's very hard to make sure that you're always pointing to the "newest" and also to manage the ripple from a release version in a lib used by lots of other libs.  Any lib that uses the changed lib must be re-released with a version update for anyone to make use of it unless you delve into the murky world of Maven version ranges (ref http://jira.codehaus.org/browse/MNG-3092).  This spot on the map should be marked "here be dragons".  I assume that if a contrib lib is so heavily used that this becomes a problem that's probably a good indication it should move towards core.

I would strongly recommend using a Maven repository manager like Nexus to manage the repo and not just a file system.  I'm sure the Maven guys would work a deal for a license for something like this.  It's certainly feasible to push jars to both a contrib repo and clojars at release if that's useful.

I am not offering to own this. ;)

---

From: Stuart Halloway <stuart....@gmail.com>
To: cloju...@googlegroups.com
Sent: Mon, October 25, 2010 6:45:13 AM
Subject: I need help distributing the new contrib

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.
To unsubscribe from this group, send email to clojure-dev+unsub...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.

[Chas Emerick]

Alex is correct on a number of the points he makes. I would only add
that, yes, the release plugin is tricky -- but not using it means that
you rewrite the release plugin (or some poor approximation thereof).
I hear there have been significant improvements in that department,
esp. with the release of Maven 3. We'll have to give it a look.

Stu, your bullet point of "Treat the pom.xml as the data of record"
strikes me as the right direction (and I'm sure no one is surprised by
my saying that), but the details are sticky. I would assume you mean
that we'll simply be using the results of e.g. `lein pom` when
deploying, not that we use it to drive the build.

To what degree would you like discussion here vs. on the wiki?

- Chas

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.

> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com

[Stuart Halloway]

Right on, the pom doesn't have to drive the build, but should be the baseline.

I would like for the *results* of the discussion to be on the wiki, so we keep a colocated record of design tradeoffs. Once this particular task has an owner, that person can make it so. :-)

Stu

[Chas Emerick]

On Oct 25, 2010, at 3:46 PM, Stuart Halloway wrote:

> I would like for the *results* of the discussion to be on the wiki, so we keep a colocated record of design tradeoffs. Once this particular task has an owner, that person can make it so. :-)

My thoughts are thus:

== Project-level build tooling

The simplest option from an administrative standpoint is to declare that all contrib builds are to use maven. This makes managing hudson and the deployments to maven repos a no-brainer.

FWIW, it would be trivial to define a clojure contrib-specific parent pom that would drop the required XML for any given actual project to something like 8 lines (defining artifactId, version, name, & description); only if the project had external dependencies would additional configuration be required. This would be advantageous for administrative purposes as well, as we could define project guidelines/requirements as part of that parent POM.

Beyond that, we can provide support for lein (for example), but that will require either setting up each lein-driven project in hudson with the necessary shell execs to "run the build" (whatever that implies given the usage of lein plugins, etc), get a pom (just `lein pom`), and then use hudson's maven install to deploy the resulting artifact to the target repository using that pom. Alternatively, a proper lein plugin for hudson would be nice, though a quick search doesn't seem to turn up anything.

Aside from technical considerations, I think we have to ask ourselves why we're supporting a diverse set of tools, and if we are, why we're supporting what we're supporting. If we're supporting lein because it's the most widely used right now (is it?), does that mean that we'll support whatever becomes more popular than it (e.g. cake or gradle or buildr or whatever) some time from now? Maybe that'll never happen, but it's worth thinking about.

A final question would be about plugins obtained from clojars (which is the case for all lein and cake plugins AFAIK). These would be running with whatever privs the hudson user has. I'm no security expert, but that sets off some bells for me...not just for the local build system, but also about what a bad actor might be able to inject into the artifacts of e.g. the main clojure build.

== Repositories

There's no good reason to not deploy artifacts to a proper repository. AFAIK, the only advantage to e.g. clojure's scp deployment to a filesystem location is that it helps to avoid running a repository app.

If we deployed project artifacts to a repository like Nexus (free version will do, though I suspect sonatype would set us up with a free "Pro" license), then we would gain:

* a passable artifact discovery UI (and various web services that would make building a more attractive frontend ostensibly straightforward)
* ability to proxy third-party artifacts (important insofar as contrib projects may have external dependencies)
* ability to provide artifact indexes to downstream proxies (aids in visibility of artifacts in users' repos as well as in IDEs)
* all sorts of user management facilities that would allow for the delegation of management activities (e.g. no need to chase down the one or two guys that have access to the build box if there's a problem, when there's a reasonably-sized group of trusted folk that have various admin rights on the nexus).

If that's convincing, then there's the further question of whether we'd like to deploy artifacts to Sonatype's OSS repo (which provides for easy syncing of releases to central):

http://nexus.sonatype.org/oss-repository-hosting.html

…rather than to our own nexus install. I think one will be necessary eventually anyway – assuming we want to provide a saner community repository – but that's a separate topic.

If we decide that getting artifacts into central is important (I think it's hard to argue that it's not), then that makes the question of supporting lein-driven build processes a difficult one. The entire OSS deployment toolchain is maven-based. There may be a reasonable way to connect the dots, but someone will have to do some serious digging.

== Aggregate artifacts

Packaging multiple libs into easy-to-consume distributions is easily accomplished via the assembly or shade plugins.

== Release process

If a project is using maven, then the path is defined: release plugin (slightly touchy to get set up, but then works well), driven by the hudson m2 release plugin (http://wiki.hudson-ci.org/display/HUDSON/M2+Release+Plugin), which provides a nifty "cut a release" action on a per-project basis. This runs a build, deploys the artifacts, tags the current SCM rev appropriately, and then pushes an incremented version number (in an automagic way, or as specified) back to git.

Projects not using maven would likely want to use the "regular" release plugin (http://wiki.hudson-ci.org/display/HUDSON/Release+Plugin), though we'd have to orchestrate the version-number-increment-and-push bit ourselves. I'm not sure how people will like their project.clj files being modified automatically (i.e. with comments lost, etc, presuming we're using the reader to slurp them).

I'm not sure if the current wiki page is talking about automating releases using commit messages or e.g. version.properties as found in the clojure project. Both options seem unnecessarily baroque IMO, especially if either of the release plugins for hudson above meet requirements.

Hopefully the above is a helpful perspective, though perhaps not a surprising one insofar as people know my history and leanings.

Cheers,

- Chas

[Alex Miller]

Since much of discussion so far hinges on whether Maven is a) requirement b) one of many allowed build tools or c) verboten, is there some way to get more feedback on that specific issue from the clojure/core team and contrib authors?

If relying on Maven:

+1 on using parent pom for common infrastructure. we did this too and found it very helpful for isolating plugin defs, etc

+1 on repository like Nexus over files. we found having the indexable artifacts for external users is really important.

+1 on deploying to central (regardless of whether there is a contrib-specific repo)

---

From: Chas Emerick <ceme...@snowtide.com>
To: cloju...@googlegroups.com
Sent: Tue, October 26, 2010 12:02:56 PM
Subject: Re: I need help distributing the new contrib

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.

To unsubscribe from this group, send email to clojure-dev+unsub...@googlegroups.com.

[Chouser]

On Tue, Oct 26, 2010 at 1:19 PM, Alex Miller <alexd...@yahoo.com> wrote:
> Since much of discussion so far hinges on whether Maven is a) requirement b)
> one of many allowed build tools or c) verboten, is there some way to get
> more feedback on that specific issue from the clojure/core team and contrib
> authors?

I'm no lover of maven, but all current solutions in this space
seem amount to: some boilerplate, a few "variables" dropped in
per contrib repo, and then generally ignoring the build spec file
until some version needs to be bumped or something. That is to
say, from this particular contrib author's point of view, I don't
really care very much as long as builds can be made to work
consistently. I think the build coordinator's job is
sufficiently undesirable as it is that priority should be given
to making that role as painless as possible.

Monolithic contrib is currently (a): maven is the only way to
build it. We're changing a lot of things in moving to one-repo
(and bug tracker and wiki, etc.) per contrib lib -- do we want to
change more than we have to right now? Would adding support for
non-maven build tools be any easier to do now than it would be
later?

--Chouser

[Stuart Sierra]

On Oct 25, 7:45 am, Stuart Halloway <stuart.hallo...@gmail.com> wrote:
> Help greatly wanted. I would appreciate discussion and feedback, but even more than that would like for somebody to take ownership of this, leading the discussion and carrying it all the way through to implementation.

I strongly favor using widely-supported, well-understood tools over
hacking up something ourselves. For me, that means Maven + Hudson +
Nexus (all free).

Let's adapt our process to whatever is easiest to do with those tools,
not try to shoehorn the tools into some process we invent.

Let's also remember that dependency management is HARD and never fun.

I've invested a lot of time in learning Maven, a little in Nexus and
Hudson. I am willing to help implement.

-S

[Chas Emerick]

On Oct 26, 2010, at 7:33 PM, Stuart Sierra wrote:

> I've invested a lot of time in learning Maven, a little in Nexus and
> Hudson. I am willing to help implement.

Just in case my prior message didn't make it obvious, the above applies to me as well.

- Chas

[Fogus]

From my perspective I know very little about Maven, Hudson, and Nexus,
but I am open to learning. I tend to fall on the side of Chas and
Stuart and think that the path of standard tools is preferred.

:f

[Sean Devlin]

+1 standard tools

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.

To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.

[Chas Emerick]

No one has jumped in with a counterpoint here, so I thought I'd take a swing at one.

--

Standard tools certainly make an administrator's job easier, but at the expense of (many) potential contributors that, given easier-to-use tools, would be far more likely to pitch in.  It's significant that the maven toolchain has not become the dominant choice among Clojure programmers, and even among new-contrib-authors.  This is significant insofar as we'll often be asking new contributors to mavenize existing projects (such as Michael's unifycle @ http://github.com/fogus/unifycle) before they are allowed under the new contrib umbrella.  Given people's typical degree of attachment to their build tools (and how they interact with other parts of their development environment, e.g. `lein swank`, plugins, etc), this requirement will likely dampen enthusiasm for contributing – perhaps to a degree surpassing contention around e.g. the CA, since this will be a persistent arrangement, rather than a one-time action.

--

Hopefully that's not a horrible channeling of a likely contra position, at least along one axis.

As much as I press for standard tools, and think that the typical contrib project's usage of e.g. maven would be perfectly reasonable for any competent contributor, I think it'd be wise for us to think about ways to provide even more accessible processes even if the initial spike of the contrib CI machine follows the reliable Maven+Hudson+Nexus path.

Some searching along the lines of "lein hudson" and variations didn't turn up much, which leads me to think that the lein community hasn't tackled the question of CI yet, and all it entails. (Someone please correct me if I'm mistaken.) Given that, perhaps a thorough examination of where Clojure Polyglot Maven is now and where it's headed would be fruitful.  There's obviously a ton of overlap between it and lein, at least in some of the details if not in their respective charters.

Phil might want to throttle me for suggesting it, but if there's a way we can make east meet west (by providing, say, a lein-maven-plugin that will allow the more dynamic lein plugins to exist and operate within polyglot maven as first-class citizens), we might end up with a more gentle ramp for new contrib participants as well as a more compelling all-clojure build tooling story (the former modulo security concerns w.r.t. lein plugins that I mentioned in my last message, which may or may not be paranoia/FUD).

- Chas

[Sean Devlin]

My motivation for recommending standard tools is more social than
technical, for one very specific reason:

People leave projects.

Each of us is one life event away from not being able to work on
Clojure any more. While I know this is not something any of would
like to do, it's going to happen. I know if I had to leave some of my
projects behind, sticking to a standard tool chain would improve the
chances of someone else picking up where I left off.

When someone is thinking about moving their library to contrib, and
eventually core, they need to consider the community aspects as well.
They are no longer owners of the code at this point, but a primary
steward. Being a good steward is being a good team player, and
sometimes that means using a tool chain you don't like.

Being able to have developers leave & other continue their work will
be pivotal to the ten, twenty and fifty year success of Clojure.

Okay, enough ranting.

On Oct 27, 10:39 am, Chas Emerick <cemer...@snowtide.com> wrote:
> No one has jumped in with a counterpoint here, so I thought I'd take a swing at one.
>
> --

> Standard tools certainly make an administrator's job easier, but at the expense of (many) potential contributors that, given easier-to-use tools, would be far more likely to pitch in.  It's significant that the maven toolchain has not become the dominant choice among Clojure programmers, and even among new-contrib-authors.  This is significant insofar as we'll often be asking new contributors to mavenize existing projects (such as Michael's unifycle @http://github.com/fogus/unifycle) before they are allowed under the new contrib umbrella.  Given people's typical degree of attachment to their build tools (and how they interact with other parts of their development environment, e.g. `lein swank`, plugins, etc), this requirement will likely dampen enthusiasm for contributing – perhaps to a degree surpassing contention around e.g. the CA, since this will be a persistent arrangement, rather than a one-time action.

> --
>
> Hopefully that's not a horrible channeling of a likely contra position, at least along one axis.
>
> As much as I press for standard tools, and think that the typical contrib project's usage of e.g. maven would be perfectly reasonable for any competent contributor, I think it'd be wise for us to think about ways to provide even more accessible processes even if the initial spike of the contrib CI machine follows the reliable Maven+Hudson+Nexus path.
>
> Some searching along the lines of "lein hudson" and variations didn't turn up much, which leads me to think that the lein community hasn't tackled the question of CI yet, and all it entails. (Someone please correct me if I'm mistaken.) Given that, perhaps a thorough examination of where Clojure Polyglot Maven is now and where it's headed would be fruitful.  There's obviously a ton of overlap between it and lein, at least in some of the details if not in their respective charters.
>
> Phil might want to throttle me for suggesting it, but if there's a way we can make east meet west (by providing, say, a lein-maven-plugin that will allow the more dynamic lein plugins to exist and operate within polyglot maven as first-class citizens), we might end up with a more gentle ramp for new contrib participants as well as a more compelling all-clojure build tooling story (the former modulo security concerns w.r.t. lein plugins that I mentioned in my last message, which may or may not be paranoia/FUD).
>
> - Chas
>
> On Oct 27, 2010, at 9:52 AM, Sean Devlin wrote:
>
> > +1 standard tools
>

[Konrad Hinsen]

On 27.10.2010, at 16:55, Sean Devlin wrote:

> My motivation for recommending standard tools is more social than
> technical, for one very specific reason:

...

You are perfectly right, but not at all in contradiction to what Chas said. The real question is: what are appropriate "standard tools" for the Clojure community? Is it Maven and friends? Leiningen? Something else yet? Can we already speak of a "standard"?

Here's my own point of view: I don't know much about Maven, and I don't like much in what I have seen until now. If contributing to the new contrib requires more Maven knowledge than installing the current contrib, then I don't expect to contribute anything any time soon. I have much less time for Clojure hacking that I'd like to have, and I certainly don't want to spend that scarce time fighting against Maven. Leiningen is a different story. I don't use it much, but it looks perfectly clear to me and I wouldn't mind working more with it. Ant would be acceptable as well.

Konrad.

[David Nolen]

On Wed, Oct 27, 2010 at 1:24 PM, Konrad Hinsen <konrad...@fastmail.net> wrote:

Here's my own point of view: I don't know much about Maven, and I don't like much in what I have seen until now. If contributing to the new contrib requires more Maven knowledge than installing the current contrib, then I don't expect to contribute anything any time soon. I have much less time for Clojure hacking that I'd like to have, and I certainly don't want to spend that scarce time fighting against Maven. Leiningen is a different story. I don't use it much, but it looks perfectly clear to me and I wouldn't mind working more with it. Ant would be acceptable as well.

Konrad.

For those of of us that don't really understand Maven, what more is required beyond generating a pom.xml?

David 

[Chas Emerick]

The development process is fundamentally the same:

- install maven

- have a pom handy (sample below)

- use the goals defined by the clojure-maven-plugin: http://github.com/talios/clojure-maven-plugin

Worth pointing out is that lein is built on top of Maven…so, you're using it already.  Doing what you're doing now with the mvn frontend instead of the lein frontend should be a pretty painless transistion unless there's something about people's workflow that I don't understand.

FWIW, if we were to establish a baseline parent POM for use with contrib (that would include references to the build.clojure.org repos, a sane default clojure-maven-plugin config, and other miscellaneous bits), a project's local pom.xml would look something like this at minimum (warning, hypothetical parent pom coordinates in use):

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>clojure.tools</groupId>

    <artifactId>nrepl</artifactId>

    <version>0.0.2-SNAPSHOT</version>

    <name>nREPL</name>

    

    <parent>

        <groupId>clojure.poms</groupId>

        <artifactId>contrib-base</artifactId>

        <version>1.0.0</version>

    </parent>

</project>

Unless the contrib project in question were to veer away from the typical build process, nothing else would be required (aside from attribution information, if the authors so desired).

- Chas

[David Nolen]

On Wed, Oct 27, 2010 at 1:55 PM, Chas Emerick <ceme...@snowtide.com> wrote:

Worth pointing out is that lein is built on top of Maven…so, you're using it already.  Doing what you're doing now with the mvn frontend instead of the lein frontend should be a pretty painless transistion unless there's something about people's workflow that I don't understand.

What transition? I can just run 'lein pom' right? To what purpose could using the mvn frontend directly have if I can generate the pom.xml with lein?

Sorry if I'm being slow.

David

[Chas Emerick]

The result of `lein pom`, while suitable for deployment, cannot be used to drive an actual build.  This implies that a pom.xml is the project-description-of-record.  One could back that into a project.clj, but that doesn't help with minimizing the work involved.

- Chas

[Stuart Sierra]

The strength of Maven is the POM (Project Object Model) itself. It is
a very generic model for describing how to build and deploy software
packages ("artifacts"). It is well-documented, widely-used, and
implemented by dozens of tools.

Leiningen cannot (yet?) handle round-trip conversions between pom.xml
and project.clj, nor do its generated pom.xml files provide sufficient
information to build a project using Maven alone. Based on Phil's
presentation at the Conj, this is not a goal of Leiningen. In fact,
his current plans for Leiningen appear to be headed in the direction
of diverging further from the POM.

I support convenience wrappers around mvn and pom.xml, but I believe
the only way to support a heterogeneous build environment is to
standardize around one formal model. The POM is the only such model
available, so we might as well use it.

-S

[David Nolen]

On Wed, Oct 27, 2010 at 2:47 PM, Chas Emerick <ceme...@snowtide.com> wrote:

The result of `lein pom`, while suitable for deployment, cannot be used to drive an actual build.  This implies that a pom.xml is the project-description-of-record.  One could back that into a project.clj, but that doesn't help with minimizing the work involved.

- Chas

So contrib build information is missing from 'lein pom'. So does this mean that what lein needs is 'lein contrib-pom' ?

David 

[Chas Emerick]

I'm not sure what 'contrib build information' means exactly.  Lein plugins are the big wildcard, along with whatever additional bits we end up needing to support the full CI lifecycle (at a minimum, being able to define a parent POM, though we may discover other requirements as we go along).

- Chas

[David Nolen]

On Thu, Oct 28, 2010 at 11:51 AM, Chas Emerick <ceme...@snowtide.com> wrote:

I'm not sure what 'contrib build information' means exactly.  Lein plugins are the big wildcard, along with whatever additional bits we end up needing to support the full CI lifecycle (at a minimum, being able to define a parent POM, though we may discover other requirements as we go along).

- Chas

So 'lein pom' is satisfactory for all Lein projects that don't rely on Lein plugins? I personally haven't used many Clojure libraries that require lein plugins for building - except Penumbra and Overtone (they need 'lein native-deps'). While both are great projects neither seem that useful in contrib.

David 

[Konrad Hinsen]

On 27.10.2010, at 19:55, Chas Emerick wrote:

>> For those of of us that don't really understand Maven, what more is required beyond generating a pom.xml?
>
> The development process is fundamentally the same:
>
> - install maven
> - have a pom handy (sample below)
> - use the goals defined by the clojure-maven-plugin: http://github.com/talios/clojure-maven-plugin

...

My gripe with Maven is that it is oriented towards deployment rather than development. What I would like to have is a build system that lets me do development with development-oriented tools, and then once I decide to do a release produce all the necessary files to turn my development source code base into a series of libraries with proper version numbers and dependencies betwee them.

During development, my priority is working straight from source code to make sure that the code I run is exactly the code I see in the editor. If my current understanding of Maven is right, it does not support this at all. Every minor change in any library forces me to a "release" first (if only for personal use) and update the dependencies in all other libraries. That's a major pain.

Konrad.

[Chouser]

Thoughtful and cogent critiques like this are valuable and
helpful. Thanks!

--Chouser

[Chas Emerick]

I'd hazard to say that your understanding of Maven is incorrect. If one of your projects' dependencies puts out a new release, *you* don't have to release in order to use it. Just change the version number (either in your pom.xml or project.clj), and carry on.

Perhaps I'm misunderstanding the problem?

- Chas

[Konrad Hinsen]

On 29 Oct 2010, at 16:02, Chas Emerick wrote:

> I'd hazard to say that your understanding of Maven is incorrect. If
> one of your projects' dependencies puts out a new release, *you*
> don't have to release in order to use it. Just change the version
> number (either in your pom.xml or project.clj), and carry on.
>
> Perhaps I'm misunderstanding the problem?

I think so. My typical situation is that I work on several libraries
in parallel. They are separate libraries because most users will use
only some of them, but there are lots of dependencies between them.
It's a bit like current contrib.

The typical situation I was referring to is that I add some new
feature to application A, library 1, and library 2. I edit source
files in all of those, and want to work from those source files during
the process. Then one day I decide that the new feature set is stable
enough and make releases for some or all of library 1, library 2, and
application A.

My understanding of Maven (which may be incorrect) is that in order to
be sure that application A uses the very latest changes to library 1,
I have to make a release of library 1 and indicate that new version in
the dependency list of application A. Otherwise Maven will stick to
the previous release that it has somewhere in its repository. If that
is true, I'll end up spending a lot of time producing releases that
noone but me will ever see - or I don't use Maven.

At the moment, I have a setup that puts all the source code of my
active libraries on the classpath, so I always work from source. When
I want to make a release, I use "lein pom" and "lein jar" and push the
results to clojars. The problem with that is of course that I never
use those files myself, so if something is wrong with the jars or the
poms (say due to a bug in my project.clj), I'd never notice. That's
why I'd prefer an integrate build tool for development and deployment,
if such a beast exists in the Java world.

Konrad.

[Alex Miller]

If you coordinate code through a managed repository, then you need to version that software to make builds repeatable and specific.  There are times when you want more flexibility.  Maven's tools for that include:

1) Version ranges - specify a range of versions [1.0, 1.1) as a dependency to be insulated from minor version changes.  The major caveat here is that Maven can pull SNAPSHOT versions from within that range which can be bad and dangerous http://jira.codehaus.org/browse/MNG-3092 and also that ranges can have unexpected corners to them.

2) -SNAPSHOT can be used during dev time to track a project still in development without changing callers on every minor delta.

In general, I have not found releasing a new version to be a problem and I've found the re-release of downstream projects to be only an occasional issue.  What I have found VERY tricky is trying to maintain multiple versions of a project against multiple versions of its dependencies which can quickly get out of hand as its n-dimensional.  

That is, if I have:

A with major releases 1.0 and 2.0

B with major releases 3.0 and 4.0

And I have project C which depends on A and B, how do I manage versions that deal with combinations of A and B (A:1.0 + B:3.0, A:1.0 + B:4.0, etc).  In general, you don't need to have the full matrix.  However, given that you have effectively a linear set of numbers in C (as Maven will order them absolutely), dealing with more than 1 combination as active branches can become problematic.  You effectively need not just a version number but a branch (as that is exactly what you do in source control).

I have spent at least a couple of hammock days on this problem in the past without coming up with any solution that fits well in the modern Maven repository world.  These situations are often temporary while you wait for the world to move towards a newer version but no less painful.  I have worked around these issues in various ways.  For example, I have been forced to use reflective checks to make a project switch between proper behavior for multiple versions of a library.  Or build dependency-specific versions like ProjectC_A3 and ProjectC_A4, etc.  I don't see Maven as the issue here - I think you would have the identical problems with any s/w that releases versions but does not have the concept of branches.  

---

From: Chas Emerick <ceme...@snowtide.com>
To: cloju...@googlegroups.com

Sent: Fri, October 29, 2010 9:02:34 AM

Subject: Re: I need help distributing the new contrib

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.

To unsubscribe from this group, send email to clojure-dev+unsub...@googlegroups.com.

[Stuart Halloway]

Phil described a leiningen feature that works around this issue, basically making it easy to say, locally, "use these jars preferentially." If maven doesn't make that easy enough, then we could add a shim that does.

Stu

[Alex Miller]

One way to do this in Maven is to set up a profile with an alternate set of dependencies.  Any time you want conditional build behavior, Maven profiles are often a good bet.  So you could set up a dev profile that pointed to a local repo instead of your main repo, then use -Pdev on the Maven build.  Or many other variations...

Lots of people use a local Hudson build on dev machines to manage this stuff too.  In fact, that is what the Sonatype guys recommend as a solution for this.  We have built scripts in the past that did similar (building a network of projects in dependency order).  

---

From: Stuart Halloway <stuart....@gmail.com>
To: cloju...@googlegroups.com
Sent: Fri, October 29, 2010 10:02:07 AM

Subject: Re: I need help distributing the new contrib

> To unsubscribe from this group, send email to clojure-dev+unsub...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.

To unsubscribe from this group, send email to clojure-dev+unsub...@googlegroups.com.

[Stuart Halloway]

IMO anything that requires "set up" as opposed to "just do it" is obstructive, hence the need for a shim.

Stu

To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.

[Konrad Hinsen]

On 29 Oct 2010, at 17:02, Stuart Halloway wrote:

> Phil described a leiningen feature that works around this issue,
> basically making it easy to say, locally, "use these jars
> preferentially." If maven doesn't make that easy enough, then we
> could add a shim that does.

If I could get "source code directories" in addition to "jars", that
would be just fine!

Konrad.

[Stuart Sierra]

On Oct 29, 12:53 am, Konrad Hinsen <konrad.hin...@fastmail.net> wrote:
> During development, my priority is working straight from source code to make sure that the code I run is exactly the code I see in the editor. If my current understanding of Maven is right, it does not support this at all. Every minor change in any library forces me to a "release" first (if only for personal use) and update the dependencies in all other libraries. That's a major pain.

If you you are working with Maven "-SNAPSHOT" versions, you do not
need to "release" anything.

Say you're working on projects A and B simultaneously. Your POM for
Project A will have a dependency on a "-SNAPSHOT" version of Project
B. You make a change in B and run "mvn install". That will compile,
test, package, and install B in your local Maven repository (~/.m2/
repository). Running any mvn goal in Project A will automatically use
the most most recent -SNAPSHOT of Project B.

When you're ready to make a release, do Project B first. Set the
version number to a non-SNAPSHOT, run "mvn deploy", and increment the
version number to the next -SNAPSHOT number. (The Maven Release
Plugin will do this for you, and even tag your git repo
automatically.) Next, set B's version number in Project A's POM to
compile/test/build with the released version of B. One you have
verified that A works with your release of B, go ahead and do the same
process with A.

This is actually simpler than Leiningen development, where you have to
manually re-run "lein deps" whenever a dependency changes. (Phil
mentioned at the Conj that he's working on this problem.)

-S

[David Nolen]

On Fri, Oct 29, 2010 at 1:34 PM, Stuart Sierra <the.stua...@gmail.com> wrote:

On Oct 29, 12:53 am, Konrad Hinsen <konrad.hin...@fastmail.net> wrote:
> During development, my priority is working straight from source code to make sure that the code I run is exactly the code I see in the editor. If my current understanding of Maven is right, it does not support this at all. Every minor change in any library forces me to a "release" first (if only for personal use) and update the dependencies in all other libraries. That's a major pain.

If you you are working with Maven "-SNAPSHOT" versions, you do not
need to "release" anything.

Say you're working on projects A and B simultaneously.  Your POM for
Project A will have a dependency on a "-SNAPSHOT" version of Project
B.  You make a change in B and run "mvn install". That will compile,
test, package, and install B in your local Maven repository (~/.m2/
repository).  Running any mvn goal in Project A will automatically use
the most most recent -SNAPSHOT of Project B.

When you're working on two projects simultaneously you don't want to have to run maven install or even lein deps for small changes to B. You don't want to goal, test, package, or even install. You're *developing*- I think this is unnecessarily tedious for the developer (it's why Lein supports sym-linking other libs instead).

Maven seems brings with it the C++/Java compile-wait frame of mind to Clojure programming. Lein/Cake try their best to keep that at bay. I also suspect this is why Rich Hickey prefers the simplicity (naivete even) of Ant.

I know that Lein's model probably doesn't scale for large projects, but considering the relatively modest scope of most of the contrib libraries at this point, do we need to force contrib authors to adopt such a build tool?

It's seems more effective to allow people to use whatever they want as long as they can provide a compliant POM. People who need more complex builds will have to bite the bullet and perhaps learn some Maven it sounds like, but it so far it doesn't seem like it needs to be requirement.

On a sidenote, the move to Maven for contrib to me was revealing how unfriendly Maven is for newcomers and contributors - perhaps it makes administration easier - though it hasn't been clear to me that even that is the case so far for Maven-ized contrib.

David

[Stuart Halloway]

> When you're working on two projects simultaneously you don't want to have to run maven install or even lein deps for small changes to B. You don't want to goal, test, package, or even install. You're *developing*- I think this is unnecessarily tedious for the developer (it's why Lein supports sym-linking other libs instead).

Agreed, and what we come up with will support this.

> I know that Lein's model probably doesn't scale for large projects, but considering the relatively modest scope of most of the contrib libraries at this point, do we need to force contrib authors to adopt such a build tool?

If we can get compliant poms out of lein then that is fine. If not let's make it so.

> It's seems more effective to allow people to use whatever they want as long as they can provide a compliant POM. People who need more complex builds will have to bite the bullet and perhaps learn some Maven it sounds like, but it so far it doesn't seem like it needs to be requirement.

Agreed. The whole point of "meeting at the pom" is that we don't have to care where your pom came from.

It would be an abject failure, and non-Clojurish, if we had to have an API (as opposed to a data format) here.

Stu

[Chas Emerick]

On Oct 29, 2010, at 2:16 PM, David Nolen wrote:

> When you're working on two projects simultaneously you don't want to have to run maven install or even lein deps for small changes to B. You don't want to goal, test, package, or even install. You're *developing*- I think this is unnecessarily tedious for the developer (it's why Lein supports sym-linking other libs instead).
>
> Maven seems brings with it the C++/Java compile-wait frame of mind to Clojure programming. Lein/Cake try their best to keep that at bay. I also suspect this is why Rich Hickey prefers the simplicity (naivete even) of Ant.

There's no doubt that Maven is from a different world than Clojure. This workflow issue is resolved (for me anyway) by excellent Maven tooling in any of the major IDEs, where dependencies are tracked without any intermediate build processes at all; i.e. I have gone days without running a single maven build across any of a half-dozen interrelated projects, all of which I'm changing and fiddling with in various REPLs.

> I know that Lein's model probably doesn't scale for large projects, but considering the relatively modest scope of most of the contrib libraries at this point, do we need to force contrib authors to adopt such a build tool?

There's far more moving pieces than just the developer's local workflow to consider when talking about contrib build tooling. Stu's (probably) right about "let's meet at the POM", but I want to emphasize that one needs to think about more than what goes on in a single contributor's environment in this context.

- Chas

[Chouser]

On Fri, Oct 29, 2010 at 3:04 PM, Chas Emerick <ceme...@snowtide.com> wrote:
>
> On Oct 29, 2010, at 2:16 PM, David Nolen wrote:
>
>> When you're working on two projects simultaneously you don't want to have to run maven install or even lein deps for small changes to B. You don't want to goal, test, package, or even install. You're *developing*- I think this is unnecessarily tedious for the developer (it's why Lein supports sym-linking other libs instead).
>>
>> Maven seems brings with it the C++/Java compile-wait frame of mind to Clojure programming. Lein/Cake try their best to keep that at bay. I also suspect this is why Rich Hickey prefers the simplicity (naivete even) of Ant.
>
> There's no doubt that Maven is from a different world than Clojure.  This workflow issue is resolved (for me anyway) by excellent Maven tooling in any of the major IDEs, where dependencies are tracked without any intermediate build processes at all; i.e. I have gone days without running  a single maven build across any of a half-dozen interrelated projects, all of which I'm changing and fiddling with in various REPLs.

Surely you can't just reload a namespace in a running REPL
though? You click some magic button that builds and installs
some tree of dependant jars, and launches a new REPL with the
appropriate classpath or something?

--Chouser

[Chas Emerick]

On Oct 29, 2010, at 3:27 PM, Chouser wrote:

>> There's no doubt that Maven is from a different world than Clojure. This workflow issue is resolved (for me anyway) by excellent Maven tooling in any of the major IDEs, where dependencies are tracked without any intermediate build processes at all; i.e. I have gone days without running a single maven build across any of a half-dozen interrelated projects, all of which I'm changing and fiddling with in various REPLs.
>
> Surely you can't just reload a namespace in a running REPL
> though? You click some magic button that builds and installs
> some tree of dependant jars, and launches a new REPL with the
> appropriate classpath or something?

Clearly, I need to do a screencast or something. ;-)

- Chas

[Konrad Hinsen]

On 29 Oct 2010, at 19:34, Stuart Sierra wrote:

> On Oct 29, 12:53 am, Konrad Hinsen <konrad.hin...@fastmail.net> wrote:
>> During development, my priority is working straight from source
>> code to make sure that the code I run is exactly the code I see in
>> the editor. If my current understanding of Maven is right, it does
>> not support this at all. Every minor change in any library forces
>> me to a "release" first (if only for personal use) and update the
>> dependencies in all other libraries. That's a major pain.
>
> If you you are working with Maven "-SNAPSHOT" versions, you do not
> need to "release" anything.
>
> Say you're working on projects A and B simultaneously. Your POM for
> Project A will have a dependency on a "-SNAPSHOT" version of Project
> B. You make a change in B and run "mvn install". That will compile,
> test, package, and install B in your local Maven repository (~/.m2/
> repository). Running any mvn goal in Project A will automatically use
> the most most recent -SNAPSHOT of Project B.

OK, that's slightly less bad than I thought, but running "mvn install"
is still too much for me. What I want is modify a function in project
B, press M-C-x to update it in SLIME, and use it right away in project
A. That's why I insist on working straight from source code files
while developping.

Konrad.

[Colin Jones]

What would need to change in the `lein pom`-generated pom file (besides adding the ability to specify a parent pom) in order to make it compliant in this context? Please excuse my Maven (and cake) ignorance.

I can't speak for Phil, but it seems like a win to me if it's possible give devs a choice of local build tooling, even if it needs to be a lein plugin.

Would something like that obviate the question marks around the new process? If so, maybe we could ping the lein/cake mailing lists and start working on plugins/tasks to make the desired poms..

Colin

On Oct 29, 2010 3:04 PM, "Chas Emerick" <ceme...@snowtide.com> wrote:

On Oct 29, 2010, at 2:16 PM, David Nolen wrote:

> When you're working on two projects simultaneous...

There's no doubt that Maven is from a different world than Clojure.  This workflow issue is resolved (for me anyway) by excellent Maven tooling in any of the major IDEs, where dependencies are tracked without any intermediate build processes at all; i.e. I have gone days without running  a single maven build across any of a half-dozen interrelated projects, all of which I'm changing and fiddling with in various REPLs.

> I know that Lein's model probably doesn't scale for large projects, but considering the relativel...

There's far more moving pieces than just the developer's local workflow to consider when talking about contrib build tooling.  Stu's (probably) right about "let's meet at the POM", but I want to emphasize that one needs to think about more than what goes on in a single contributor's environment in this context.

- Chas

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

...

[Chas Emerick]

On Oct 29, 2010, at 2:42 PM, Stuart Halloway wrote:

>> I know that Lein's model probably doesn't scale for large projects, but considering the relatively modest scope of most of the contrib libraries at this point, do we need to force contrib authors to adopt such a build tool?
>
> If we can get compliant poms out of lein then that is fine. If not let's make it so.

For my money, it seems like we should rely upon pmaven to emit our pom.xml serializations, simply because we'll likely want/need to fiddle with the POM along the way in various ways in order to get what we want. This will work as long as lein, cake, and pmaven (mostly) share a compatible POM serialization, but none of the leads of those projects have participated in this thread so far AFAICT. Cake and lein have already diverged from the POM in various ways; will they diverge more?

Along these lines, my concerns about lein/cake plugins remain, both in terms of linking them up into a maven build process (after pom-ifying the project.clj) as well as running them in a shared/privileged environment and their provenance (though this could be resolved through clojars v2).

Then there's wildcards like `lein multitest`. Something like it should probably be strongly recommended of all projects eventually, with the best corollary being use of the invoker plugin in maven (with a punt position of multiple profiles and multi-stage builds for all such projects). Trying to determine where the cutoff is between a project that can use lein/cake and one that really just has to use maven may get tricky.

This is all to say that it seems that "compliant pom" is a loaded concept.

- Chas

[Sean Devlin]

You'd be surprised how often you need to refer people back to it :)
 

- Chas

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

[Stuart Sierra]

On Oct 29, 3:39 pm, Konrad Hinsen <konrad.hin...@fastmail.net> wrote:
> That's why I insist on working straight from source code files  
> while developping.

And nothing about Maven stops you from doing that.

-S

[Konrad Hinsen]

I guess not, if I continue to work in two parallel worlds: one for
development, one for publishing releases. That's in fact how I work
now with contrib. But I don't consider this a good setup, since I
never work with the setup that ultimately gets published.

What I expect from a development-build-deployment system for a dynamic
language is a single specification of a dependency graph from which it
generates a classpath pointing to source directories for development
AND jar files for distribution.

Konrad.

[Chouser]

Wouldn't it be possible to add to clojure-maven-plugin the
ability to take a list of locally-available projects (path to the
pom.xml in a locally checked-out dir) from which to find source
paths to add to the classpath? This would add to maven itself
roughly the ability that I understand some of the IDEs to have
already. It would allow the project's own pom to remain the
standard of how to build, but would allow developers to easily
use locally modified versions of any of the project's
dependencies.

Is this a bad idea for some reason?

--Chouser

[Phil Hagelberg]

On Fri, Oct 29, 2010 at 12:42 PM, Colin Jones <trpt...@gmail.com> wrote:
> What would need to change in the `lein pom`-generated pom file (besides
> adding the ability to specify a parent pom) in order to make it compliant in
> this context? Please excuse my Maven (and cake) ignorance.

Thanks for bringing this up, Colin. I've been traveling and only
following this conversation loosely, but it seems like the concerns
around non-maven tools have not been clearly laid out other than the
fear of malicious plugins. It should be noted that Leiningen plugins
are almost always used for convenience during development time and
only rarely required to build jars. Now that Leiningen contains a java
compilation task, the only exception I'm aware of is native-deps,
which I don't think any contribs need.

> I can't speak for Phil, but it seems like a win to me if it's possible give
> devs a choice of local build tooling, even if it needs to be a lein plugin.

Indeed, if there are any specific requirements that are getting in the
way of allowing contrib developers to choose for themselves I would be
glad to discuss them.

On Oct 27, 10:39 am, Chas Emerick <cemer...@snowtide.com> wrote:
> Some searching along the lines of "lein hudson" and variations didn't turn up much,
> which leads me to think that the lein community hasn't tackled the question of CI yet, and all it entails.

Not at all; there's nothing out there because no plugins or nontrivial
configuration are needed. It's just a matter of running "lein test,
install" from the Hudson job.

-Phil

[Phil Hagelberg]

On Fri, Oct 29, 2010 at 12:39 PM, Konrad Hinsen
<konrad...@fastmail.net> wrote:
> OK, that's slightly less bad than I thought, but running "mvn install" is
> still too much for me. What I want is modify a function in project B, press
> M-C-x to update it in SLIME, and use it right away in project A. That's why
> I insist on working straight from source code files while developping.

You can do this with Maven right now, but if you have any :reloads in
your require/use clauses it will revert back to the source on disk. It
sounds like some IDEs have ways of working around the problem as Chas
said, but that doesn't seem like an appropriate place for that
functionality. I was very frustrated by this when using maven as well;
it's what caused me to add the "checkout deps" feature to Leiningen.

See the FAQ section of http://github.com/technomancy/leiningen#readme

-Phil

[Chas Emerick]

On Oct 30, 2010, at 6:00 PM, Phil Hagelberg wrote:

> On Oct 27, 10:39 am, Chas Emerick <cemer...@snowtide.com> wrote:
>> Some searching along the lines of "lein hudson" and variations didn't turn up much,
>> which leads me to think that the lein community hasn't tackled the question of CI yet, and all it entails.
>
> Not at all; there's nothing out there because no plugins or nontrivial
> configuration are needed. It's just a matter of running "lein test,
> install" from the Hudson job.

Stu et al. can correct me if I'm off base re: our future objectives, but that seems insufficient. Various thoughts:

* `lein install` is equivalent to `mvn install`, and therefore targets the local repository. What would be the lein corollary to `mvn deploy`, so that we may get artifacts flowing into external repositories such as our own Nexus instance or Sonatype OSS/central?

* The Hudson provides a variety of features based on (or enabled by) maven. For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built. This has never been needed/desired given contrib's monolithic organization, but given the forthcoming modularization of contrib, cross-project dependencies will exist, and will change over time -- and it would be unfortunate if we had to fall back to manually managing such project relationships in Hudson.

* There are a variety of things that we'd like to do (such as POM validation as mentioned in the wiki, as well as OSS/central deployment) for which having a designated parent POM is either required or an obvious easy choice.

These are the sorts of issues I was thinking of re: "tackling the question of CI". My apologies for not being precise.

- Chas

[David Nolen]

On Sun, Oct 31, 2010 at 9:26 AM, Chas Emerick <ceme...@snowtide.com> wrote:

* `lein install` is equivalent to `mvn install`, and therefore targets the local repository. What would be the lein corollary to `mvn deploy`, so that we may get artifacts flowing into external repositories such as our own Nexus instance or Sonatype OSS/central?

This information lives in the POM right?

 

* The Hudson provides a variety of features based on (or enabled by) maven.  For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built.  This has never been needed/desired given contrib's monolithic organization, but given the forthcoming modularization of contrib, cross-project dependencies will exist, and will change over time -- and it would be unfortunate if we had to fall back to manually managing such project relationships in Hudson.

Isn't this determined from the POM?

David

[Chas Emerick]

Yes, everything I mentioned flows from having a pom.xml handy.  I was responding to the idea that Lein could be invoked directly from within Hudson (what I thought Phil was suggesting?) and meet the various requirements as I understand them.

- Chas

--

[Phil Hagelberg]

On Sun, Oct 31, 2010 at 6:26 AM, Chas Emerick <ceme...@snowtide.com> wrote:

Thanks for the clarification. I think it might be helpful to enumerate
all the options here just for completeness sake.

0) Have the pom.xml file checked into git and use it to drive Hudson.
This means any use of non-maven tools must be confined to developers'
machines and the projects must be buildable from the description in
the pom alone. Leiningen actually already supports setting the
<parent> element in the pom.xml files as of a few weeks ago. There may
be other things missing from leiningen-generated pom files, but this
may be the simplest thing from the perspective of the Hudson admins.

1) Add a deploy task to Leiningen so that the Hudson builds for
contrib projects using Leiningen can push straight out to Nexus upon a
successful run. I actually started this on the plane back from the
Conj, but I stalled out due to a combination of a gnarly API and
unsureness of its utility for our own situation at work. Is there much
of a difference between running deploy to a server and running install
on a server where the .m2 is served over HTTP? I understand deploy
will place the artifact in a different repository if it's a snapshot
vs releases, but is there anything else to it? The only thing getting
in the way to getting this implemented is the mismatch between Maven's
API vs maven-ant-tasks, but getting around this is just some grunt
work. I can try to get this into Leiningen 1.4 if there are contrib
authors who wish to use it.

2) Just use "lein test, install" in Hudson and simply serve up the
hudson user's .m2 directory over HTTP as the build.clojure.org
repository. I'm only listing this for completeness sake as I
understand there are objections here, but I'd like to be sure I
understand the issues. This setup has worked very well for us at work.
It seems like the main problems are a) no releases/snapshots
distinction, b) no caching of upstream repositories, (are there
upstream repositories for contrib?) and c) providing downloadable
lucene indexes from which clients may search. I can see how a) would
be a big deal for contrib especially, and c) seems pretty important
given that Clojars search facilities are not so great.

> * The Hudson provides a variety of features based on (or enabled by) maven.  For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built.

Hudson also supports specifying which upstream builds should trigger a
build. It's convenient to have it autodetected from the pom, but the
feature may easily be manually configured as well.

Anything I'm missing?

-Phil

[Meikel Brandmeyer]

Hi,

Am 30.10.2010 um 09:30 schrieb Konrad Hinsen:

> What I expect from a development-build-deployment system for a dynamic language is a single specification of a dependency graph from which it generates a classpath pointing to source directories for development AND jar files for distribution.

I tried to resist, but I can't. Have a look at gradle: http://gradle.org

It sports quite powerful multiproject builds, which seems to have solved your problem.

Sincerely
Meikel

[Chas Emerick]

On Oct 31, 2010, at 3:16 PM, Phil Hagelberg wrote:

> On Sun, Oct 31, 2010 at 6:26 AM, Chas Emerick <ceme...@snowtide.com> wrote:
>
> Thanks for the clarification. I think it might be helpful to enumerate
> all the options here just for completeness sake.
>
> 0) Have the pom.xml file checked into git and use it to drive Hudson.
> This means any use of non-maven tools must be confined to developers'
> machines and the projects must be buildable from the description in
> the pom alone. Leiningen actually already supports setting the
> <parent> element in the pom.xml files as of a few weeks ago. There may
> be other things missing from leiningen-generated pom files, but this
> may be the simplest thing from the perspective of the Hudson admins.

This is what Stu (I think it was Stu who coined the phrase) means by "let's meet at the POM"; or, alternately, that build.clojure.org generates the pom.xml file for non-maven contrib projects (which carries some of the same disadvantages as running actual non-maven builds in hudson).

Having a generated file checked into git is an unfortunate wart, but seems like a minor tradeoff to me compared to the alternatives.

Getting parents into lein-generated pom.xml files is a big step forward; I wasn't aware of that. That would very nearly bring us home.

Looking at a lein-generated pom.xml file right now, I see that src and test are designated as source directories. That's fine, but src also needs to be designated as a resource directory in order for Clojure source files to end up in jars that maven produces. I wonder if we can establish that from the parent POM somehow, but I suspect not.

> 1) Add a deploy task to Leiningen so that the Hudson builds for
> contrib projects using Leiningen can push straight out to Nexus upon a
> successful run. I actually started this on the plane back from the
> Conj, but I stalled out due to a combination of a gnarly API and
> unsureness of its utility for our own situation at work. Is there much
> of a difference between running deploy to a server and running install
> on a server where the .m2 is served over HTTP? I understand deploy
> will place the artifact in a different repository if it's a snapshot
> vs releases, but is there anything else to it? The only thing getting
> in the way to getting this implemented is the mismatch between Maven's
> API vs maven-ant-tasks, but getting around this is just some grunt
> work. I can try to get this into Leiningen 1.4 if there are contrib
> authors who wish to use it.

I know nothing about the actual mechanics of an `mvn deploy`, but I suspect that there's more to it. At the very least, a deploy operation is supposed to be transactional, and have some kind of validation of the checksums that are being provided (presumably via HTTP response codes).

I suspect that deploy capability would be of great use for lein users that want/need to interop with nexus et al., but just having a deploy operation doesn't actually help in the contrib case (as I explain below).

> 2) Just use "lein test, install" in Hudson and simply serve up the
> hudson user's .m2 directory over HTTP as the build.clojure.org
> repository. I'm only listing this for completeness sake as I
> understand there are objections here, but I'd like to be sure I
> understand the issues. This setup has worked very well for us at work.
> It seems like the main problems are a) no releases/snapshots
> distinction, b) no caching of upstream repositories, (are there
> upstream repositories for contrib?) and c) providing downloadable
> lucene indexes from which clients may search. I can see how a) would
> be a big deal for contrib especially, and c) seems pretty important
> given that Clojars search facilities are not so great.

Hosting the repo out of hudson's .m2 *works*, but hosting a proper repo is too easy to not use IMO. Your enumeration of the downsides is essentially complete; my rendition of them can be found here (http://dev.clojure.org/display/design/Common+Contrib+Build), in the repo management section.

FWIW, I suspect there will be upstream repos and non-clojure and non-contrib dependencies in use by contrib projects in the future.

Our situation is complicated by our wanting to have artifacts streaming into central (almost necessarily via Sonatype OSS: https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide). At a minimum, this requires gpg signing all artifacts, and automating the promotion to central of release artifacts staged into OSS; maven plugins exist for doing these things in a straightforward way. Compared with the option of consistently generating a suitable POM, reimplementing/scripting such things for each desirable build tool doesn't seem to have much upside.

>> * The Hudson provides a variety of features based on (or enabled by) maven. For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built.
>
> Hudson also supports specifying which upstream builds should trigger a
> build. It's convenient to have it autodetected from the pom, but the
> feature may easily be manually configured as well.

When it comes to builds and CI, (= :manual :unholy), no? ;-)

- Chas

[Stuart Halloway]

*Please* capture this kind of stuff in Confluence, and then use email only to prod people to take a look. I am participating in 30+ design and implementation discussions related to Clojure, and combing through email doesn't work at that scale.

Stu

> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.

[Chas Emerick]

FYI, contrib libraries can now deploy snapshots to Sonatype's OSS repository. Next steps – particularly, deployment of release artifacts to OSS and therefore central – and related issues are enumerated here:

http://dev.clojure.org/display/design/Common+Contrib+Build#CommonContribBuild-November3,2010%28Chas%29

Feedback is needed from Stu, Rich, and anyone else involved in the release of artifacts (Clojure core, classic contrib, new contrib libraries, etc).

For those curious about what a pom.xml in a new-contrib library looks like at this point, see the first 12 lines of this:

https://github.com/clojure/tools.nrepl/blob/master/pom.xml

Only the first 12 lines are required; the rest is either slated for removal, or for "promotion" in some form or another into the contrib parent POM.

At this point, anyone using any other build toolchain (lein, cake, gradle, buildr, rake, make, or M-x butterfly) that can get a pom.xml into git that follows the model of nREPL's pom.xml above will gain access to all the same CI and artifact deployment (including into central) as any maven-based contrib project would.

(That wiki page is in need of some serious weeding, though I hesitate to be the one to strip old content.)

- Chas

[Stuart Halloway]

Chas,

It is OK for Clojure to have a maven build (as opposed to the Ant build) in order to unify around a common approach.

However, it is important that the interactive development process *not* require maven. Rich and I need to be able to build Clojure locally, and then test it against local builds of other libraries, before pushing changes.

I have also edited the wiki page (partially). If there are other open questions let me know.

Stu

[Chas Emerick]

Stu: All of the questions/issues at the bottom of http://dev.clojure.org/display/design/Common+Contrib+Build are open AFAICT.

I'm happy enough re: leaving ant be for interactive development of Clojure. Much of the complexity in that build.xml is in its usage of the maven ant tasks and other CI-related bits. As a first step, I'd simply rip that out, leave a slim build.xml suitable for interactive use, and have a parallel pom.xml that did the real builds in hudson, etc.

That said, though there's been various discussions about Rich's pain with maven in irc (and perhaps yours as well), I'd like to take a stab at seeing if they can be alleviated entirely given a more productive mode of communication (either sitting down in person or remote pairing at some point). Getting a comprehensive view on what the pain points are would be very helpful in general; and, obviously, having to maintain one build process instead of two would be preferable, if we can make all other things be equal.

- Chas

[Stuart Halloway]

Chas: I think they are all answered now, and I am working to make sure we have someone available to work on this tomorrow. Stuart, can you help out with this tomorrow?

[Chas Emerick]

Should I set the <license> for Clojure core and all contrib projects (current and future) to EPL in a suitable parent POM?

I presume that this is exactly the intention given the CA, but thought I'd double-check.

- Chas

[Rich Hickey]

Why does the license info go in the POM? Isn't it in the binaries and
source?

Rich

[Chas Emerick]

The license being in binaries and source is a per-project governance / legal question (which we already have a standard practice around AFAIK). Having the license(s) specified in the POM is intended to facilitate downstream maven users/repositories that wish to establish automated acceptable-license policies (e.g. "we'll allow our devs to use EPL- and BSD-licensed artifacts, but nothing else", etc).

See https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide#SonatypeOSSMavenRepositoryUsageGuide-6.CentralSyncRequirement for the specific POM elements required to be able to deploy artifacts to central (these are verified by the OSS and/or repo1 nexus instances in the process of promoting from the former to the latter). See http://www.sonatype.com/people/2010/01/nexus-oss-ecosystem/ for an informal treatment of a variety of factors that are/have been considered when Sonatype established the metadata requirements that are in place now.

- Chas