Help greatly wanted. I would appreciate discussion and feedback, but even more than that would like for somebody to take ownership of this, leading the discussion and carrying it all the way through to implementation.
Thanks,
Stu
Stu, your bullet point of "Treat the pom.xml as the data of record"
strikes me as the right direction (and I'm sure no one is surprised by
my saying that), but the details are sticky. I would assume you mean
that we'll simply be using the results of e.g. `lein pom` when
deploying, not that we use it to drive the build.
To what degree would you like discussion here vs. on the wiki?
- Chas
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com
I would like for the *results* of the discussion to be on the wiki, so we keep a colocated record of design tradeoffs. Once this particular task has an owner, that person can make it so. :-)
Stu
> I would like for the *results* of the discussion to be on the wiki, so we keep a colocated record of design tradeoffs. Once this particular task has an owner, that person can make it so. :-)
My thoughts are thus:
== Project-level build tooling
The simplest option from an administrative standpoint is to declare that all contrib builds are to use maven. This makes managing hudson and the deployments to maven repos a no-brainer.
FWIW, it would be trivial to define a clojure contrib-specific parent pom that would drop the required XML for any given actual project to something like 8 lines (defining artifactId, version, name, & description); only if the project had external dependencies would additional configuration be required. This would be advantageous for administrative purposes as well, as we could define project guidelines/requirements as part of that parent POM.
Beyond that, we can provide support for lein (for example), but that will require either setting up each lein-driven project in hudson with the necessary shell execs to "run the build" (whatever that implies given the usage of lein plugins, etc), get a pom (just `lein pom`), and then use hudson's maven install to deploy the resulting artifact to the target repository using that pom. Alternatively, a proper lein plugin for hudson would be nice, though a quick search doesn't seem to turn up anything.
Aside from technical considerations, I think we have to ask ourselves why we're supporting a diverse set of tools, and if we are, why we're supporting what we're supporting. If we're supporting lein because it's the most widely used right now (is it?), does that mean that we'll support whatever becomes more popular than it (e.g. cake or gradle or buildr or whatever) some time from now? Maybe that'll never happen, but it's worth thinking about.
A final question would be about plugins obtained from clojars (which is the case for all lein and cake plugins AFAIK). These would be running with whatever privs the hudson user has. I'm no security expert, but that sets off some bells for me...not just for the local build system, but also about what a bad actor might be able to inject into the artifacts of e.g. the main clojure build.
== Repositories
There's no good reason to not deploy artifacts to a proper repository. AFAIK, the only advantage to e.g. clojure's scp deployment to a filesystem location is that it helps to avoid running a repository app.
If we deployed project artifacts to a repository like Nexus (free version will do, though I suspect sonatype would set us up with a free "Pro" license), then we would gain:
* a passable artifact discovery UI (and various web services that would make building a more attractive frontend ostensibly straightforward)
* ability to proxy third-party artifacts (important insofar as contrib projects may have external dependencies)
* ability to provide artifact indexes to downstream proxies (aids in visibility of artifacts in users' repos as well as in IDEs)
* all sorts of user management facilities that would allow for the delegation of management activities (e.g. no need to chase down the one or two guys that have access to the build box if there's a problem, when there's a reasonably-sized group of trusted folk that have various admin rights on the nexus).
If that's convincing, then there's the further question of whether we'd like to deploy artifacts to Sonatype's OSS repo (which provides for easy syncing of releases to central):
http://nexus.sonatype.org/oss-repository-hosting.html
…rather than to our own nexus install. I think one will be necessary eventually anyway – assuming we want to provide a saner community repository – but that's a separate topic.
If we decide that getting artifacts into central is important (I think it's hard to argue that it's not), then that makes the question of supporting lein-driven build processes a difficult one. The entire OSS deployment toolchain is maven-based. There may be a reasonable way to connect the dots, but someone will have to do some serious digging.
== Aggregate artifacts
Packaging multiple libs into easy-to-consume distributions is easily accomplished via the assembly or shade plugins.
== Release process
If a project is using maven, then the path is defined: release plugin (slightly touchy to get set up, but then works well), driven by the hudson m2 release plugin (http://wiki.hudson-ci.org/display/HUDSON/M2+Release+Plugin), which provides a nifty "cut a release" action on a per-project basis. This runs a build, deploys the artifacts, tags the current SCM rev appropriately, and then pushes an incremented version number (in an automagic way, or as specified) back to git.
Projects not using maven would likely want to use the "regular" release plugin (http://wiki.hudson-ci.org/display/HUDSON/Release+Plugin), though we'd have to orchestrate the version-number-increment-and-push bit ourselves. I'm not sure how people will like their project.clj files being modified automatically (i.e. with comments lost, etc, presuming we're using the reader to slurp them).
I'm not sure if the current wiki page is talking about automating releases using commit messages or e.g. version.properties as found in the clojure project. Both options seem unnecessarily baroque IMO, especially if either of the release plugins for hudson above meet requirements.
Hopefully the above is a helpful perspective, though perhaps not a surprising one insofar as people know my history and leanings.
Cheers,
- Chas
I'm no lover of maven, but all current solutions in this space
seem amount to: some boilerplate, a few "variables" dropped in
per contrib repo, and then generally ignoring the build spec file
until some version needs to be bumped or something. That is to
say, from this particular contrib author's point of view, I don't
really care very much as long as builds can be made to work
consistently. I think the build coordinator's job is
sufficiently undesirable as it is that priority should be given
to making that role as painless as possible.
Monolithic contrib is currently (a): maven is the only way to
build it. We're changing a lot of things in moving to one-repo
(and bug tracker and wiki, etc.) per contrib lib -- do we want to
change more than we have to right now? Would adding support for
non-maven build tools be any easier to do now than it would be
later?
--Chouser
> I've invested a lot of time in learning Maven, a little in Nexus and
> Hudson. I am willing to help implement.
Just in case my prior message didn't make it obvious, the above applies to me as well.
- Chas
--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.
To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> My motivation for recommending standard tools is more social than
> technical, for one very specific reason:
...
You are perfectly right, but not at all in contradiction to what Chas said. The real question is: what are appropriate "standard tools" for the Clojure community? Is it Maven and friends? Leiningen? Something else yet? Can we already speak of a "standard"?
Here's my own point of view: I don't know much about Maven, and I don't like much in what I have seen until now. If contributing to the new contrib requires more Maven knowledge than installing the current contrib, then I don't expect to contribute anything any time soon. I have much less time for Clojure hacking that I'd like to have, and I certainly don't want to spend that scarce time fighting against Maven. Leiningen is a different story. I don't use it much, but it looks perfectly clear to me and I wouldn't mind working more with it. Ant would be acceptable as well.
Konrad.
Here's my own point of view: I don't know much about Maven, and I don't like much in what I have seen until now. If contributing to the new contrib requires more Maven knowledge than installing the current contrib, then I don't expect to contribute anything any time soon. I have much less time for Clojure hacking that I'd like to have, and I certainly don't want to spend that scarce time fighting against Maven. Leiningen is a different story. I don't use it much, but it looks perfectly clear to me and I wouldn't mind working more with it. Ant would be acceptable as well.
Konrad.
Worth pointing out is that lein is built on top of Maven…so, you're using it already. Doing what you're doing now with the mvn frontend instead of the lein frontend should be a pretty painless transistion unless there's something about people's workflow that I don't understand.
The result of `lein pom`, while suitable for deployment, cannot be used to drive an actual build. This implies that a pom.xml is the project-description-of-record. One could back that into a project.clj, but that doesn't help with minimizing the work involved.- Chas
I'm not sure what 'contrib build information' means exactly. Lein plugins are the big wildcard, along with whatever additional bits we end up needing to support the full CI lifecycle (at a minimum, being able to define a parent POM, though we may discover other requirements as we go along).- Chas
>> For those of of us that don't really understand Maven, what more is required beyond generating a pom.xml?
>
> The development process is fundamentally the same:
>
> - install maven
> - have a pom handy (sample below)
> - use the goals defined by the clojure-maven-plugin: http://github.com/talios/clojure-maven-plugin
...
My gripe with Maven is that it is oriented towards deployment rather than development. What I would like to have is a build system that lets me do development with development-oriented tools, and then once I decide to do a release produce all the necessary files to turn my development source code base into a series of libraries with proper version numbers and dependencies betwee them.
During development, my priority is working straight from source code to make sure that the code I run is exactly the code I see in the editor. If my current understanding of Maven is right, it does not support this at all. Every minor change in any library forces me to a "release" first (if only for personal use) and update the dependencies in all other libraries. That's a major pain.
Konrad.
Thoughtful and cogent critiques like this are valuable and
helpful. Thanks!
--Chouser
I'd hazard to say that your understanding of Maven is incorrect. If one of your projects' dependencies puts out a new release, *you* don't have to release in order to use it. Just change the version number (either in your pom.xml or project.clj), and carry on.
Perhaps I'm misunderstanding the problem?
- Chas
> I'd hazard to say that your understanding of Maven is incorrect. If
> one of your projects' dependencies puts out a new release, *you*
> don't have to release in order to use it. Just change the version
> number (either in your pom.xml or project.clj), and carry on.
>
> Perhaps I'm misunderstanding the problem?
I think so. My typical situation is that I work on several libraries
in parallel. They are separate libraries because most users will use
only some of them, but there are lots of dependencies between them.
It's a bit like current contrib.
The typical situation I was referring to is that I add some new
feature to application A, library 1, and library 2. I edit source
files in all of those, and want to work from those source files during
the process. Then one day I decide that the new feature set is stable
enough and make releases for some or all of library 1, library 2, and
application A.
My understanding of Maven (which may be incorrect) is that in order to
be sure that application A uses the very latest changes to library 1,
I have to make a release of library 1 and indicate that new version in
the dependency list of application A. Otherwise Maven will stick to
the previous release that it has somewhere in its repository. If that
is true, I'll end up spending a lot of time producing releases that
noone but me will ever see - or I don't use Maven.
At the moment, I have a setup that puts all the source code of my
active libraries on the classpath, so I always work from source. When
I want to make a release, I use "lein pom" and "lein jar" and push the
results to clojars. The problem with that is of course that I never
use those files myself, so if something is wrong with the jars or the
poms (say due to a bug in my project.clj), I'd never notice. That's
why I'd prefer an integrate build tool for development and deployment,
if such a beast exists in the Java world.
Konrad.
Stu
To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> Phil described a leiningen feature that works around this issue,
> basically making it easy to say, locally, "use these jars
> preferentially." If maven doesn't make that easy enough, then we
> could add a shim that does.
If I could get "source code directories" in addition to "jars", that
would be just fine!
Konrad.
On Oct 29, 12:53 am, Konrad Hinsen <konrad.hin...@fastmail.net> wrote:If you you are working with Maven "-SNAPSHOT" versions, you do not
> During development, my priority is working straight from source code to make sure that the code I run is exactly the code I see in the editor. If my current understanding of Maven is right, it does not support this at all. Every minor change in any library forces me to a "release" first (if only for personal use) and update the dependencies in all other libraries. That's a major pain.
need to "release" anything.
Say you're working on projects A and B simultaneously. Your POM for
Project A will have a dependency on a "-SNAPSHOT" version of Project
B. You make a change in B and run "mvn install". That will compile,
test, package, and install B in your local Maven repository (~/.m2/
repository). Running any mvn goal in Project A will automatically use
the most most recent -SNAPSHOT of Project B.
Agreed, and what we come up with will support this.
> I know that Lein's model probably doesn't scale for large projects, but considering the relatively modest scope of most of the contrib libraries at this point, do we need to force contrib authors to adopt such a build tool?
If we can get compliant poms out of lein then that is fine. If not let's make it so.
> It's seems more effective to allow people to use whatever they want as long as they can provide a compliant POM. People who need more complex builds will have to bite the bullet and perhaps learn some Maven it sounds like, but it so far it doesn't seem like it needs to be requirement.
Agreed. The whole point of "meeting at the pom" is that we don't have to care where your pom came from.
It would be an abject failure, and non-Clojurish, if we had to have an API (as opposed to a data format) here.
Stu
> When you're working on two projects simultaneously you don't want to have to run maven install or even lein deps for small changes to B. You don't want to goal, test, package, or even install. You're *developing*- I think this is unnecessarily tedious for the developer (it's why Lein supports sym-linking other libs instead).
>
> Maven seems brings with it the C++/Java compile-wait frame of mind to Clojure programming. Lein/Cake try their best to keep that at bay. I also suspect this is why Rich Hickey prefers the simplicity (naivete even) of Ant.
There's no doubt that Maven is from a different world than Clojure. This workflow issue is resolved (for me anyway) by excellent Maven tooling in any of the major IDEs, where dependencies are tracked without any intermediate build processes at all; i.e. I have gone days without running a single maven build across any of a half-dozen interrelated projects, all of which I'm changing and fiddling with in various REPLs.
> I know that Lein's model probably doesn't scale for large projects, but considering the relatively modest scope of most of the contrib libraries at this point, do we need to force contrib authors to adopt such a build tool?
There's far more moving pieces than just the developer's local workflow to consider when talking about contrib build tooling. Stu's (probably) right about "let's meet at the POM", but I want to emphasize that one needs to think about more than what goes on in a single contributor's environment in this context.
- Chas
Surely you can't just reload a namespace in a running REPL
though? You click some magic button that builds and installs
some tree of dependant jars, and launches a new REPL with the
appropriate classpath or something?
--Chouser
>> There's no doubt that Maven is from a different world than Clojure. This workflow issue is resolved (for me anyway) by excellent Maven tooling in any of the major IDEs, where dependencies are tracked without any intermediate build processes at all; i.e. I have gone days without running a single maven build across any of a half-dozen interrelated projects, all of which I'm changing and fiddling with in various REPLs.
>
> Surely you can't just reload a namespace in a running REPL
> though? You click some magic button that builds and installs
> some tree of dependant jars, and launches a new REPL with the
> appropriate classpath or something?
Clearly, I need to do a screencast or something. ;-)
- Chas
> On Oct 29, 12:53 am, Konrad Hinsen <konrad.hin...@fastmail.net> wrote:
>> During development, my priority is working straight from source
>> code to make sure that the code I run is exactly the code I see in
>> the editor. If my current understanding of Maven is right, it does
>> not support this at all. Every minor change in any library forces
>> me to a "release" first (if only for personal use) and update the
>> dependencies in all other libraries. That's a major pain.
>
> If you you are working with Maven "-SNAPSHOT" versions, you do not
> need to "release" anything.
>
> Say you're working on projects A and B simultaneously. Your POM for
> Project A will have a dependency on a "-SNAPSHOT" version of Project
> B. You make a change in B and run "mvn install". That will compile,
> test, package, and install B in your local Maven repository (~/.m2/
> repository). Running any mvn goal in Project A will automatically use
> the most most recent -SNAPSHOT of Project B.
OK, that's slightly less bad than I thought, but running "mvn install"
is still too much for me. What I want is modify a function in project
B, press M-C-x to update it in SLIME, and use it right away in project
A. That's why I insist on working straight from source code files
while developping.
Konrad.
What would need to change in the `lein pom`-generated pom file (besides adding the ability to specify a parent pom) in order to make it compliant in this context? Please excuse my Maven (and cake) ignorance.
I can't speak for Phil, but it seems like a win to me if it's possible give devs a choice of local build tooling, even if it needs to be a lein plugin.
Would something like that obviate the question marks around the new process? If so, maybe we could ping the lein/cake mailing lists and start working on plugins/tasks to make the desired poms..
Colin
On Oct 29, 2010 3:04 PM, "Chas Emerick" <ceme...@snowtide.com> wrote:
On Oct 29, 2010, at 2:16 PM, David Nolen wrote:
> When you're working on two projects simultaneous...
There's no doubt that Maven is from a different world than Clojure. This workflow issue is resolved (for me anyway) by excellent Maven tooling in any of the major IDEs, where dependencies are tracked without any intermediate build processes at all; i.e. I have gone days without running a single maven build across any of a half-dozen interrelated projects, all of which I'm changing and fiddling with in various REPLs.
> I know that Lein's model probably doesn't scale for large projects, but considering the relativel...
There's far more moving pieces than just the developer's local workflow to consider when talking about contrib build tooling. Stu's (probably) right about "let's meet at the POM", but I want to emphasize that one needs to think about more than what goes on in a single contributor's environment in this context.
- Chas
--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
...
>> I know that Lein's model probably doesn't scale for large projects, but considering the relatively modest scope of most of the contrib libraries at this point, do we need to force contrib authors to adopt such a build tool?
>
> If we can get compliant poms out of lein then that is fine. If not let's make it so.
For my money, it seems like we should rely upon pmaven to emit our pom.xml serializations, simply because we'll likely want/need to fiddle with the POM along the way in various ways in order to get what we want. This will work as long as lein, cake, and pmaven (mostly) share a compatible POM serialization, but none of the leads of those projects have participated in this thread so far AFAICT. Cake and lein have already diverged from the POM in various ways; will they diverge more?
Along these lines, my concerns about lein/cake plugins remain, both in terms of linking them up into a maven build process (after pom-ifying the project.clj) as well as running them in a shared/privileged environment and their provenance (though this could be resolved through clojars v2).
Then there's wildcards like `lein multitest`. Something like it should probably be strongly recommended of all projects eventually, with the best corollary being use of the invoker plugin in maven (with a punt position of multiple profiles and multi-stage builds for all such projects). Trying to determine where the cutoff is between a project that can use lein/cake and one that really just has to use maven may get tricky.
This is all to say that it seems that "compliant pom" is a loaded concept.
- Chas
- Chas
--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
I guess not, if I continue to work in two parallel worlds: one for
development, one for publishing releases. That's in fact how I work
now with contrib. But I don't consider this a good setup, since I
never work with the setup that ultimately gets published.
What I expect from a development-build-deployment system for a dynamic
language is a single specification of a dependency graph from which it
generates a classpath pointing to source directories for development
AND jar files for distribution.
Konrad.
Wouldn't it be possible to add to clojure-maven-plugin the
ability to take a list of locally-available projects (path to the
pom.xml in a locally checked-out dir) from which to find source
paths to add to the classpath? This would add to maven itself
roughly the ability that I understand some of the IDEs to have
already. It would allow the project's own pom to remain the
standard of how to build, but would allow developers to easily
use locally modified versions of any of the project's
dependencies.
Is this a bad idea for some reason?
--Chouser
Thanks for bringing this up, Colin. I've been traveling and only
following this conversation loosely, but it seems like the concerns
around non-maven tools have not been clearly laid out other than the
fear of malicious plugins. It should be noted that Leiningen plugins
are almost always used for convenience during development time and
only rarely required to build jars. Now that Leiningen contains a java
compilation task, the only exception I'm aware of is native-deps,
which I don't think any contribs need.
> I can't speak for Phil, but it seems like a win to me if it's possible give
> devs a choice of local build tooling, even if it needs to be a lein plugin.
Indeed, if there are any specific requirements that are getting in the
way of allowing contrib developers to choose for themselves I would be
glad to discuss them.
On Oct 27, 10:39 am, Chas Emerick <cemer...@snowtide.com> wrote:
> Some searching along the lines of "lein hudson" and variations didn't turn up much,
> which leads me to think that the lein community hasn't tackled the question of CI yet, and all it entails.
Not at all; there's nothing out there because no plugins or nontrivial
configuration are needed. It's just a matter of running "lein test,
install" from the Hudson job.
-Phil
You can do this with Maven right now, but if you have any :reloads in
your require/use clauses it will revert back to the source on disk. It
sounds like some IDEs have ways of working around the problem as Chas
said, but that doesn't seem like an appropriate place for that
functionality. I was very frustrated by this when using maven as well;
it's what caused me to add the "checkout deps" feature to Leiningen.
See the FAQ section of http://github.com/technomancy/leiningen#readme
-Phil
> On Oct 27, 10:39 am, Chas Emerick <cemer...@snowtide.com> wrote:
>> Some searching along the lines of "lein hudson" and variations didn't turn up much,
>> which leads me to think that the lein community hasn't tackled the question of CI yet, and all it entails.
>
> Not at all; there's nothing out there because no plugins or nontrivial
> configuration are needed. It's just a matter of running "lein test,
> install" from the Hudson job.
Stu et al. can correct me if I'm off base re: our future objectives, but that seems insufficient. Various thoughts:
* `lein install` is equivalent to `mvn install`, and therefore targets the local repository. What would be the lein corollary to `mvn deploy`, so that we may get artifacts flowing into external repositories such as our own Nexus instance or Sonatype OSS/central?
* The Hudson provides a variety of features based on (or enabled by) maven. For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built. This has never been needed/desired given contrib's monolithic organization, but given the forthcoming modularization of contrib, cross-project dependencies will exist, and will change over time -- and it would be unfortunate if we had to fall back to manually managing such project relationships in Hudson.
* There are a variety of things that we'd like to do (such as POM validation as mentioned in the wiki, as well as OSS/central deployment) for which having a designated parent POM is either required or an obvious easy choice.
These are the sorts of issues I was thinking of re: "tackling the question of CI". My apologies for not being precise.
- Chas
* `lein install` is equivalent to `mvn install`, and therefore targets the local repository. What would be the lein corollary to `mvn deploy`, so that we may get artifacts flowing into external repositories such as our own Nexus instance or Sonatype OSS/central?
* The Hudson provides a variety of features based on (or enabled by) maven. For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built. This has never been needed/desired given contrib's monolithic organization, but given the forthcoming modularization of contrib, cross-project dependencies will exist, and will change over time -- and it would be unfortunate if we had to fall back to manually managing such project relationships in Hudson.
Thanks for the clarification. I think it might be helpful to enumerate
all the options here just for completeness sake.
0) Have the pom.xml file checked into git and use it to drive Hudson.
This means any use of non-maven tools must be confined to developers'
machines and the projects must be buildable from the description in
the pom alone. Leiningen actually already supports setting the
<parent> element in the pom.xml files as of a few weeks ago. There may
be other things missing from leiningen-generated pom files, but this
may be the simplest thing from the perspective of the Hudson admins.
1) Add a deploy task to Leiningen so that the Hudson builds for
contrib projects using Leiningen can push straight out to Nexus upon a
successful run. I actually started this on the plane back from the
Conj, but I stalled out due to a combination of a gnarly API and
unsureness of its utility for our own situation at work. Is there much
of a difference between running deploy to a server and running install
on a server where the .m2 is served over HTTP? I understand deploy
will place the artifact in a different repository if it's a snapshot
vs releases, but is there anything else to it? The only thing getting
in the way to getting this implemented is the mismatch between Maven's
API vs maven-ant-tasks, but getting around this is just some grunt
work. I can try to get this into Leiningen 1.4 if there are contrib
authors who wish to use it.
2) Just use "lein test, install" in Hudson and simply serve up the
hudson user's .m2 directory over HTTP as the build.clojure.org
repository. I'm only listing this for completeness sake as I
understand there are objections here, but I'd like to be sure I
understand the issues. This setup has worked very well for us at work.
It seems like the main problems are a) no releases/snapshots
distinction, b) no caching of upstream repositories, (are there
upstream repositories for contrib?) and c) providing downloadable
lucene indexes from which clients may search. I can see how a) would
be a big deal for contrib especially, and c) seems pretty important
given that Clojars search facilities are not so great.
> * The Hudson provides a variety of features based on (or enabled by) maven. For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built.
Hudson also supports specifying which upstream builds should trigger a
build. It's convenient to have it autodetected from the pom, but the
feature may easily be manually configured as well.
Anything I'm missing?
-Phil
Am 30.10.2010 um 09:30 schrieb Konrad Hinsen:
> What I expect from a development-build-deployment system for a dynamic language is a single specification of a dependency graph from which it generates a classpath pointing to source directories for development AND jar files for distribution.
I tried to resist, but I can't. Have a look at gradle: http://gradle.org
It sports quite powerful multiproject builds, which seems to have solved your problem.
Sincerely
Meikel
> On Sun, Oct 31, 2010 at 6:26 AM, Chas Emerick <ceme...@snowtide.com> wrote:
>
> Thanks for the clarification. I think it might be helpful to enumerate
> all the options here just for completeness sake.
>
> 0) Have the pom.xml file checked into git and use it to drive Hudson.
> This means any use of non-maven tools must be confined to developers'
> machines and the projects must be buildable from the description in
> the pom alone. Leiningen actually already supports setting the
> <parent> element in the pom.xml files as of a few weeks ago. There may
> be other things missing from leiningen-generated pom files, but this
> may be the simplest thing from the perspective of the Hudson admins.
This is what Stu (I think it was Stu who coined the phrase) means by "let's meet at the POM"; or, alternately, that build.clojure.org generates the pom.xml file for non-maven contrib projects (which carries some of the same disadvantages as running actual non-maven builds in hudson).
Having a generated file checked into git is an unfortunate wart, but seems like a minor tradeoff to me compared to the alternatives.
Getting parents into lein-generated pom.xml files is a big step forward; I wasn't aware of that. That would very nearly bring us home.
Looking at a lein-generated pom.xml file right now, I see that src and test are designated as source directories. That's fine, but src also needs to be designated as a resource directory in order for Clojure source files to end up in jars that maven produces. I wonder if we can establish that from the parent POM somehow, but I suspect not.
> 1) Add a deploy task to Leiningen so that the Hudson builds for
> contrib projects using Leiningen can push straight out to Nexus upon a
> successful run. I actually started this on the plane back from the
> Conj, but I stalled out due to a combination of a gnarly API and
> unsureness of its utility for our own situation at work. Is there much
> of a difference between running deploy to a server and running install
> on a server where the .m2 is served over HTTP? I understand deploy
> will place the artifact in a different repository if it's a snapshot
> vs releases, but is there anything else to it? The only thing getting
> in the way to getting this implemented is the mismatch between Maven's
> API vs maven-ant-tasks, but getting around this is just some grunt
> work. I can try to get this into Leiningen 1.4 if there are contrib
> authors who wish to use it.
I know nothing about the actual mechanics of an `mvn deploy`, but I suspect that there's more to it. At the very least, a deploy operation is supposed to be transactional, and have some kind of validation of the checksums that are being provided (presumably via HTTP response codes).
I suspect that deploy capability would be of great use for lein users that want/need to interop with nexus et al., but just having a deploy operation doesn't actually help in the contrib case (as I explain below).
> 2) Just use "lein test, install" in Hudson and simply serve up the
> hudson user's .m2 directory over HTTP as the build.clojure.org
> repository. I'm only listing this for completeness sake as I
> understand there are objections here, but I'd like to be sure I
> understand the issues. This setup has worked very well for us at work.
> It seems like the main problems are a) no releases/snapshots
> distinction, b) no caching of upstream repositories, (are there
> upstream repositories for contrib?) and c) providing downloadable
> lucene indexes from which clients may search. I can see how a) would
> be a big deal for contrib especially, and c) seems pretty important
> given that Clojars search facilities are not so great.
Hosting the repo out of hudson's .m2 *works*, but hosting a proper repo is too easy to not use IMO. Your enumeration of the downsides is essentially complete; my rendition of them can be found here (http://dev.clojure.org/display/design/Common+Contrib+Build), in the repo management section.
FWIW, I suspect there will be upstream repos and non-clojure and non-contrib dependencies in use by contrib projects in the future.
Our situation is complicated by our wanting to have artifacts streaming into central (almost necessarily via Sonatype OSS: https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide). At a minimum, this requires gpg signing all artifacts, and automating the promotion to central of release artifacts staged into OSS; maven plugins exist for doing these things in a straightforward way. Compared with the option of consistently generating a suitable POM, reimplementing/scripting such things for each desirable build tool doesn't seem to have much upside.
>> * The Hudson provides a variety of features based on (or enabled by) maven. For example, Hudson can trigger builds of projects with a SNAPSHOT version anytime a SNAPSHOT-versioned dependency is built.
>
> Hudson also supports specifying which upstream builds should trigger a
> build. It's convenient to have it autodetected from the pom, but the
> feature may easily be manually configured as well.
When it comes to builds and CI, (= :manual :unholy), no? ;-)
- Chas
Stu
> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
Feedback is needed from Stu, Rich, and anyone else involved in the release of artifacts (Clojure core, classic contrib, new contrib libraries, etc).
For those curious about what a pom.xml in a new-contrib library looks like at this point, see the first 12 lines of this:
https://github.com/clojure/tools.nrepl/blob/master/pom.xml
Only the first 12 lines are required; the rest is either slated for removal, or for "promotion" in some form or another into the contrib parent POM.
At this point, anyone using any other build toolchain (lein, cake, gradle, buildr, rake, make, or M-x butterfly) that can get a pom.xml into git that follows the model of nREPL's pom.xml above will gain access to all the same CI and artifact deployment (including into central) as any maven-based contrib project would.
(That wiki page is in need of some serious weeding, though I hesitate to be the one to strip old content.)
- Chas
It is OK for Clojure to have a maven build (as opposed to the Ant build) in order to unify around a common approach.
However, it is important that the interactive development process *not* require maven. Rich and I need to be able to build Clojure locally, and then test it against local builds of other libraries, before pushing changes.
I have also edited the wiki page (partially). If there are other open questions let me know.
Stu
I'm happy enough re: leaving ant be for interactive development of Clojure. Much of the complexity in that build.xml is in its usage of the maven ant tasks and other CI-related bits. As a first step, I'd simply rip that out, leave a slim build.xml suitable for interactive use, and have a parallel pom.xml that did the real builds in hudson, etc.
That said, though there's been various discussions about Rich's pain with maven in irc (and perhaps yours as well), I'd like to take a stab at seeing if they can be alleviated entirely given a more productive mode of communication (either sitting down in person or remote pairing at some point). Getting a comprehensive view on what the pain points are would be very helpful in general; and, obviously, having to maintain one build process instead of two would be preferable, if we can make all other things be equal.
- Chas
I presume that this is exactly the intention given the CA, but thought I'd double-check.
- Chas
Rich
See https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide#SonatypeOSSMavenRepositoryUsageGuide-6.CentralSyncRequirement for the specific POM elements required to be able to deploy artifacts to central (these are verified by the OSS and/or repo1 nexus instances in the process of promoting from the former to the latter). See http://www.sonatype.com/people/2010/01/nexus-oss-ecosystem/ for an informal treatment of a variety of factors that are/have been considered when Sonatype established the metadata requirements that are in place now.
- Chas