Ivy & Snapshots

[Thomas Lockney]

Trying this again as a new post (to avoid adding junk to the original thread).

On Friday, May 25, 2012 6:08:25 AM UTC-7, jsuereth wrote:

Now, as for SNAPSHOTs, remember SBT uses *ivy* which means SNAPSHOTS are the wrong way of doing it.   We really should be using integrations and someone needs to figure out how to do it the right ivy way, or we're just asking for issues.

Be deleting artifacts you're *asking* for problems from users and resolution nightmares.   Let's not add to an already terrible situations.   Let's not fight Ivy if we have to use it.

I'd like to understand this better. We're currently a mixed maven & sbt shop and one of our points of major pain has been trying to get sbt to handle snapshots in a sane way. I did not realize that Ivy & snapshots just don't mix (which is how I'm reading this and fits with our experiences). Is there any good reference reading to understand this better and/or to understand what might be the right way to do this? So far I've found the Ivy docs to be essentially impenetrable for "casual" understanding (I know, we should be well past the casual level, but oddly enough, build tools are not our primary focus ;~).

Thanks,

Thomas

[eugene yokota]

For sbt plugins repo purpose, my proposal is (https://groups.google.com/forum/#!topic/simple-build-tool/wQ-fEXQTWUA).

I am really no expert in this, but here are some the info I've gathered.

As far as consuming the -SNAPSHOT jars goes, sbt/ivy handles it ok thanks to -SNAPSHOT artifacts automatically treated as "changing" artifact.

See [1] and [2].

Changes in artifacts

Some people, especially those coming from maven 2 land, like to use one special revision to handle often updated modules. In maven 2 this is called a SNAPSHOT version, and some argue that it helps save disk space to keep only one version for the high number of intermediary builds you can make whilst developing.

 

Ivy supports this kind of approach with the notion of "changing revision". A changing revision is just that: a revision for which Ivy should consider that the artifacts may change over time. To handle this, you can either specify a dependency as changing on thedependency tag, or use the changingPattern and changingMatcher attributes on your resolvers to indicate which revision or group of revisions should be considered as changing.

 

Once Ivy knows that a revision is changing, it will follow this principle to avoid checking your repository too often: if the module metadata has not changed, it will considered the whole module (including artifacts) as not changed. Even if the module descriptor file has changed, it will check the publication data of the module to see if this is a new publication of the same revision or not. Then if the publication date has changed, it will check the artifacts' last modified timestamps, and download them accordingly.

The only thing I've ran into is the fact that Ivy cache is sticky to where it last got the snapshot from,

so it may not go after the lastest in the resolvers if you have used a stale one published locally, vice versa.

That's why I use sbt-dirty-money for usually.

Publishing Maven-style snapshot is more tricky. Here's from [Maven reference][3]:

SNAPSHOT Versions

Maven versions can contain a string literal to signify that a project is currently under active development. If a version contains the string “-SNAPSHOT,” then Maven will expand this token to a date and time value converted to UTC (Coordinated Universal Time) when you install or release this component. For example, if your project has a version of “1.0-SNAPSHOT” and you deploy this project’s artifacts to a Maven repository, Maven would expand this version to “1.0-20080207-230803-1” if you were to deploy a release at 11:08 PM on February 7th, 2008 UTC. In other words, when you deploy a snapshot, you are not making a release of a software component; you are releasing a snapshot of a component at a specific time.

I think this is describing the server-side behavior of Maven repo. In pom.xml there's also [uniqueVersion][4] setting.

uniqueVersion: The unique version takes a true or false value to denote whether artifacts deployed to this repository should get a uniquely generated version number, or use the version number defined as part of the address.

I don't think there's equivalent facility in sbt/ivy if we want to avoid overwriting jars. Instead what we have is [dynamic revision][5]:

Fixed and dynamic revisions

The revision can be given as a fixed one (1.5.2, for instance) or as a latest (or dynamic) one. Several possibilities for dynamic revisions are implemented in Ivy:

• latest.integration

selects the latest revision of the dependency module.

• latest.[any status] since 1.4

selects the latest revision of the dependency module with at least the specified status. 

-eugene

  [1]: http://ant.apache.org/ivy/history/latest-milestone/concept.html#change

  [2]: https://github.com/harrah/xsbt/blob/e8b8be754713be2c2200653ecb927226050c9ba9/launch/Update.scala#L275-284

  [3]: http://www.sonatype.com/books/mvnref-book/reference/pom-relationships-sect-pom-syntax.html#pom-reationships-sect-versions

  [4]: http://maven.apache.org/pom.html

  [5]: http://ant.apache.org/ivy/history/latest-milestone/ivyfile/dependency.html#revision

[Thomas Lockney]

Eugene,

thanks for the detailed reply. Some of this confirms what I was experiencing, but I'll take some time to digest this and see if there's anything relatively simple I can do to make things easier on us. Unfortunately, I'm hearing more and more the clamor to move the sbt projects back to Maven. I'd prefer not to go down that route, if possible, but this seems to be causing us a lot of ongoing pain. 

[Thomas Lockney]

Ok, so I've finally had the time to read through this and try to get a sense for how we can improve our current situation. I've also tried a few local experiments. Unfortunately, for the interaction of SBT/Ivy and Maven snapshot projects, it looks like there's really no solution. At least not one that I can see. I've tried using changing(), but that apparently doesn't affect how Ivy resolves from resources from the local Maven repository. So, if we continue to try and use snapshot version for our Maven projects, we have to manually remove the versions that previous sbt runs have pulled into the cache.

Anyone have any additional ideas?

[Peter Niederwieser]

Just to let you know, problems with Ivy and Maven snapshot dependencies were one of the reasons why Gradle eventually replaced Ivy with its own dependency management code. It was a big task, but brought us a lot of goodness.

Cheers,

Peter

--

Peter Niederwieser

Principal Engineer, Gradleware 

http://gradleware.com

Creator, Spock Framework 

http://spockframework.org

Twitter: @pniederw

[Thomas Lockney]

On Thu, Jun 14, 2012 at 2:44 PM, Peter Niederwieser <pnie...@gmail.com> wrote:
> Just to let you know, problems with Ivy and Maven snapshot dependencies were
> one of the reasons why Gradle eventually replaced Ivy with its own
> dependency management code. It was a big task, but brought us a lot of
> goodness.

Are there any truth to the rumors you guys built this in a way that it
could be used by other build tools? You know, should certain other
build tools be so inclined.

[Peter Niederwieser]

It's possible today but not convenient. Over time we plan to make our dependency management code more independent from the rest of the Gradle codebase.

Cheers,

Peter