publish-local not working

[John Sullivan]

Good morning! My teammates and I have been having a heck of a time getting sbt publish-local to do what we want. The command does post artifacts to ~HOME/.ivy2/local/, but then projects with dependencies do not pick up the new versions of the artifacts that are pushed there. If I issue an sbt "last" command after an sbt "update" command, and scroll up, I see the following:

https://gist.github.com/sullivan-/5576638

As you can see, SBT does initially find the foo artifact in $HOME/.ivy2/local, but then barfs when trying to read an XML file for the artifact from $HOME/.ivy2/cache. After barfing, it decides to skip the version in $HOME/.ivy2/local/, and falls back on the version in cache.

I am working with a totally barebones example here, two very simple projects, foo and bar, with bar depending on foo. In order to produce this error, I follow these simple steps:

- run SBT publish command for project foo

- run SBT compile command for project bar

- run SBT publish-local command for project foo

- run SBT update command for project bar - does not pick up the local foo artifact

My build.sbt files are as follows:

https://gist.github.com/sullivan-/5576713

https://gist.github.com/sullivan-/5576719

There is no Play, no plugins, and no GPG stuff in my builds.

Does anyone have any idea what's going wrong?

Thanks, John

[John Sullivan]

Could it be because our NEXUS repository is so old? We're at "Sonatype Nexus™ Open Source Edition, Version: 1.5.0". I know, that's *really* old, but I have no indication that old means broken in this case.

[John Sullivan]

I just confirmed that the problem persists even after I went through the whole rigamarole of upgrading to "Sonatype Nexus™ 2.4.0-09".

[Josh Suereth]

The best thing I can recomend is to use unique version numbers, and bump bar when foo has an internal "release" change.  You can even automate this bump based on a jenkins build.   I think you'll have a far smother experience all around in the team if instead of just "publish to nexus a -SNAPSHOT" you think: "Publish an integration release", and automate that.

As far as getting sbt to recognize/use snapshots, it's definitely not the best way to go.   You want to look into using "latest.integration" as your version instead of -SNAPSHOT.  I think this should work better with publish-local, but maybe Mark can chime in with more details.   I'd also recommend reading about Ivy's version usage:

http://ant.apache.org/ivy/history/2.3.0-rc1/ivyfile/dependency.html#revision

IIRC -> by default publishLocal sets the "integration" status for an ivy module, so "latest.integration" would always grab the latest locally published module.

On Tue, May 14, 2013 at 2:01 PM, John Sullivan <sull...@broadinstitute.org> wrote:

This is a real problem for us. We have multiple projects with dependencies between them, so we need to be able to publish from one project where it can be picked up by another project. But I can not be publishing to the team's repository for work-in-progress, because I am making breaking changes. I've seen other people posting with similar problems, but I haven't found any resolution.

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
To post to this group, send email to simple-b...@googlegroups.com.
Visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[John Sullivan]

Sorry for being a little gruff yesterday. I got myself into too much of a rush, and then couldn't get past the problem fast enough.

I have a workaround for the time being, and a deadline, but I'd like to investigate this a little further when I have the chance.

Best, John

[John Sullivan]

Confirmed that the problem does not go away when using Apache Archiva 1.3.6

[John Sullivan]

Hi Fluffy,

Thanks for your response, I appreciate the feedback.

The best thing I can recomend is to use unique version numbers, and bump bar when foo has an internal "release" change.  You can even automate this bump based on a jenkins build.   I think you'll have a far smother experience all around in the team if instead of just "publish to nexus a -SNAPSHOT" you think: "Publish an integration release", and automate that.

As near as I can tell, your suggestion is to simply not use SNAPSHOT or any other changing() kinds of dependencies. As we have multiple developers working on the same projects, each of us pushing multiple commits per day, upping the version number for each commit does not seem very practical. We need to integrate with each other fast. We can't wait for a jenkins build to push a version change, and we are going to quickly get lost if we each need to find a new, larger version number for every commit. I regularly end up fetching my co-workers changes immediately before committing and pushing.

I don't mean to say that we wouldn't consider this. Just that we would rather continue to try to figure out why SBT publish-local is not handling SNAPSHOT releases as we expect.

 

As far as getting sbt to recognize/use snapshots, it's definitely not the best way to go.

Do you say this because there is something inherently wrong with using SNAPSHOT releases, or because SBT doesn't handle them well/properly?

In any case, current SBT documentation such as [1] certainly suggests that SNAPSHOT and changing() releases should be handled by SBT. I would much rather proceed by fixing the problem than trying to work around it simply abandoning the use of SNAPSHOT/changing() releases on our projects. I'm still not entirely sure if we are doing something wrong, or if it's a bug in SBT. But given the simplicity of the example I've provided, and the fact that I can produce the same error with both NEXUS and Archiva, the evidence suggests to me that this is a bug in SBT. If so, we should probably file it as a bug, and hope someone fixes it some day. I have provided a reasonably isolated example that would make a good basis for a bug report. I'm happy to file it as well. (I would be interested in taking a look at what SBT is doing, and seeing if I can figure out what's going wrong. But my employer is certainly not interested in me doing that, and my free time is very constrained, so I have no idea when I would be able to do so.)

If this is not a bug in SBT, I would like to see or hear about an example where people are successfully using SBT publish-local with changing() releases, since it should help me diagnose the problem. Are there any known examples of people successfully using SBT publish-local with changing() releases?

You want to look into using "latest.integration" as your version instead of -SNAPSHOT.  I think this should work better with publish-local, but maybe Mark can chime in with more details.

Thanks, but as I said earlier, we would prefer a fix to the problem over abandoning SNAPSHOT/changing() releases.

 

  I'd also recommend reading about Ivy's version usage:

http://ant.apache.org/ivy/history/2.3.0-rc1/ivyfile/dependency.html#revision

Nice read, thanks.

Best, John

[1] http://www.scala-sbt.org/release/docs/Detailed-Topics/Dependency-Management-Flow.html 

[Mark Harrah]

Hi John,

Something I noticed was that the builds set sbtVersion := "0.12.3".
That is really a read-only setting (not enforced) because sbt is
already loaded by the time the build file is parsed. Set
`sbt.version=0.12.3` in project/build.properties. (Otherwise, the
default is to use the same version as the launcher.)

There were some issues fixed in various 0.12.x versions related to
local artifacts and 0.12.3 should be ok. There was one issue fixed in
0.13 that hasn't been backported to 0.12.x yet, so you might also test
with 0.13.0-M2 as well if you have no plugin dependencies.

However, it sounds like the problem might be new (in the sense of
unknown, not recently introduced). I am unfortunately unable to look
at this for another week, but if your linked builds are reproducible
for me, I will evaluate it. That workflow should be supported.

Thanks,
Mark

> Thanks, John

[John Sullivan]

Hi Mark,

Thanks for your response.

Something I noticed was that the builds set sbtVersion := "0.12.3".
That is really a read-only setting (not enforced) because sbt is
already loaded by the time the build file is parsed.  Set
`sbt.version=0.12.3` in project/build.properties.  (Otherwise, the
default is to use the same version as the launcher.)

Thanks! That is good to know. I tried again with sbt.version set in project/build.properties as you suggested, but it didn't resolve the problem. I tried both 0.12.3 and 0.13.0-20130516-052158.

 

However, it sounds like the problem might be new (in the sense of 
unknown, not recently introduced).  I am unfortunately unable to look
at this for another week, but if your linked builds are reproducible
for me, I will evaluate it.  That workflow should be supported.

 Thanks Mark. We really appreciate it. When you do get a chance to look at it, please let me know about any trouble you run into reproducing the problem. I'll do what I can to assist.

Best, John

[John Sullivan]

I'm going to take a stab at this this morning. Please let me know if I'm duplicating effort. Best, John

[Erlend Hamnaberg]

Try the https://github.com/arktekk/sbt-aether-deploy plugin and see if that solves your problem.

Peace.

Erlend

--

You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
To post to this group, send email to simple-b...@googlegroups.com.

Visit this group at http://groups.google.com/group/simple-build-tool.

[John Sullivan]

Thanks Erlend!

I wish I could say that solved my problem out of the box. I can successfully publish with your plugin, but then I am left with an unresolved dependency on the dependent project. It looks like my NEXUS server is not able to handle the maven-metadata.xml file that was put up by the `sbt aether-deploy` command. "unrecognized tag 'snapshotVersions'." NEXUS version 1.5.0. I pasted in the relevant section of the NEXUS logs here:

https://gist.github.com/sullivan-/5823633

Best, John

[Erlend Hamnaberg]

I guess that is a problem with your nexus version. 

Try upgrading.

[John Sullivan]

Yes of course. Now I am kicking myself because I had installed an upgraded version a couple of weeks back, but for some reason I didn't keep it around. Oh well, pretty easy to reinstall. Best, John

[John Sullivan]

Okay! I upgraded to the latest nexus (2.5.0-04) and the silly problem I mentioned 2 hours ago did indeed go away. However, I still have the same problem I reported when initiating this thread on May 14th. Best, John

On Thursday, June 20, 2013 12:13:37 PM UTC-4, Erlend Hamnaberg wrote:

[Erlend Hamnaberg]

A potential workaround could be to use aether-install and add .m2/repository as a remote.

Contact me off-list if you want more information about that.