How can I tell sbt to put something in the ivy cache that IvyDE can read, including sources?

[James Moore]

I'm having a problem where IvyDE can't find source attachments for jars that sbt has downloaded.  I think the problem is that sbt ends up putting a file in the ivy cache that looks like this:

/cygdrive/c/Users/james/.ivy2/cache/org.ow2.asm/asm/ivy-4.1.xml:
...
        <publications>
                <artifact name="asm" type="jar" ext="jar" conf="master"/>
        </publications>
...

If I remove /cygdrive/c/Users/james/.ivy2/cache/org.ow2.asm/asm and let IvyDE download that dependency, that section looks like:

        <publications>
                <artifact name="asm" type="jar" ext="jar" conf="master"/>
                <artifact name="asm" type="source" ext="jar" conf="sources" m:classifier="sources"/>
                <artifact name="asm" type="javadoc" ext="jar" conf="javadoc" m:classifier="javadoc"/>
        </publications>

Is there way to tell sbt to create something that looks like what IvyDE builds?

[Mark Harrah]

In order to do this, Ivy pings the remote repository looking for sources and javadoc classifiers when resolving the metadata. This meant that 'update' typically took 3x as long as necessary, so sbt disabled looking for the classified artifacts and instead looks for them on-demand in the 'update-classifiers' task. I thought IvyDE would look them up on demand as well.

-Mark

> Is there way to tell sbt to create something that looks like what IvyDE
> builds?
>

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/gaOb_GqbOGIJ.
> To post to this group, send email to simple-b...@googlegroups.com.
> To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
>

[James Moore]

On Monday, December 10, 2012 6:41:41 AM UTC-8, Mark Harrah wrote:

In order to do this, Ivy pings the remote repository looking for sources and javadoc classifiers when resolving the metadata.  This meant that 'update' typically took 3x as long as necessary, so sbt disabled looking for the classified artifacts and instead looks for them on-demand in the 'update-classifiers' task.  I thought IvyDE would look them up on demand as well.

Oh, wow.  No wonder this was so frustrating to understand.  I had no idea it could be timing dependent, so sometimes it would work and sometimes it would fail mysteriously with exactly the same configuration.  I thought I was going crazy.

Is there any way to turn off that feature, or set the timeout to very long (a day...)?  I want it to do the same thing, every time, always.  I'd much prefer a hang over a partial complete.

(My local nexus server is on archaic hardware with little ram and multiple terabytes of storage.  The first time you hit it for stuff, it's glacial; 10-20 seconds to respond to a request (Hellloooo, swap!).  The next request is usually pretty fast.  Replacing the hardware is something like #29 on my list of important things to do...)

[Mark Harrah]

On Mon, 10 Dec 2012 10:38:22 -0800 (PST)
James Moore <ja...@banshee.com> wrote:

> On Monday, December 10, 2012 6:41:41 AM UTC-8, Mark Harrah wrote:
>
> > In order to do this, Ivy pings the remote repository looking for sources
> > and javadoc classifiers when resolving the metadata. This meant that
> > 'update' typically took 3x as long as necessary, so sbt disabled looking
> > for the classified artifacts and instead looks for them on-demand in the
> > 'update-classifiers' task. I thought IvyDE would look them up on demand as
> > well.
> >
> >
> Oh, wow. No wonder this was so frustrating to understand. I had no idea
> it could be timing dependent, so sometimes it would work and sometimes it
> would fail mysteriously with exactly the same configuration. I thought I
> was going crazy.

Sorry I wasn't clear. This isn't a race condition or anything and behavior isn't being enabled/disabled on the fly.

The default Ivy approach to look up classified artifacts at normal resolution time is just a bit slower than necessary. It requires a request to get the metadata (pom.xml) and then two more requests to check whether there is an artifact -sources.jar and -javadoc.jar. This is necessary because pom.xml does not declare classified artifacts. However, it is better to look for the sources and javadocs later, when the user runs 'update-classifiers'. So, sbt doesn't look up sources and javadocs.

> Is there any way to turn off that feature, or set the timeout to very long
> (a day...)? I want it to do the same thing, every time, always. I'd much
> prefer a hang over a partial complete.

It should do the same thing every time. Does the above explanation make more sense now or am I the one missing something?

-Mark

> (My local nexus server is on archaic hardware with little ram and multiple
> terabytes of storage. The first time you hit it for stuff, it's glacial;
> 10-20 seconds to respond to a request (Hellloooo, swap!). The next request
> is usually pretty fast. Replacing the hardware is something like #29 on my
> list of important things to do...)
>

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/vOilMdnhamQJ.

[James Moore]

On Monday, December 10, 2012 7:26:37 PM UTC-8, Mark Harrah wrote:

It should do the same thing every time.  Does the above explanation make more sense now or am I the one missing something?

-Mark

Yes - I totally misread your first explanation - thanks for the clarification.  What it actually does makes much more sense than what I imagined it doing...

Should update-classifiers be updating the ivy.xml file to have the extra two lines for artifacts in the publications section?  It doesn't, at least when I run it.   If I remove the cache directory (in this case,  C:\Users\james\.ivy2\cache\org.ow2.asm\asm) sbt rebuilds the xml files there and redownloads the jars, but there's only the one artifact line when it's done.

IvyDE seems to see that xml file with only the one line and then assume there's nothing else to do.  I end up with an IvyDE container that has the jars, but no source attachment.  It looks like IvyDE does see the xml, and then decides that it's done:

From the IvyDE log:

found ivy file in cache for org.ow2.asm#asm;4.1 (resolved by nexus): C:\Users\james\.ivy2\cache\org.ow2.asm\asm\ivy-4.1.xml

nexus: revision in cache: org.ow2.asm#asm;4.1

checking org.ow2.asm#asm;4.1 from nexus against [none]

module revision kept as first found: org.ow2.asm#asm;4.1 from nexus

If I delete C:\Users\james\.ivy2\cache\org.ow2.asm\asm and then tell IvyDE to resolve, then IvyDE goes through and downloads everything and hooks up sources to its classpath container.

IvyDE does have some options that are related to clearing the cache, but as far as I can tell they don't do anything in the C:\Users\james\.ivy2\cache tree; I think they just clear IvyDE's own internal caches.

[Mark Harrah]

On Thu, 13 Dec 2012 22:48:50 -0800 (PST)
James Moore <ja...@banshee.com> wrote:

> On Monday, December 10, 2012 7:26:37 PM UTC-8, Mark Harrah wrote:
>
> > It should do the same thing every time. Does the above explanation make
> > more sense now or am I the one missing something?
> >
> > -Mark
> >
> >
> Yes - I totally misread your first explanation - thanks for the
> clarification. What it actually does makes much more sense than what I
> imagined it doing...
>
> Should update-classifiers be updating the ivy.xml file to have the extra
> two lines for artifacts in the publications section? It doesn't, at least
> when I run it. If I remove the cache directory (in this case,
> C:\Users\james\.ivy2\cache\org.ow2.asm\asm) sbt rebuilds the xml files
> there and redownloads the jars, but there's only the one artifact line when
> it's done.

No. Classified artifacts are not list in the pom.xml, so Ivy normally tries to work around this by explicitly checking for them when it retrieves the pom.xml. sbt disables this because it unnecessarily takes 3x as long. Metadata is cached, so it does not get updated once it is retrieved.

> IvyDE seems to see that xml file with only the one line and then assume
> there's nothing else to do. I end up with an IvyDE container that has the
> jars, but no source attachment. It looks like IvyDE does see the xml, and
> then decides that it's done:
>
> From the IvyDE log:
>
> found ivy file in cache for org.ow2.asm#asm;4.1 (resolved by nexus):
> C:\Users\james\.ivy2\cache\org.ow2.asm\asm\ivy-4.1.xml
> nexus: revision in cache: org.ow2.asm#asm;4.1
> checking org.ow2.asm#asm;4.1 from nexus against [none]
> module revision kept as first found: org.ow2.asm#asm;4.1 from nexus
>
> If I delete C:\Users\james\.ivy2\cache\org.ow2.asm\asm and then tell IvyDE
> to resolve, then IvyDE goes through and downloads everything and hooks up
> sources to its classpath container.

That is because metadata is cached. When you delete it and resolve with IvyDE, it uses the standard Ivy resolution that checks for classified artifacts. You could use a different cache for IvyDE, but you'll probably download artifacts multiple times.

-Mark

> IvyDE does have some options that are related to clearing the cache, but as
> far as I can tell they don't do anything in the C:\Users\james\.ivy2\cache
> tree; I think they just clear IvyDE's own internal caches.
>

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/9dvTYi0i3QwJ.

[James Moore]

On Tuesday, December 18, 2012 6:05:58 AM UTC-8, Mark Harrah wrote:

When you delete it and resolve with IvyDE, it uses the standard Ivy resolution that checks for classified artifacts.  You could use a different cache for IvyDE, but you'll probably download artifacts multiple times. 

That sounds like the best solution.  It's cheap (2x disk space for the local cache isn't a big deal) and fast (as long as you set up some sort of cache repository).  Thanks for suggesting it; it's the sort of thing that should have been obvious to me, but wasn't.