[sbt] Sbt update incredibly slow?

[Marco]

I'm just playing around with sbt, but noticed a very strange thing
trying to compile a tiny sample sbt application as well as akka trunk.

sbt update seems to take forever, meaning sbt completelyl hangs at the
"update" stage.....and then continues downloading the projects
dependencies after...say anything from 3-10minutes.

As I'm writing this I'm still "sbt updating" akka trunk, and after
~20mins I'm at the second(!) module "==akka-core / update==" where sbt
continues to hang.

Anyone else experiencing this?

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To post to this group, send email to simple-b...@googlegroups.com.
To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.

[Stefan Langer]

This could well be a problem with your network connection and proxy
settings and long timeouts. Try doing a wget for one of the listed
dependency from the same console and see what it tells you.

2010/4/29 Marco <beh...@googlemail.com>:

[David Brown]

On Thu, Apr 29, 2010 at 07:06:11AM -0700, Marco wrote:

>As I'm writing this I'm still "sbt updating" akka trunk, and after
>~20mins I'm at the second(!) module "==akka-core / update==" where sbt
>continues to hang.
>
>Anyone else experiencing this?

I see this if I use the griddynamics nexus:

val nexusRepo = "nexus" at "https://nexus.griddynamics.net/nexus/content/groups/public"

I'm not sure what is causing it, though.

David

[Marco]

Hi Stefan,
I would love it to be my network and/or proxy settings, but I still
have my doubts - a full blown maven2 build (after deleting the .m2
cache and a lot of dependencies) works just fine. Sbt also doesn't
hang while "downloading" the dependencies, but right at the beginning
of the "update" stage where it simply displays "update" on the command
line.

WGetting the first dependencies sbt is looking for, such as scala-
compiler etc worked just fine as well.

hmpf..

[Alexandre Bertails]

+1

I have exactly the same behaviour. I don't really know when it began.

Alexandre.

2010/4/29 Marco <beh...@googlemail.com>:

[Mark Harrah]

If you are experiencing this problem, please check if earlier versions of sbt work for you or if specific repositories or dependencies are causing the problem.

Thanks,
Mark

[Marco]

Hi Mark,
alright, I tried this on Windows 7 and Ubuntu, both times the results
were the same:

I created a plain sbt project with the command line tool and sbt
versions 0.7.0 as well as 0.7.3. The only thing I then added was a
HelloWorldProject file with spring and hibernate dependencies , i.e.
dependencies that are available from the standard maven repo and can
easily be downloaded with wget or maven (tried that beforehand, works
as expected and is really fast).

The file looks like this:


class HelloWorldProject(info: ProjectInfo) extends
DefaultProject(info)
{
val springcore = "org.springframework" % "spring-core" % "2.5.6"
val springaop = "org.springframework" % "spring-aop" % "2.5.6"
val springtx = "org.springframework" % "spring-tx" % "2.5.6"


val hib = "org.hibernate" % "hibernate-core" % "3.3.2.GA"
val hibannot = "org.hibernate" % "hibernate-annotations" %
"3.4.0.GA"
}

Now, after "reloading" the project, when I enter "update", sbt hangs
for about ~2minutes before starting to download hibernate. Actually
sbt also feels really sluggish downloading the scala-compiler version
when creating a new project, i.e. between "Getting Scala 2.7.7....."
and "downloading ......scala-compiler-2.7.7.jar"..... there's a
noticable delay (~10 seconds, whereas the download itself is quite
fast).

Dunno if that helps :)

[Heiko Seeberger]

Maybe you could rerun your tests with .ivy2 and .m2 deleted before that?

Heiko

--
Heiko Seeberger

Company: weiglewilczek.com
Blog: heikoseeberger.name
Follow me: twitter.com/hseeberger
OSGi on Scala: scalamodules.org
Lift, the simply functional web framework: liftweb.net
Stambecco, highly scalable computing: stambecco.org

[Marco]

That's actually exactly what I did :)

On 30 Apr., 10:55, Heiko Seeberger <heiko.seeber...@googlemail.com>
wrote:

> > simple-build-t...@googlegroups.com<simple-build-tool%2Bunsu...@googlegroups.com>
> > .

> > > > >> For more options, visit this group athttp://
> > groups.google.com/group/simple-build-tool?hl=en.
>

> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > Groups "simple-build-tool" group.
> > > > > To post to this group, send email to
> > simple-b...@googlegroups.com.
> > > > > To unsubscribe from this group, send email to

> > simple-build-t...@googlegroups.com<simple-build-tool%2Bunsu...@googlegroups.com>
> > .

> > > > > For more options, visit this group athttp://
> > groups.google.com/group/simple-build-tool?hl=en.
>

> > > --
> > > You received this message because you are subscribed to the Google Groups
> > "simple-build-tool" group.
> > > To post to this group, send email to simple-b...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > simple-build-t...@googlegroups.com<simple-build-tool%2Bunsu...@googlegroups.com>
> > .

> > > For more options, visit this group athttp://
> > groups.google.com/group/simple-build-tool?hl=en.
>

> > --
> > You received this message because you are subscribed to the Google Groups
> > "simple-build-tool" group.
> > To post to this group, send email to simple-b...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > simple-build-t...@googlegroups.com<simple-build-tool%2Bunsu...@googlegroups.com>
> > .

[Robert Wills]

I've noticed this issue as well. I haven't bothered to investigate it
properly because for me the benefits of sbt outweight this minor
annoyance.

I guess it's something to do with how sbt interacts with ivy as
updateTask is just a wrapper around ivy:
def updateTask(module: => IvySbt#Module, configuration: =>
UpdateConfiguration) =
ivyTask { IvyActions.update(module, configuration) }

Would it be worthwhile running those dependencies directly through an
ivy project?

-Rob

[Mark Harrah]

On Friday 30 April 2010, Robert Wills wrote:
> I've noticed this issue as well. I haven't bothered to investigate it
> properly because for me the benefits of sbt outweight this minor
> annoyance.
>
> I guess it's something to do with how sbt interacts with ivy as
> updateTask is just a wrapper around ivy:
> def updateTask(module: => IvySbt#Module, configuration: =>
> UpdateConfiguration) =
> ivyTask { IvyActions.update(module, configuration) }

I ran the HelloWorldProject from this thread both in sbt and as a pure Ivy file. With a clean cache on a ~5Mbps connection, it takes 50-60s, which seems reasonable to me.

Some notes: Ivy resolves all metadata first and then downloads jars. You'll see this explicitly if you run 'update' after enabling the 'debug' logging level. This is why there is an initial pause (it is checking each repository). The 10 s pause before downloading a version of Scala not previously used is not of huge concern for me. It should take maybe 1 s for a new project when sbt and Scala have already been downloaded.

> Would it be worthwhile running those dependencies directly through an
> ivy project?

Sure. I posted the equivalent Ivy file to the group files section[1]. Run Ivy in standalone mode with this command in an existing sbt project.

java -jar project/boot/scala-2.7.7/org.scala-tools.sbt/sbt/0.7.3/ivy-2.1.0.jar -ivy ivy-1.0.xml -retrieve 'test_lib/[conf]/[artifact].[ext]'

Running in an sbt project is not essential- just change the location of ivy.jar appropriately. If you are familiar with maven, I'd of course be interested in timing an equivalent pom.

-Mark

[1] http://groups.google.com/group/simple-build-tool/files

[Heiko Seeberger]

On 30 April 2010 14:15, Mark Harrah <dmha...@gmail.com> wrote:

Some notes: Ivy resolves all metadata first and then downloads jars.  You'll see this explicitly if you run 'update' after enabling the 'debug' logging level.  

Good hint!

Heiko

Company: weiglewilczek.com
Blog: heikoseeberger.name
Follow me: twitter.com/hseeberger
OSGi on Scala: scalamodules.org
Lift, the simply functional web framework: liftweb.net
Stambecco, highly scalable computing: stambecco.org

[Jorge Ortiz]

I've seen this using griddynamics nexus too. When I run 'update' after setting log level to debug, every jar hangs for like 20-30s after printing stuff like this:

[debug] trying https://nexus.griddynamics.net/nexus/content/groups/public/com/rabbitmq/amqp-client/1.7.0/amqp-client-1.7.0-javadoc.jar

[debug] tried https://nexus.griddynamics.net/nexus/content/groups/public/com/rabbitmq/amqp-client/1.7.0/amqp-client-1.7.0-javadoc.jar

and before printing stuff like this:

[debug] HTTP response status: 404 url=https://nexus.griddynamics.net/nexus/content/groups/public/com/rabbitmq/amqp-client/1.7.0/amqp-client-1.7.0-javadoc.jar

[debug] CLIENT ERROR: Item not found on path "/com/rabbitmq/amqp-client/1.7.0/amqp-client-1.7.0-javadoc.jar" in repository "public"! url=https://nexus.griddynamics.net/nexus/content/groups/public/com/rabbitmq/amqp-client/1.7.0/amqp-client-1.7.0-javadoc.jar

[debug] nexus: resource not reachable for com/rabbitmq#amqp-client;1.7.0: res=https://nexus.griddynamics.net/nexus/content/groups/public/com/rabbitmq/amqp-client/1.7.0/amqp-client-1.7.0-javadoc.jar

--j

[Indrajit Raychaudhuri]

FWIW, newer (and compatible) amqp client is available on Maven Central [1] for quite sometime now. So you might consider this one instead.
griddynamics.net repo had been cause of inconvenience for us in Lift as well and we have discontinued using this repo since.

- Indrajit

[1] http://repo1.maven.org/maven2/com/rabbitmq/amqp-client/1.7.2/

[Jorge Ortiz]

Is there a way to specify the order in which repositories should be checked for artifacts?

--j

[Mark Harrah]

On Tuesday 04 May 2010, Jorge Ortiz wrote:
> Is there a way to specify the order in which repositories should be checked
> for artifacts?

You can explicitly override 'repositories' if you only need to order added repositories or you can override 'ivyRepositories' to control the default repositories as well.

The default implementation of 'ivyRepositories' is equivalent to:

override def ivyRepositories =
Seq(Resolver.defaultLocal(None)) ++
repositories ++
Seq(DefaultMavenRepository, ScalaToolsReleases)

You might also see the response I gave in this thread related to ModuleConfigurations, which are a way to specify that a repository is only use for certain modules.

-Mark

[Seance]

Hi all,

Just adding my experience. I downloaded the akka-template-rest from
http://github.com/efleming969/akka-template-rest/ and run it without
any changes. It's an SBT-built project.

This took 1524 seconds (25 minutes). I'm on a 100 Mbit line. Most of
the time was spent with the ==update== text sitting there without any
apparent progress.

It's probably not relevant to SBT but incidentally jetty-run failed
with java.lang.NoSuchMethodError: scala.collection.mutable.ArrayBuffer.
$plus$plus$eq at net.lag.logging.Formatter.format(Formatter.scala:
171).

All this made me very weary... :(

Hopefully the authors will address this issue since many interesting
projects use SBT which seems quite promising itself too!

[Heiko Seeberger]

Hi,

I recently refactored Akka's SBT project definition resulting in about 10 times faster update. akka-template-rest obviously still does the wrong thing: Using lots of repositories without restricting these to specific modules (artifacts).

This is not a SBT issue but a Ivy feature. In order to really speed up SBT one has to use module configurations for each dependency which comes from a not-default (Maven Central or Scala Tools Releases) repository.

Regarding the NoSuchMethodError: This looks like you are using the wrong Scala version. You need to use the same as Akka was compiled against. So this is not a SBT issue, neither.

Heiko

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To post to this group, send email to simple-b...@googlegroups.com.
To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.

--
Heiko Seeberger

[Viktor Klang]

Hello Jukka,

As a very beneficial work-around, if you migrate your project to use the freshly baked Akka 0.9 you'll see a drop in update time.

And I second Jukkas opinion, the update command is perceived to be real slow.

Cheers,

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To post to this group, send email to simple-b...@googlegroups.com.
To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

[Heiko Seeberger]

Victor,

On 2 June 2010 16:58, Viktor Klang <viktor...@gmail.com> wrote:

As a very beneficial work-around, if you migrate your project to use the freshly baked Akka 0.9 you'll see a drop in update time.

The issue is, that Jukka uses the akka-template-rest project which hasn't got an optimized SBT project definition like Akka 0.9 now has got.

 

And I second Jukkas opinion, the update command is perceived to be real slow.

Ivy update is incredibly slow when there are multiple repositories. Therefore projects should not define additional repositories but module configurations => These will uses the additional repos only for specific modules / artifacts.

Heiko

[Viktor Klang]

On Wed, Jun 2, 2010 at 5:26 PM, Heiko Seeberger <heiko.s...@googlemail.com> wrote:

Victor,

On 2 June 2010 16:58, Viktor Klang <viktor...@gmail.com> wrote:

As a very beneficial work-around, if you migrate your project to use the freshly baked Akka 0.9 you'll see a drop in update time.

The issue is, that Jukka uses the akka-template-rest project which hasn't got an optimized SBT project definition like Akka 0.9 now has got.

Ah, crap sorry, I was too fast on that one. Of course the project'd need to be modified to use the Akka 0.9 project module definition approach.
 

 

And I second Jukkas opinion, the update command is perceived to be real slow.

Ivy update is incredibly slow when there are multiple repositories.

i've been intrigued as to what it's actually trying to do when it's updating, why is it that slow?
 

Therefore projects should not define additional repositories but module configurations => These will uses the additional repos only for specific modules / artifacts.

Heiko

Company: weiglewilczek.com
Blog: heikoseeberger.name
Follow me: twitter.com/hseeberger
OSGi on Scala: scalamodules.org
Lift, the simply functional web framework: liftweb.net
Stambecco, highly scalable computing: stambecco.org

--

You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To post to this group, send email to simple-b...@googlegroups.com.
To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.

[Heiko Seeberger]

On 2 June 2010 17:37, Viktor Klang <viktor...@gmail.com> wrote:

Ivy update is incredibly slow when there are multiple repositories.

i've been intrigued as to what it's actually trying to do when it's updating, why is it that slow?

I just know that it is slow, but I don't know why.

Heiko