New sbt projects take too long (hours) to get the dependencies

[Armin Keyvanloo]

Hi all,

In Australia I am experiencing appalling experience with sbt. 

Every time I start a new project it takes many hours to download the dependencies. It just downloads jars after jars with very inconsistence download speeds between them.

What's going on here? Do others around the world have a similar experience? Any pointers to fix/improve the performance?

I even checked out the sbt code and interested in providing more diagnosis and potentially improve/fix this issue if someone kindly gave me some clues on how to go about it.

[eugene yokota]

There seems to be something going on with Australia (and other countries too I assume).

Which version of sbt are you using?

Some of the potential parameters:

- the network distance between your ISP and the repositories where the artifacts are hosted

- CDN used by these repositories, and latency and throughput

- HTTP vs HTTPS, and their initial handshakes

For instance, HTTPS was enabled in 0.13.6 by default[1].

Here are a few things you could try out to help us diagnose the situation:

1. Put sbt-dirty-money[2] in ~/.sbt/0.13/plugins/dirtymoney.sbt, which introduces

      cleanCache (cleans current project's JAR from Ivy cache), and

      cleanCache "com.github.scopt" %% "scopt" (cleans specified library's JAR from Ivy cache)

   This allows you to experiment with various settings.

2. Use JCenter as the default Maven repository:

      fullResolvers := (Resolver.jcenterRepo +: fullResolvers.value)

    Maybe Australia is far from Maven Central, but JCenter, backed by Amazon's CDN at least in United States,

    gives you better result.

3. Turn off HTTPS. Running sbt with:

        -Dsbt.repository.secure=false

    should turn off HTTPS for resolving artifacts. That could reduce the latency.

4. Test your latency. James Roper opened [#590][590] discussing this topic,

    and in there I posted:

        $ curl -w "tcp:%{time_connect}, ssl:%{time_appconnect}, start_transfer:%{time_starttransfer}, total:%{time_total}" -o /dev/null -s https://jcenter.bintray.com/org/scala-sbt/launcher-interface/1.0.0-M1/launcher-interface-1.0.0-M1.pom

    If you could post your result for the above, it might be helpful. While you're at it could you also post for Maven Central as well?:

        $ curl -w "tcp:%{time_connect}, ssl:%{time_appconnect}, start_transfer:%{time_starttransfer}, total:%{time_total}" -o /dev/null -s https://repo1.maven.org/maven2/org/scala-sbt/launcher-interface/1.0.0-M1/launcher-interface-1.0.0-M1.pom 

5. Once you find the right combination, you could try installing a local proxy such as Artifactory,

    and resolve artifacts from that for all projects. See [3]

Another idea James had is pooling the connection.

AFAIK no attempt is made currently to pool the connections, so the initial handshake is repeated for each files.

I'm not sure if pooling would help given that many of the repositories would need to redirect actual downloading to CDNs,

but that might be something worth trying.

-eugene

[1]: http://www.scala-sbt.org/0.13/docs/sbt-0.13-Tech-Previews.html#HTTPS+related+changes

[2]: https://github.com/sbt/sbt-dirty-money

[590]: https://github.com/sbt/sbt/issues/590

[3]: http://www.scala-sbt.org/0.13/docs/Proxy-Repositories.html

--
You received this message because you are subscribed to the Google Groups "sbt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbt-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/13ae1134-f193-4152-96c2-fd19cb421fad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Naftoli Gugenheim]

Couldn't SBT be made to download files in parallel?

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/CA%2BCq6KSMu%3DpjYyVuzfz_8P0ZMr-P1XaCtnf-em0a5V0bcR00fw%40mail.gmail.com.

[naturallog]

I am in India and have a very similar experience.

A new play project takes 2-3 hours to download all dependencies.

[Armin Keyvanloo]

Eugene,

Many thanks for the suggestions. Not much luck with them unfortunately.

My sbt version is 0.13.8 (via activator 1.3.5)

As you suggested, I cleared the ~/.ivy2/cache/ dir manually as the cleanCache task failed to remove anything:

> cleanCache

[success] Total time: 0 s, completed 16-Jul-2015 21:51:16

And added  fullResolvers := (Resolver.jcenterRepo +: fullResolvers.value) to my build.sbt and started activator with  -Dsbt.repository.secure=false and it seems to have made no performance improvements and may be even gotten worse:

> activator  -Dsbt.repository.secure=false

[info] Loading global plugins from /Users/armin/.sbt/0.13/plugins

[info] Updating {file:/Users/armin/.sbt/0.13/plugins/}global-plugins...

[info] Resolving org.scala-sbt#process;0.13.8 ...

[warn] problem while downloading module descriptor: https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/process/0.13.8/ivys/ivy.xml: Operation timed out (75021ms)

[error] Server access Error: Operation timed out url=https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.scala-sbt/process/0.13.8/ivys/ivy.xml

[error] Server access Error: Operation timed out url=https://repo1.maven.org/maven2/org/scala-sbt/process/0.13.8/process-0.13.8.pom 

And the curl results you asked for:

>curl -w "tcp:%{time_connect}, ssl:%{time_appconnect}, start_transfer:%{time_starttransfer}, total:%{time_total}" -o /dev/null -s https://jcenter.bintray.com/org/scala-sbt/launcher-interface/1.0.0-M1/launcher-interface-1.0.0-M1.pom

tcp:0.310, ssl:1.232, start_transfer:1.538, total:1.538%

>curl -w "tcp:%{time_connect}, ssl:%{time_appconnect}, start_transfer:%{time_starttransfer}, total:%{time_total}" -o /dev/null -s https://repo1.maven.org/maven2/org/scala-sbt/launcher-interface/1.0.0-M1/launcher-interface-1.0.0-M1.pom

tcp:0.205, ssl:1.069, start_transfer:1.479, total:1.479%

An interesting observation though:

One of the many slow downloads that the terminal sits on for ~7 minutes is:

[info] downloading http://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.10.4/scala-compiler-2.10.4.jar ...

[info] [SUCCESSFUL ] org.scala-lang#scala-compiler;2.10.4!scala-compiler.jar (406570ms)

I did the curl for that jar and got:

>curl -w "tcp:%{time_connect}, ssl:%{time_appconnect}, start_transfer:%{time_starttransfer}, total:%{time_total}" -o /dev/null -s http://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.10.4/scala-compiler-2.10.4.jar

tcp:0.000, ssl:0.000, start_transfer:0.000, total:29.603%

Felt reasonably quick via curl when the sbt download took 7 mins.

When this run is done (may be by the time I wake up tomorrow morning!), I will attach the output of it to this post. This should help you as it has all the times and durations for each download.

I think if you really want to experience this for yourself you can spin up an aws ec2 instance in the Sydney (ap-southeast-2) and give it a crack. 

I have to say, many people that I speak to, who have had an interest and tried to give Scala a go, have been put off by this terrible performance experience. I am not joking when I say that I believe that this is hurting the adoption of Scala here in Australia. Please do something about it.

 

[Armin Keyvanloo]

Please see the log attached.

[Mateusz Karbowy]

Hey guys,

I'm having exactly the same issue in Australia.

Any progress/update on this?

Cheers,

Mateusz