Sbt in Jenkins (concurrent) builds

[Iulian Dragos]

I've spent more than a day and a half setting up Jenkins jobs using Sbt, and I'd like to share my findings. I am surprised I couldn't find much on Google about this.

When building the Eclipse IDE we need to build versions for Scala 2.9 and 2.10. That means we rebuild all our dependencies with a given version of Scala, build the IDE itself and run tests. Often there are even more Scala versions that we care about (2.10.1-SNAPSHOT, 2.11.0-M2, etc). We have some powerful machines, so I wanted to run these jobs in parallel. That's when I noticed that jobs started failing spuriously, *always* while building an Sbt project (that means sbinary or sbt itself -- yes, it's one of our dependencies) -- most of our build is actually based on Maven and Tycho. Once the Sbt build succeeeded, the job was guaranteed to complete successfully.

I tracked it down to one underlying issue: concurrent access to a shared disk location. I found two instances of this problem:

- the local Ivy 2 repository (~/.ivy2)

- the Sbt boot directory (~/.sbt/boot)

Once I fixed these (by passing them as java environment variables), I still saw strange job interference patterns. It seems to be related to a certain ~/.sbt/staging directory. Can someone explain to me what is that for? And what can I do to finally get concurrent builds working reliably in Jenkins?

thanks,

iulian

[Paul Phillips]

On Fri, Jan 18, 2013 at 6:16 AM, Iulian Dragos <jagu...@gmail.com> wrote:

I tracked it down to one underlying issue: concurrent access to a shared disk location. I found two instances of this problem:

- the local Ivy 2 repository (~/.ivy2)

- the Sbt boot directory (~/.sbt/boot)

If you look at what my runner does if you give -no-share, it is:

  noshare_opts="-Dsbt.global.base=project/.sbtboot -Dsbt.boot.directory=project/.boot -Dsbt.ivy.home=project/.ivy"

Since you have 2/3 of those already, maybe throw in the other one. I don't have any reason to think it will help, because the only way I know that sbt.global.base is used is to derive the two values which follow it, only that when I long ago looked into the question of how to prevent sharing, this is what I came up with.

It is common to see both ~/.sbt and ~/.ivy2 hardcoded. Any sbt plugin you're using is a possible culprit there, as is sbt itself I suppose.

[Paul Phillips]

Poking around the sbt source, I found another candidate property:

main/src/main/scala/sbt/BuildPaths.scala

23: getFileSetting(stagingDirectory, StagingProperty, defaultStaging(globalBase))(state)

66: final val StagingProperty = "sbt.global.staging"

[Paul Phillips]

"BuildPaths.scala" looks like something to investigate.

final val GlobalBaseProperty = "sbt.global.base"

final val StagingProperty = "sbt.global.staging"

final val GlobalPluginsProperty = "sbt.global.plugins"

final val GlobalSettingsProperty = "sbt.global.settings"

[Iulian Dragos]

Right, thanks a lot Paul!

I think the best would be to rely on sbt-extras from now on... I'll look into adding the missing parts.

iulian

[Robin Green]

Can this be filed in the sbt issue tracker? Is it considered a bug?

[Josh Suereth]

Staging is where git/svn projects are checked out before being built.  I assume you see this thanks to your fork of sbt being from the time when we used that for sbinary.

I'll try to write up an sbt best practices on Jenkins article (unless you already have something).  What's important to know is that ~/.ivy2 is globally locked amongst sbt processes to avoid cache corruption.   So, as Paul suggested, if you want to run in parallel you need different caches.  This includes the "target/" directory, which has the projects resolution cache.  I believe that this is a standard option for the maven integration on Jenkins, so perhaps we need to improve the sbt plugin on Jenkins.

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/KMMF1oJRKN0J.
To post to this group, send email to simple-b...@googlegroups.com.
To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.