Re: [sbt] Configuring resolvers globally to benefit from local mirrors/caching proxied repos
2,004 views
Skip to first unread message
Mark Harrah
Jun 20, 2012, 9:59:08 PM6/20/12
to simple-b...@googlegroups.com
On Tue, 19 Jun 2012 12:15:36 -0700 (PDT)
Michael Klishin <michael....@gmail.com> wrote:
> Hi!
>
> I am one of the travis-ci.org core team members and we are looking to speed
> up dependency installation for Scala
> projects that use SBT. We have a few Maven mirrors and ~/.m2/settings.xml
> that makes projects use them without
> changing anything in their pom files. I am looking into doing something
> similar for SBT and a little stuck.
0.12.0, which is on RC2 right now, has support for overriding the repositories. This is probably the most reliable and straightforward way, but it obviously only works for 0.12 and later. From the 0.12 documentation (unpublished):
### Override all resolvers for all builds
The repositories used to retrieve sbt, Scala, plugins, and application dependencies can be configured globally and declared to override the resolvers configured in a build or plugin definition.
There are two parts:
1. Define the repositories used by the launcher.
2. Specify that these repositories should override those in build definitions.
The repositories used by the launcher can be overridden by defining `~/.sbt/repositories`, which must contain a `[repositories]` section with the same format as the [[Launcher]] configuration file. For example:
[repositories]
local
my-maven-repo: http://example.org/repo
my-ivy-repo: http://example.org/ivy-repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
> This documentation page suggest there is a way to configure SBT globally:
>
> https://github.com/harrah/xsbt/wiki/Global-Settings
>
> but then it mentions ~/.sbt/.sbt and ~/.sbt/global.sbt. Which one should I
> use? Is it a typo in the docs?
I updated the text. Is it clearer now?
> Another thing it does not explain is how to configure resolvers via that
> file and if it is the same as build.sbt, how the settings
> are merged with project-specific ones.
They are the same settings, but you would probably want to override fullResolvers. Users shouldn't normally touch fullResolvers, so the precedence shouldn't matter.
> Am I looking in the right direction? Is there a better way? Do we
> absolutely have to proxy Ivy repositories or
> just proxying common Maven ones would be sufficient? What repositories we
> should proxy?
With the default repositories, sbt will pull Scala and its dependencies from Maven central. sbt itself comes from http://repo.typesafe.com/typesafe/ivy-releases/ (an Ivy repository).
Several sbt plugins are hosted on http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/ (also an Ivy repository).
For snapshots, sbt comes from http://repo.typesafe.com/typesafe/ivy-releases/ and Scala comes from Sonatype's OSS repository https://oss.sonatype.org/content/repositories/snapshots.
As for proxying, it sounds like users have not had success getting Nexus (I am assuming you are using Nexus or a custom solution if you are concerned about Ivy repositories being a problem) to mirror the Ivy repositories properly, although I believe Sonatype has indicated it should in theory work (I may be incorrectly remembering the details of what exactly works).
> The goal is to make SBT use our mirrors by default, so that projects we
> host won't have to
> tweak their settings. We build our own Debian packages of SBT [1], so
> bundling property files and similar approaches will
> be considered, too.
Once sbt and Scala are retrieved, sbt copies them to its "boot" directory. This is by default in ~/.sbt/boot in 0.11 and later. One option is to point projects at this boot directory and make it read-only and shared. This is usually used by organizations locked to a fixed/controlled number of Scala/sbt versions and is probably not too useful for you if you need to support arbitrary sbt and Scala versions.
> In case you need to understand better what is Travis CI and what we do,
> take a look at
>
> http://about.travis-ci.org/
> http://about.travis-ci.org/docs/user/getting-started/
>
> I am sorry about asking such trivial things but last time I used SBT it was
> at 0.7.
Not a problem at all and unfortunately they haven't really been trivial before Josh submitted the patch that got included in 0.12.0. Let us know if there is anything else that would help you support sbt+Scala better.
-Mark
> Thank you.
>
> MK
>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/Vl5Ll5dWpl4J.
> To post to this group, send email to simple-b...@googlegroups.com.
> To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
>
Michael Klishin <michael....@gmail.com> wrote:
> Hi!
>
> I am one of the travis-ci.org core team members and we are looking to speed
> up dependency installation for Scala
> projects that use SBT. We have a few Maven mirrors and ~/.m2/settings.xml
> that makes projects use them without
> changing anything in their pom files. I am looking into doing something
> similar for SBT and a little stuck.
0.12.0, which is on RC2 right now, has support for overriding the repositories. This is probably the most reliable and straightforward way, but it obviously only works for 0.12 and later. From the 0.12 documentation (unpublished):
### Override all resolvers for all builds
The repositories used to retrieve sbt, Scala, plugins, and application dependencies can be configured globally and declared to override the resolvers configured in a build or plugin definition.
There are two parts:
1. Define the repositories used by the launcher.
2. Specify that these repositories should override those in build definitions.
The repositories used by the launcher can be overridden by defining `~/.sbt/repositories`, which must contain a `[repositories]` section with the same format as the [[Launcher]] configuration file. For example:
[repositories]
local
my-maven-repo: http://example.org/repo
my-ivy-repo: http://example.org/ivy-repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
> This documentation page suggest there is a way to configure SBT globally:
>
> https://github.com/harrah/xsbt/wiki/Global-Settings
>
> but then it mentions ~/.sbt/.sbt and ~/.sbt/global.sbt. Which one should I
> use? Is it a typo in the docs?
I updated the text. Is it clearer now?
> Another thing it does not explain is how to configure resolvers via that
> file and if it is the same as build.sbt, how the settings
> are merged with project-specific ones.
They are the same settings, but you would probably want to override fullResolvers. Users shouldn't normally touch fullResolvers, so the precedence shouldn't matter.
> Am I looking in the right direction? Is there a better way? Do we
> absolutely have to proxy Ivy repositories or
> just proxying common Maven ones would be sufficient? What repositories we
> should proxy?
With the default repositories, sbt will pull Scala and its dependencies from Maven central. sbt itself comes from http://repo.typesafe.com/typesafe/ivy-releases/ (an Ivy repository).
Several sbt plugins are hosted on http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/ (also an Ivy repository).
For snapshots, sbt comes from http://repo.typesafe.com/typesafe/ivy-releases/ and Scala comes from Sonatype's OSS repository https://oss.sonatype.org/content/repositories/snapshots.
As for proxying, it sounds like users have not had success getting Nexus (I am assuming you are using Nexus or a custom solution if you are concerned about Ivy repositories being a problem) to mirror the Ivy repositories properly, although I believe Sonatype has indicated it should in theory work (I may be incorrectly remembering the details of what exactly works).
> The goal is to make SBT use our mirrors by default, so that projects we
> host won't have to
> tweak their settings. We build our own Debian packages of SBT [1], so
> bundling property files and similar approaches will
> be considered, too.
Once sbt and Scala are retrieved, sbt copies them to its "boot" directory. This is by default in ~/.sbt/boot in 0.11 and later. One option is to point projects at this boot directory and make it read-only and shared. This is usually used by organizations locked to a fixed/controlled number of Scala/sbt versions and is probably not too useful for you if you need to support arbitrary sbt and Scala versions.
> In case you need to understand better what is Travis CI and what we do,
> take a look at
>
> http://about.travis-ci.org/
> http://about.travis-ci.org/docs/user/getting-started/
>
> I am sorry about asking such trivial things but last time I used SBT it was
> at 0.7.
Not a problem at all and unfortunately they haven't really been trivial before Josh submitted the patch that got included in 0.12.0. Let us know if there is anything else that would help you support sbt+Scala better.
-Mark
> Thank you.
>
> MK
>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/Vl5Ll5dWpl4J.
> To post to this group, send email to simple-b...@googlegroups.com.
> To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
>
Michael Klishin
Jun 21, 2012, 11:20:24 AM6/21/12
to simple-b...@googlegroups.com
Mark Harrah:
> 0.12.0, which is on RC2 right now, has support for overriding the repositories. This is probably the most reliable and straightforward way, but it obviously only works for 0.12 and later.
Is 0.12.0 supposed to be backwards-compatible with 0.11.3? If it is, we can start using it very soon. Also:
our users actively test agaisnt 2.8.1, 2.8.2, 2.9.1 and some 2.9.2. Will the SBT upgrade change anything for
them? I am very not up date with the state of binary compatibility between things in the Scala land.
> From the 0.12 documentation (unpublished):
>
> ### Override all resolvers for all builds
>
> The repositories used to retrieve sbt, Scala, plugins, and application dependencies can be configured globally and declared to override the resolvers configured in a build or plugin definition.
> There are two parts:
>
> 1. Define the repositories used by the launcher.
> 2. Specify that these repositories should override those in build definitions.
>
> The repositories used by the launcher can be overridden by defining `~/.sbt/repositories`, which must contain a `[repositories]` section with the same format as the [[Launcher]] configuration file. For example:
>
> [repositories]
> local
> my-maven-repo: http://example.org/repo
> my-ivy-repo: http://example.org/ivy-repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
>
This sounds perfect for our needs.
> A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
> The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
>
>
>> This documentation page suggest there is a way to configure SBT globally:
>>
>> https://github.com/harrah/xsbt/wiki/Global-Settings
>>
>> but then it mentions ~/.sbt/.sbt and ~/.sbt/global.sbt. Which one should I
>> use? Is it a typo in the docs?
>
> I updated the text. Is it clearer now?
It is, thanks!
>
> As for proxying, it sounds like users have not had success getting Nexus (I am assuming you are using Nexus or a custom solution if you are concerned about Ivy repositories being a problem) to mirror the Ivy repositories properly, although I believe Sonatype has indicated it should in theory work (I may be incorrectly remembering the details of what exactly works).
That's not a problem for us: we will deploy Artifactory side by side with Nexus and then probably switch to run Artifactory only at some point.
MK
> 0.12.0, which is on RC2 right now, has support for overriding the repositories. This is probably the most reliable and straightforward way, but it obviously only works for 0.12 and later.
our users actively test agaisnt 2.8.1, 2.8.2, 2.9.1 and some 2.9.2. Will the SBT upgrade change anything for
them? I am very not up date with the state of binary compatibility between things in the Scala land.
> From the 0.12 documentation (unpublished):
>
> ### Override all resolvers for all builds
>
> The repositories used to retrieve sbt, Scala, plugins, and application dependencies can be configured globally and declared to override the resolvers configured in a build or plugin definition.
> There are two parts:
>
> 1. Define the repositories used by the launcher.
> 2. Specify that these repositories should override those in build definitions.
>
> The repositories used by the launcher can be overridden by defining `~/.sbt/repositories`, which must contain a `[repositories]` section with the same format as the [[Launcher]] configuration file. For example:
>
> [repositories]
> local
> my-maven-repo: http://example.org/repo
> my-ivy-repo: http://example.org/ivy-repo/, [organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext]
>
> A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
> The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
>
>
>> This documentation page suggest there is a way to configure SBT globally:
>>
>> https://github.com/harrah/xsbt/wiki/Global-Settings
>>
>> but then it mentions ~/.sbt/.sbt and ~/.sbt/global.sbt. Which one should I
>> use? Is it a typo in the docs?
>
> I updated the text. Is it clearer now?
>
> As for proxying, it sounds like users have not had success getting Nexus (I am assuming you are using Nexus or a custom solution if you are concerned about Ivy repositories being a problem) to mirror the Ivy repositories properly, although I believe Sonatype has indicated it should in theory work (I may be incorrectly remembering the details of what exactly works).
MK
Mark Harrah
Jun 21, 2012, 9:15:45 PM6/21/12
to simple-b...@googlegroups.com
On Thu, 21 Jun 2012 19:20:24 +0400
Michael Klishin <michael....@gmail.com> wrote:
> Mark Harrah:
>
> > 0.12.0, which is on RC2 right now, has support for overriding the repositories. This is probably the most reliable and straightforward way, but it obviously only works for 0.12 and later.
>
> Is 0.12.0 supposed to be backwards-compatible with 0.11.3?
It is mostly source compatible, but definitely not binary compatible. The goal with a minor version bump is to have minimal changes needed to upgrade, but not to be a drop-in replacement. Binary and source compatibility will be provided within the 0.12.x series.
Michael Klishin <michael....@gmail.com> wrote:
> Mark Harrah:
>
> > 0.12.0, which is on RC2 right now, has support for overriding the repositories. This is probably the most reliable and straightforward way, but it obviously only works for 0.12 and later.
>
> Is 0.12.0 supposed to be backwards-compatible with 0.11.3?
Here is the (again, unpublished) list of compatibility notes:
## Features, fixes, changes with compatibility implications (incomplete, please help)
* The cross versioning convention has changed for Scala versions 2.10 and later as well as for sbt plugins.
* when invoked directly, 'update' will always perform an update ([#335])
* The sbt plugins repository is added by default for plugins and plugin definitions. [#380]
* Plugin configuration directory precedence has changed (see details section below)
* Source dependencies have been fixed, but the fix required changes (see details section below)
* Aggregation has changed to be more flexible (see details section below)
* Task axis syntax has changed from key(for task) to task::key (see details section below)
* The organization for sbt has to changed to `org.scala-sbt` (was: org.scala-tools.sbt). This affects users of the scripted plugin in particular.
* `artifactName` type has changed to `(ScalaVersion, Artifact, ModuleID) => String`
* `javacOptions` is now a task
* `session save` overwrites settings in `build.sbt` (when appropriate). [#369]
> If it is, we can start using it very soon. Also:
> our users actively test agaisnt 2.8.1, 2.8.2, 2.9.1 and some 2.9.2. Will the SBT upgrade change anything for
> them?
>I am very not up date with the state of binary compatibility between things in the Scala land.
> > As for proxying, it sounds like users have not had success getting Nexus (I am assuming you are using Nexus or a custom solution if you are concerned about Ivy repositories being a problem) to mirror the Ivy repositories properly, although I believe Sonatype has indicated it should in theory work (I may be incorrectly remembering the details of what exactly works).
>
> That's not a problem for us: we will deploy Artifactory side by side with Nexus and then probably switch to run Artifactory only at some point.
-Mark
>
> MK
>
>
Michael Klishin
Jun 24, 2012, 1:40:12 AM6/24/12
to simple-b...@googlegroups.com
Mark Harrah:
> A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
> The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
Mark,
The 0.12.0 change summary page [1] says that
«Define the repositories to use by putting a standalone [repositories] section (see the Launcher page) in ~/.sbt/repositories and pass -Dsbt.override.build.repos=true to sbt. Only the repositories in that file will be used by the launcher for retrieving sbt and Scala and by sbt when retrieving project dependencies. »
Do I understand it correctly that if we use our own list of repositories, project-specific ones will be ignored? Tools like Maven
and Leinigen merge repository lists from user-wide settings and project's build file, do I understand it correctly that
SBT 0.12 does not?
Thank you.
1. https://github.com/harrah/xsbt/wiki/ChangeSummary_0.12.0
MK
mic...@defprotocol.org
> A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
> The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
The 0.12.0 change summary page [1] says that
«Define the repositories to use by putting a standalone [repositories] section (see the Launcher page) in ~/.sbt/repositories and pass -Dsbt.override.build.repos=true to sbt. Only the repositories in that file will be used by the launcher for retrieving sbt and Scala and by sbt when retrieving project dependencies. »
Do I understand it correctly that if we use our own list of repositories, project-specific ones will be ignored? Tools like Maven
and Leinigen merge repository lists from user-wide settings and project's build file, do I understand it correctly that
SBT 0.12 does not?
Thank you.
1. https://github.com/harrah/xsbt/wiki/ChangeSummary_0.12.0
MK
mic...@defprotocol.org
Mark Harrah
Jun 26, 2012, 11:08:32 AM6/26/12
to simple-b...@googlegroups.com
On Sun, 24 Jun 2012 09:40:12 +0400
Michael Klishin <michael....@gmail.com> wrote:
> Mark Harrah:
>
> > A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
> > The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
>
> Mark,
>
> The 0.12.0 change summary page [1] says that
>
> «Define the repositories to use by putting a standalone [repositories] section (see the Launcher page) in ~/.sbt/repositories and pass -Dsbt.override.build.repos=true to sbt. Only the repositories in that file will be used by the launcher for retrieving sbt and Scala and by sbt when retrieving project dependencies. »
>
> Do I understand it correctly that if we use our own list of repositories, project-specific ones will be ignored? Tools like Maven
> and Leinigen merge repository lists from user-wide settings and project's build file, do I understand it correctly that
> SBT 0.12 does not?
-Dsbt.override.build.repos=true will completely override the repositories, yes. If you want to augment the list instead, that gets back to the original question. In ~/.sbt/global.sbt, add:
Michael Klishin <michael....@gmail.com> wrote:
> Mark Harrah:
>
> > A different location for the repositories file may be specified by the `sbt.repository.config` system property in the sbt startup script.
> > The final step is to set `sbt.override.build.repos` to true to use these repositories for dependency resolution and retrieval.
>
> Mark,
>
> The 0.12.0 change summary page [1] says that
>
> «Define the repositories to use by putting a standalone [repositories] section (see the Launcher page) in ~/.sbt/repositories and pass -Dsbt.override.build.repos=true to sbt. Only the repositories in that file will be used by the launcher for retrieving sbt and Scala and by sbt when retrieving project dependencies. »
>
> Do I understand it correctly that if we use our own list of repositories, project-specific ones will be ignored? Tools like Maven
> and Leinigen merge repository lists from user-wide settings and project's build file, do I understand it correctly that
> SBT 0.12 does not?
externalResolvers <<= (bootResolvers, externalResolvers) map {
(boot, ext) => (boot ++ ext).distinct
}
This prepends the repositories defined in ~/.sbt/repositories to the resolvers configured in the build. For 0.11, bootResolvers doesn't exist, so you'd prepend them inline.
-Mark
Reply all
Reply to author
Forward
0 new messages