sbt 0.13.2-M2 is released!

[Josh Suereth]

sbt 0.13.2-M2 is published. 

0.13.2-M2 is predominantly a bugfix release, but this milestone contains new features improvements we'd like to ensure are stable before releasing:

• New incremental compilation algorithm which should help reduce the amount of files touched on recompile.  To enable incremental compilation, add this setting:
  
  incOptions := incOptions.value.withNameHashing(true)
  
  
  
• The sbt launcher[1] has the ability to launch servers in addition to applications.  You can now create servers which bind to dynamic ports and discover the port using a new --locate option on the launcher.  Fledgling docs [2] [3] are available for this feature.
  
  
• There is a new plugin feature called "AutoPlugins" [4].  This feature allows plugins to be automatically enabled on projects based on "nature" strings (rather than having settings added to every project).  We'll be documenting/improving this feature before the final 0.13.2-RC, in the meantime early adopters/plugin authors are encouraged to read the scaladoc comments and offer feedback.

Use it in an existing 0.13 project by modifying project/build.properties to be:

sbt.version=0.13.2-M2

There is a new launcher[1], but the 0.13.{0,1} launcher should work fine with 0.13.2-M2.  No changes should be necessary to your project definition and all plugins published for 0.13.x remain binary compatible with this release. 

Please try out the new features and report back any issues you find!

Thanks, 
The sbt Team

[1] http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.2-M1/sbt-launch.jar

[2] https://github.com/sbt/sbt/blob/0.13/src/sphinx/Launcher/Configuration.rst#6-the-server-section

[3] https://github.com/sbt/sbt/blob/0.13/src/sphinx/Launcher/Architecture.rst#service-discovery-and-isolation

[4] https://github.com/sbt/sbt/blob/0.13/main/src/main/scala/sbt/Natures.scala#L16-L47

[Eric Torreborre]

Hi Josh,

If I understand correctly it is up to the plugin author to do decide of which "Nature(s)" the plugin is? Why don't we decouple this from the plugin definition and let the use decide to "tag" plugins with different "natures".

This way:

 - there's less risk of having ambiguous/conflicting "natures" being defined. My "Web" definition might not be yours

 - there's more flexibility in deciding what's in and out for a given project

My other comment is: is there a way to reuse existing concepts in sbt to support the selective loading / unloading of plugins? There are already quite a few scopings. Adding a new type of scoping makes SBT even more complex IMHO.

Eric.

[Josh Suereth]

Hey Eric.   I think perhaps I jumped the gun on announcing project natures without throwing together a loosely organized document of all the use cases Mark and I had talked about for this feature.  

Your concern over complexity is a good concern. This will add an orthogonal "piece" to sbt.  However, as the natures  are a more restricted subset than what's allowed currently by plugins, we hope that this systems will INCREASE usability and most importantly discoverability/error reporting.

Here's the quick draft of the user stories we hope to solve with project natures https://github.com/sbt/sbt/wiki/User-Stories:-Project-Natures  (as well as a few I haven't written down).

One thing I've been debating with in sbt 1.0 is whether or not we can fragment all aspects of Defaults.scala into "plugins", enabled with default natures (besides build-level settings which would have to be defined first).  I think this would be good.   So to answer your specific questions:

On Sun, Feb 23, 2014 at 6:24 PM, Eric Torreborre <etorr...@gmail.com> wrote:

Hi Josh,

If I understand correctly it is up to the plugin author to do decide of which "Nature(s)" the plugin is? Why don't we decouple this from the plugin definition and let the use decide to "tag" plugins with different "natures".

Sort of.   All plugins are by-definition a nature.  This way if you depend on another plugin, you "select" on it directly, ensuring its settings are in the build *and* yours.  This solves the issue where, e.g.  play sbt plugin settings couldn't be safely added.

Also, we have similar problems with keys now.  If two plugins declare the same "string" key with different types, they are incompatible.  It's the same with the string-ified natures.

What we want to do is provide a default set of string natures that folks can use.   Ideally this will be a limited-high-level set which plugins will see.  In addiitonal, all aspects of the default build will turn into AutoPlugins that can be depending on directly.   Finally, since we hope plugins which interoperate will depend *either* on a shared interface plugin (like sbt-site/sbt-web) *or* directly on the other plugin (like sbt-ghpages), I think that mitigates the issue.

I could well be wrong, but it's actually great you opened this discussion.  I put together a document of the use cases. The next bit I want to do is flesh out a possible fragmenetation of Defaults.scala into plugins/base natures and get some more eyes/thoughts on this.

Would be curious if this helps alleviate your concerns or makes them worse :)

- Josh

 

--
You received this message because you are subscribed to the Google Groups "sbt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbt-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/fde79fa4-38f5-49f3-bfd3-2b8c33101575%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Eric Torreborre]

Hi Josh,

> One thing I've been debating with in sbt 1.0 is whether or not we can fragment all aspects of Defaults.scala into "plugins", enabled with default natures (besides build-level settings which would have to be defined first).  I think this would be good.

That's a very good idea.

> What we want to do is provide a default set of string natures that folks can use

I'm not sure how well this is going to work because we might have different definitions of what "Publish" means for example (but I might be wrong we need to see that on concrete examples).

Overall I'm still concerned with complexity. It appears to me that there are already mechanisms to deal with loading and dependencies in sbt: Configurations and ivy dependencies/configurations. Isn't it possible to use that to support all the user stories?

addSbtPlugin("com.typesafe.sbt" % "sbt-pgp" % "1.0" % "publish->default")

Eric.

[Havoc Pennington]

Hi,

I wonder if we'll primarily use the natures which are simply
equivalent to plugins. In the current code each plugin has a nature
defined based on the plugin object name.

In the third story
https://github.com/sbt/sbt/wiki/User-Stories:-Project-Natures#wiki-user-story-user-wants-to-create-an-sbt-plugin-that-enhances-play-projects
MyPlugin is simply conditional on PlayPlugin (`def select = PlayPlugin`)

In the second story I don't think there's a difference between Play
and PlayPlugin:
https://github.com/sbt/sbt/wiki/User-Stories:-Project-Natures#wiki-story-user-wants-to-aggregate-tasks-on-a-project-but-not-have-that-project-actually-have-any-behavior
the story uses Play but it could just be PlayPlugin. And Web could
just be WebPlugin (sbt-web). This removes any ambiguity (what does
"Web" mean? it means sbt-web)

For publish, what would Publish mean? It would just mean the built-in
plugin provided by sbt core which contains the publish task. Other
concepts of publish would have a different name (or live in a
different package at least).

One way to simplify then could be to remove these "virtual" or
"symlink" nature names such as Publish and Web and just say that
natures == plugins, though "plugin" here means a Plugin object
instance, not project library dependency.

If you still wanted the idea of a nature with multiple
implementations, that could be done through a Plugin instance which
just provided task keys and no task implementations or something like
that, and people would then create other plugins that depended on that
one and provided concrete implementations. Or for a nature which was
"just" conceptual with no tasks, provide an empty Plugin instance...

The main advantage here would be to drop the word/concept of a
"nature" and just have "instances of Plugin"

A remaining confusion is that "plugin" means both "Plugin object
instance" and "project library dependency as in addSbtPlugin." Though
that confusion does already exist, it isn't as visible as we'd be
making it here.

The difference between this kind of dependency and transitive ivy deps
is that it's per-project (settings dependency) rather than per-build
(classpath dependency). Also the semantics are different; in these
stories, when I add PlayPlugin explicitly to a project, then all
plugins on classpath which depend on PlayPlugin would be auto-added
unless I exclude them specifically. While in ivy deps, normally I
would figure out my leaf dependencies and that would then pull in
PlayPlugin. So we are doing "find and use all plugins which have deps
satisfied" rather than "find all deps of the plugins I've asked to
use."

One problem we don't solve is that certain plugins (e.g. PlayPlugin)
still have to be manually specified, because we don't have any
convention-over-configuration mechanism like "if the files in this
project look kinda like a Play project assume it is one" or other
heuristic. But we do greatly reduce the number of plugins which have
to be manually added per-project, e.g. there's no config for pgp
plugin anymore beyond addSbtPlugin.

Anyway thinking out loud but the short version is maybe we can drop
the word/concept "nature" by collapsing it with Plugin.

Havoc

[Naftoli Gugenheim]

Why strings?

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/CAFLqJkwB7K4Dz39rU9dYdYN-KSLmNqS07uemHMjcG7tTNei6ng%40mail.gmail.com.

[Josh Suereth]

On Tue, Feb 25, 2014 at 11:47 PM, Naftoli Gugenheim <nafto...@gmail.com> wrote:

Why strings?

We use strings *specifically* for error messages.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sbt-dev/CANpg8PBgbPMdncRFEn73XKVstRKrEkSAv%3D2tv4vhvD7xsWwx-g%40mail.gmail.com.

[Josh Suereth]

On Tue, Feb 25, 2014 at 10:39 AM, Havoc Pennington <h...@typesafe.com> wrote:

Hi,

I wonder if we'll primarily use the natures which are simply
equivalent to plugins. In the current code each plugin has a nature
defined based on the plugin object name.

In the third story
https://github.com/sbt/sbt/wiki/User-Stories:-Project-Natures#wiki-user-story-user-wants-to-create-an-sbt-plugin-that-enhances-play-projects
MyPlugin is simply conditional on PlayPlugin (`def select = PlayPlugin`)

In the second story I don't think there's a difference between Play
and PlayPlugin:
https://github.com/sbt/sbt/wiki/User-Stories:-Project-Natures#wiki-story-user-wants-to-aggregate-tasks-on-a-project-but-not-have-that-project-actually-have-any-behavior
the story uses Play but it could just be PlayPlugin. And Web could
just be WebPlugin (sbt-web). This removes any ambiguity (what does
"Web" mean? it means sbt-web)

For publish, what would Publish mean? It would just mean the built-in
plugin provided by sbt core which contains the publish task. Other
concepts of publish would have a different name (or live in a
different package at least).

One way to simplify then could be to remove these "virtual" or
"symlink" nature names such as Publish and Web and just say that
natures == plugins, though "plugin" here means a Plugin object
instance, not project library dependency.

It's not a bad idea.  I'm not 100% sold we won't need generic names.  E.g. if a plugin has two different "modes" of operation, it may want you to select which settings are included via a nature, rather than another mechanism.  However, I can buy the fact that in practice, we may *just* want to be controlling what plugins are added to which projects.

 

If you still wanted the idea of a nature with multiple
implementations, that could be done through a Plugin instance which
just provided task keys and no task implementations or something like
that, and people would then create other plugins that depended on that
one and provided concrete implementations. Or for a nature which was
"just" conceptual with no tasks, provide an empty Plugin instance...

Yeah, I assume things like sbt-web may provide 'default' tasks that do nothing that other plugins can wire into.  I'm thinking of providing a guide to 'writing interfaces in sbt".

 

The main advantage here would be to drop the word/concept of a
"nature" and just have "instances of Plugin"

Right, that is a nice simplification.

 

A remaining confusion is that "plugin" means both "Plugin object
instance" and "project library dependency as in addSbtPlugin." Though
that confusion does already exist, it isn't as visible as we'd be
making it here.

We'll actually be removing that in the future. The nomenclature I want to us is this:

• Build library -   jars/classes added via project/*.sbt or project/project/*.scala code.  These are libraries used to define your build.  They may contain one or more plugins.
• Plugin - Something which extends autoplugin

This would lead towards migrating from `addSbtPlugin` to `addSbtVersionedLibrary`.  However, i'm not sure that actually simplifies things.  The reality of AutoPlugins is that if we can have addding plugins to projects be "safe" again, then addSbtPlugin will *usually* be an accurate statement.

 

The difference between this kind of dependency and transitive ivy deps
is that it's per-project (settings dependency) rather than per-build
(classpath dependency). Also the semantics are different; in these
stories, when I add PlayPlugin explicitly to a project, then all
plugins on classpath which depend on PlayPlugin would be auto-added
unless I exclude them specifically. While in ivy deps, normally I
would figure out my leaf dependencies and that would then pull in
PlayPlugin. So we are doing "find and use all plugins which have deps
satisfied" rather than "find all deps of the plugins I've asked to
use."

Yes.  AutoPlugins are about adding plugins per-project, while project/plugins.sbt is about adding build-level dependencies.  I think you may be confusing things.  If you addSbtPlugin play, then *all* of its dependency are on the classpath.  However, the settings don't make it into any project unless you configure them *or* you have some kind of nature which automatically imports them.

Right now Ivy is handling the inclusion of JARs on the build-definition.  It is *not* managing the inclusion of settings on projects well.  That's why the current "best practices" involves *not* using the `projectSettings` method on plugins.   This feature can fix that issue, by allowing configuration of plugin inclusion.

In practice, because you need to directly reference a plugin object to depend on it via the natures feature, it means that if your projectSettings depends on projectSettings from another plugin, you *also* have an ivy dependency on it (otherwise how did you compile?).  So the net effect is we have this layer of "project-plugin" inclusion.   We try to use Ivy for as much as we possibly can and let the AutoPlugins/ProjectNatures guide how settings are wired into projects.

 

One problem we don't solve is that certain plugins (e.g. PlayPlugin)
still have to be manually specified, because we don't have any
convention-over-configuration mechanism like "if the files in this
project look kinda like a Play project assume it is one"  or other
heuristic. But we do greatly reduce the number of plugins which have
to be manually added per-project, e.g. there's no config for pgp
plugin anymore beyond addSbtPlugin.

Yes.  There will be plugins that have to be manually included.  The feature helps plugins which depend on others.  If the core sbt defaults *ALSO* become autoplugins, what we end up with is much safer automatic ordering of settings and automatic inclusion, as well as self-healing error messages.  The goal is that users should know HOW to fix their build so plugins are defined on the projects they want.

Since the AutoPlugins interface defines all settings and inclusion, we can actually index tasks which *don't* show up on projects and give you guidance on *why* they don't, and how to add them.   From my experience, this is a significant issue.

The only thing I haven't figured out yet is how to warn people when they've added a plugin to the wrong directory, as we don't even have a chance to index information about the plugin in that situation.

 

Anyway thinking out loud but the short version is maybe we can drop
the word/concept "nature" by collapsing it with Plugin.

A good point. We could just call them AutoPlugins.  Anyone else have thoughts about this?

[Josh Suereth]

On Mon, Feb 24, 2014 at 5:38 PM, Eric Torreborre <etorr...@gmail.com> wrote:

Hi Josh,

> One thing I've been debating with in sbt 1.0 is whether or not we can fragment all aspects of Defaults.scala into "plugins", enabled with default natures (besides build-level settings which would have to be defined first).  I think this would be good.

That's a very good idea.

> What we want to do is provide a default set of string natures that folks can use

I'm not sure how well this is going to work because we might have different definitions of what "Publish" means for example (but I might be wrong we need to see that on concrete examples).

Overall I'm still concerned with complexity. It appears to me that there are already mechanisms to deal with loading and dependencies in sbt: Configurations and ivy dependencies/configurations. Isn't it possible to use that to support all the user stories?

addSbtPlugin("com.typesafe.sbt" % "sbt-pgp" % "1.0" % "publish->default")

Eric.

As mentioned in the discussion with Havoc, there's an issue we've always faced here:  Build vs. project-level dependencies.

Right now addSbtPlugin is done at a BUILD level. if you have multiple projects within a build, you can't just add plugins to *all* projects.  AutoPlugins are the mechanism by which you control the setting-additions after resolution is complete.  Right now we *already* have the issue that adding a plugin doesn't mean settings are added to a project.  The AutoPlugin feature gives us the possibility of minimizing the impact of that.

We still use Ivy at the build-level, and it works as expected, including resolving version conflicts.   The AutoPlugins feature is more about including on projects.  Notice that it has no notion of version, because that's handled by ivy.