Re: [scala-user] How SBT blocks Scala adoption, and some suggestions

[Alec Zorab]

Moving to sbt mailing list, please remove scala user from replies.

I actually have a great deal of sympathy with both sides of this argument. Having been using scala and sbt for a while, sbt seems very easy to use to me, especially having tried to use maven for similar things at my last job.

That said, the pain of getting a new hire set up with a working system has made me realise just how horrible the process is at the moment. Case in point: after installing the typesafe stack the other day, I then had to go find a different copy of the launcher jar because the bundled one refused to play ball with 0.11.3.

Having got that working, I'm then sat trying to explain to someone what each of the seven files split across three directories does.

I understand the reasons and I'm certainly not suggesting I'm going to stop using sbt, but sbt certainly presents a barrier to entry for some people.

That said, being able to add a single line to a file, reload sbt and suddenly have magic dependency graphs diagrams is *fantastic*

On 5 Oct 2012 06:47, "Evan Chan" <vel...@gmail.com> wrote:

Thanks for the replies, I'll follow up on the sbt mailing list.  I thought this was a place to start, because the issue of adoption affects all Scala users.

Josh: Good to know you guys are working on many of those issues.   If someone could give me a hand, I'd want to understand what it takes to write an alternative build script engine around SBT.

Daniel:  ~/.sbt is pretty important.   There are certain build plugins, such as the idea plugin, which you don't want to pollute every project with, but put on a global level.   There are other global settings or organizational settings too, such as common repos.  And even if you don't change those very often, the pain of changing them tends to discourage new comers.

On Thursday, October 4, 2012 6:43:59 PM UTC-7, Daniel Sobral wrote:

I don't understand the complain about multiple files. You only need
*two* files: one for the project, another for the plugins. You can
choose between the simplified .sbt format or a full fledged .scala
format, but one can just set a standard for how to do things if people
are getting confused.

There's no need to put anything on ~/.sbt -- that's just something to
help people have things applied over all projects, and if rake doesn't
have something like that, then -- imho -- rake sucks.

I also wonder why do you change the project itself frequently enough
for that to be a hindrance, but I agree it's boringly slow when you
have to do it.

On Thu, Oct 4, 2012 at 8:13 PM, Evan Chan <vel...@gmail.com> wrote:
> Dear Scala community,
>
> I wanted to share my experience as someone trying to introduce Scala into
> our small company over the last year or so.   My goal is not to start a
> flame war, but to improve the tools (particularly SBT) we all love and use.
>
> We have always been a Ruby shop.  Everyone loves Ruby here.  We started
> exploring Scala a year ago for higher performance applications, as certain
> things in Ruby (such as MapReduce jobs) are too slow.   We use SBT, and a
> variety of editors (IDEA is the most common one, but some use vim, and some
> want to use Eclipse, so IDE standardization is out of the question).
>
> We have Scala in production now, but its use is relegated to high
> performance backend stuff.   There is at least one very vocal Scala
> detractor.  The feeling seems to be that the language is good, but the
> development and build tooling (esp SBT) is severely lacking, to put it very
> nicely.   It seems to be a huge hindrance to those who want to explore doing
> Scala projects, where the reaction is "why would you even consider setting
> up a Scala project????"   To summarize the complaints about SBT:
> - too many build files.  build.sbt, project/Build.scala,
> project/plugins.sbt, project/project/Build.scala, and then there's the same
> structure repeated in ~/.sbt.   People are very confused about what goes
> where.
> - related to the above, it's often impossible to understand where changes
> come in that impact build settings, due to the # of build files.
> - SBT is very slow whenever any change to the build project occurs, because
> it has to recompile and reiterate through dependencies of the build project,
> plus the same for the ~/.sbt project, etc.
>
> For comparison (if you are not familiar with rake, or gradle, etc.), rake
> builds things using one single Rakefile, it loads and builds instantly.
>
> Another point of comparison is Go!    The same people who hate Scala's build
> tools are in love with Go, because of its lightning compilation speeds and
> simple build (no build files needed).   So it's not a matter of static vs
> compiled, etc., but more a matter of tooling.
>
> However, it is clear that SBT is where the vast majority of build tooling in
> the Scala community is focused on, and is superior to its competition
> (gradle, buildr, etc.) in terms of actual build speed of source code itself
> (as opposed to build.sbt, etc.), triggered execution, community,
> scala-specific plugins, etc.
>
> So I have some suggestions for the Scala community regarding SBT, that would
> IMHO enhance its ease of use significantly and enable those of us wanting to
> push for Scala adoption in the startup sphere:
> 1. Clearly separate out SBT core functionality (compilation, dependency
> management, task execution) from the .sbt build files and environment
> 2. Make it easy to develop an alternative build file specification on top of
> the core functionality, so if someone wanted to develop a much easier,
> Rakefile-like alternative to .sbt files, they can
> 3.  Focus on the simplest scheme possible for build files.  Namely:
>    a) one build file, not multiple ones in multiple hierarchies
>    b) declare plugins and use them in the same file
>    c)  more of a convention for plugins, so I don't need separate lines
> like (addSbtPlugin(...);    import org.someplugin.blah;
> seq(someplugin.newSettings:_*);  )
> 4. I know many people love their Scala-based build files and may not like me
> for saying this, but I think dynamic languages are a much better fit for
> build scripts/files than compiled languages.   You get much more
> instantaneous response.
>
> What do you guys think?
>
> Peace,
> Evan



--
Daniel C. Sobral

I travel to the future all the time.

[Heiko Seeberger]

Evan,

On the one hand you want to have a single build file. On the other hand you tell that separation between project and certain "global" settings is important for you, i.e. you want two files. Will be pretty hard to fulfill your wish ;-)

Heiko

--
You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
To post to this group, send email to simple-b...@googlegroups.com.
To unsubscribe from this group, send email to simple-build-t...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-build-tool?hl=en.

[Erik Post]

Hi Alec, 

On Friday, 5 October 2012 09:36:35 UTC+2, Alec Zorab wrote:

Moving to sbt mailing list, please remove scala user from replies.

I actually have a great deal of sympathy with both sides of this argument. Having been using scala and sbt for a while, sbt seems very easy to use to me, especially having tried to use maven for similar things at my last job.

That said, the pain of getting a new hire set up with a working system has made me realise just how horrible the process is at the moment. Case in point: after installing the typesafe stack the other day, I then had to go find a different copy of the launcher jar because the bundled one refused to play ball with 0.11.3.

I've been using Paul Phillips' SBT launcher for this lately, which has saved me a couple of weeks of my life, and significantly increased its quality: https://github.com/paulp/sbt-extras/blob/master/sbt.

[Alec Zorab]

If only I didn't work in a windows-only company :)

[Alec Zorab]

ugh, apologies for the cross post and the double post. bad bad bad.

[Doug Tangren]

That said, the pain of getting a new hire set up with a working system has made me realise just how horrible the process is at the moment. Case in point: after installing the typesafe stack the other day, I then had to go find a different copy of the launcher jar because the bundled one refused to play ball with 0.11.3.

I'd like to point out that this is not sbt's fault. 

This happens with bundled software and it's up to the "bundler" to make sure their bundle is easy to use.

In my experience, it's better to not use bundled software but rather to use libraries independently. This removes the potential headache for installing things I don't want and that can cause friction.

I have never had a problem installing sbt with Mark's vanilla instructions in any version. For newer users, some of  Josh's great work with making platform specific package managers that wrap Mark's instructions helps also. Someone extracting just the bash part of paulps sbt-extras and rolling into something more discoverable, say the sbt github org and or repo would help.

Any problem I've ever had booting up sbt has never been sbt's fault. The first place to look if you have errors is probably your plugin configuration.

-d

[Eric Bowman]

We're rolling sbt out across a medium sized engineering organization. It's still early, but so far enthusiasm is fairly high.

We've made a single plugin that has the bulk of everything we need to do: it knows about our maven repos, it knows how to send release emails, and encapsulates as much as we could that's special about our particular organization.

A typical project needs 4 things:

build.sbt, with a line to pull in the settings they need (e.g., a jar project or an rpm project), and the name, organization, and libraryDependencies

version.sbt for the sbt-release plugin

project/build.sbt with standard boilerplate to pull in our plugin (and find a nexus repo to bootstrap from)

project/build.properties to specify the sbt version (so builds are reproducible)

Each of these files is small and pretty much completely self-explanatory to a newbie.

A much smaller team has taken the somewhat painful path of figuring out sbt well enough to do this. But now we pretty much like it, and we have a very simple (scripted) recipe to either convert a legacy ant/ivy build to sbt, or create a new project from scratch.

I would strongly advise against trying to roll out sbt in a company without taking a similar approach. You don't want to make everybody figure out everything all at once.  And in fact the same is true for other build tools ... have a small team figure out what people need to get started in as painless a way as possible.  Otherwise everybody uses it in anger, and everybody loses.

cheers,

Eric

[Mark Harrah]

The global plugins and multiple sbt versions problem is indeed a problem and is known. See for example: https://groups.google.com/d/topic/simple-build-tool/T2_Jk5kLdRM/discussion. A pull request is still welcome. There is a report for better error messages related to global plugins here: https://github.com/harrah/xsbt/issues/528. A pull request is welcome there as well, although I may eventually get to that myself.

For writing an alternative format, see http://www.scala-sbt.org/release/docs/Extending/Build-Loaders.html. There isn't a way to register the new build loader without a project/Build.scala yet, but that can be addressed once someone actually needs it. You can also just write some code that takes the base directory and constructs a Seq[Project]. Call it a plugin and users can just call it in their Build object. (Remove the need for Build.scala after the proof of concept.)

For the main/default/standard .sbt/.scala format, build.sbt+Build.scala are 99% pure Scala. The build.sbt syntax is basically (<Scala expression> <blank line>)+ and Build.scala is a standard Scala source file. The implementation complexity is much lower than a custom format and adds no new syntax to learn. However, there are many problems that have been raised over the past year (see the mailing list archives for discussions) that cannot be addressed by working with 99% pure Scala and would require an enhancement to the .sbt format (i.e., non-Scala syntax), such as:

1) cannot define multi-project builds in .sbt
2) cannot define vals, defs, or anything not a Scala expression
3) sbt version specified in separate project/build.properties file
4) plugins defined in a separate file, common to see confusion what to edit
...

What I don't think are problems are:

1) global configuration. It does require work to address some issues.
2) Types. I don't see less type safety in the build as an improvement. Types are rather fundamental to sbt's approach. You can of course use the Java scripting API to write build code (like tasks) in other jvm languages, although I'd be surprised if it is that useful.

I discussed a very rough proposal for an enhanced .sbt format at Scalathon, but there wasn't clear support for it. I developed it in conjunction with a rough introduction, but that drifted towards reference documentation as usual. It is all rough and incomplete and needs further development and discussion. I posted my very rough and incomplete notes on the wiki:

https://github.com/harrah/xsbt/wiki/Definition-Format-Enhancement.md

Please feel free to add to the problems sections and discuss it on this thread. At the end there is a discussion section where we can evolve the latest proposals/changes. This is an opportunity to revisit old assumptions concerning the build definition format. At a minimum, we can use this as an opportunity to document why things are done the way they are.

I don't think compatibility needs to be discussed at this point other than to say that it is assumed any new format needs to be implemented in a way that existing users do not need to change their builds. An initial prototype can probably be implemented and evolved as a plugin.

Finally, note that even if everyone loves this proposal or some other proposal, it is too much work for me to implement and maintain myself in addition to everything else in sbt. Please get involved if this is interesting to you.

Thanks,
Mark

[Evan Chan]

Eric,

This is pretty interesting for comparison.

So at our company we pretty much debuted Scala within one team.  The problem is that one or two people hated the experience so much that it kind of has poisoned Scala's reputation for the whole company.   Maybe we should have contained it to even less people first   :)

Re the 4 files you have, I believe you can cut it down to two:

• version.sbt is not needed if you use Twitter's sbt-package-dist plugin, which automagically finds the version string in your build.sbt or Build.scala and changes it for you.
• build.properties is not needed either if you use paulp's sbt-extras script, which lets you pass in the sbt version to use.

I made a sbt plugin for the company, but some people thought maybe we didn't even need plugins.sbt.   However trying to integrate all the needed plugins didn't work out.

-Evan

[Robin Green]

On Saturday, 6 October 2012 00:28:36 UTC+1, Evan Chan wrote:

• build.properties is not needed either if you use paulp's sbt-extras script, which lets you pass in the sbt version to use.

But if you are using (or think you might use in future - highly likely IMO) multiple versions of sbt, you still need to store somewhere which version of sbt you are using for a project, so you might as well use build.properties for that.

Also, the sbt plugin for Jenkins doesn't use that script (or Josh's fork of it) though that would probably be relatively easy to fix.

[Evan Chan]

Mark,

I had a quick glance at the Definition-Format-Enhancement.   I personally think this would be a big win, for beginners definitely, but for others too.    Clearer syntax, not needing a separate .properties file for the SBT version, being able to define multiple projects, and not needing the plugins.sbt are all wins.

I'd be interested to know what the arguments against are.

-Evan

[Mark Harrah]

Hi Evan,

On Mon, 8 Oct 2012 09:41:17 -0700 (PDT)
Evan Chan <e...@ooyala.com> wrote:

> Mark,
>
> I had a quick glance at the Definition-Format-Enhancement. I personally
> think this would be a big win, for beginners definitely, but for others
> too. Clearer syntax, not needing a separate .properties file for the SBT
> version, being able to define multiple projects, and not needing the
> plugins.sbt are all wins.

> I'd be interested to know what the arguments against are.

There is a section towards the bottom "Open issues and known problems/drawbacks". A big one is implementation complexity and effort to implement and maintain. Before, scalac did the real parsing and generated all of the error messages. The compilation process is complicated by mixing defs and vals, separate files, synthetic values and types, ... and it isn't standard compilation anymore.

On the user side, there is the danger of introducing another set of syntax/concepts to learn. Non-Scala syntax and names should be the same as Scala code where possible to avoid needing to learn two things, but then does it also make it confusing where Scala code can go? Also, it introduces a barrier between writing a build definition and writing a plugin. Right now, there is almost no difference. I think it is possible to iterate the proposal to work these things out, but they are still possible pitfalls.

-Mark

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/8j448X-ydvIJ.

[nafg]

Regarding the backticks, one possible concern is what if the scala code needs to use them? Would it be a possible alternative, to use indentation, like python?

Regarding the syntax in general, is there no way to get this much information in scala code, as a dsl, using macros? Perhaps it would require compiling more at a time (IIRC currently .sbt files only recompile changed lines), but is there any reason it's not possible?

Maybe something like:

"sbt.version=0.13.0"

def x = project {

  depends(a, c)

  aggregates(a,b,c)

  settings {

    demo := 33

    organization := "org.example"

    libraryDependencies += junitDep

    libraryDependencies += Seq( "org.example" % "demo" % "1.0" )
  }

}

A lot of things would be done differently than in the current proposal, perhaps, but in terms of the objectives, what's not possible?

[Mark Harrah]

On Thu, 11 Oct 2012 19:35:54 -0700 (PDT)
nafg <nafto...@gmail.com> wrote:

Thanks for taking a look and providing feedback.

> Regarding the backticks, one possible concern is what if the scala code
> needs to use them? Would it be a possible alternative, to use indentation,
> like python?

I think the proposal says use double backticks for escaping, but I'm not sure supporting backticks in the build file is really necessary. (I know sbt uses `package`, but I think backticks for reserved words or including spaces isn't a good idea for normal usage and I'd like to move away from it.)

> Regarding the syntax in general, is there no way to get this much
> information in scala code, as a dsl, using macros? Perhaps it would require
> compiling more at a time (IIRC currently .sbt files only recompile changed
> lines), but is there any reason it's not possible?

It is indeed a good idea to consider a syntax that is still 99% Scala.

In my experience, macros are a huge amount of work. There aren't really that many advantages for a more complicated macro and there are plenty of disadvantages. As examples of disadvantages, they effectively define their own rules and the implementation is going to be opaque to more people than non-macro code. Someone would have to show otherwise with a prototype.

> Maybe something like:
>
> "sbt.version=0.13.0"
>
> def x = project {
> depends(a, c)
> aggregates(a,b,c)
> settings {
> demo := 33
> organization := "org.example"
> libraryDependencies += junitDep
> libraryDependencies += Seq( "org.example" % "demo" % "1.0" )
> }
> }
>
> A lot of things would be done differently than in the current proposal,
> perhaps, but in terms of the objectives, what's not possible?

A minor point is that it would be a lazy val instead of a def. I assume you propose using the name of the def to set the id of the project?

Before, the data structures were immutable and thus trivially thread-safe. You can understand Project("x", file("x")).dependsOn(x) because they are all plain Scala with no secret mutable state, implicit parameters, or thread locals or whatever it would take to implement the above. This is a semantic difference that would count against this approach.

Next, this would change := to mutate something as well, which would complicate compatibility. Dropping the comma in the external proposal was done because it was easier. I think in a Scala-base approach that preserving the comma and making settings accept normal, repeated parameters would fix this.

depends, aggregates, and settings now need to be visible without prefix. They are only valid in the context of the project method and this can only be checked at runtime. So, I'm not sure this adds much over:

lazy val x = Project().
dependsOn(a,c).
aggregates(a,b,c).
settings(
demo := 33,
organization := "org.example"
)

The trailing . is perhaps unfortunate, but it uses the same Scala types as is currently used in a Build.scala file without any semantic tradeoffs or magic.

It seems worth pursuing, though. Things that need to be addressed for this to move to the next step of being a feasible alternative:

1) defining new task/setting keys
2) defining plugins in the same file
3) compilation/loading model, including how multiple files are merged/imported
4) preserving the ability to programmatically manipulate settings in a build.sbt file
5) unified source/binary/internal dependency declarations
6) whether it is possible to have the equivalent of:

settings in x,a,b
scalaVersion := "2.8.1"

-Mark

> > > To post to this group, send email to simple-b...@googlegroups.com<javascript:>.

> >
> > > To unsubscribe from this group, send email to

> > simple-build-t...@googlegroups.com <javascript:>.

> > > For more options, visit this group at
> > http://groups.google.com/group/simple-build-tool?hl=en.
> > >
> >
> >
>

> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/X380NMb_b3sJ.

[nafg]

On Friday, October 12, 2012 3:24:59 PM UTC-4, Mark Harrah wrote:

On Thu, 11 Oct 2012 19:35:54 -0700 (PDT)
nafg <nafto...@gmail.com> wrote:

Thanks for taking a look and providing feedback.

> Regarding the backticks, one possible concern is what if the scala code
> needs to use them? Would it be a possible alternative, to use indentation,
> like python?

I think the proposal says use double backticks for escaping, but I'm not sure supporting backticks in the build file is really necessary.  (I know sbt uses `package`, but I think backticks for reserved words or including spaces isn't a good idea for normal usage and I'd like to move away from it.)

> Regarding the syntax in general, is there no way to get this much
> information in scala code, as a dsl, using macros? Perhaps it would require
> compiling more at a time (IIRC currently .sbt files only recompile changed
> lines), but is there any reason it's not possible?

It is indeed a good idea to consider a syntax that is still 99% Scala.

In my experience, macros are a huge amount of work.  There aren't really that many advantages for a more complicated macro and there are plenty of disadvantages.  As examples of disadvantages, they effectively define their own rules and the implementation is going to be opaque to more people than non-macro code.  Someone would have to show otherwise with a prototype.

IIRC what I had in mind as the reason for using a macro, was to convert the coma-less format to a sequence of immutable expressions.

> Maybe something like:
>
> "sbt.version=0.13.0"
>
> def x = project {
>   depends(a, c)
>   aggregates(a,b,c)
>   settings {
>     demo := 33
>     organization := "org.example"
>     libraryDependencies += junitDep
>     libraryDependencies += Seq( "org.example" % "demo" % "1.0" )
>   }
> }
>
> A lot of things would be done differently than in the current proposal,
> perhaps, but in terms of the objectives, what's not possible?

A minor point is that it would be a lazy val instead of a def.  I assume you propose using the name of the def to set the id of the project?

Before, the data structures were immutable and thus trivially thread-safe.  You can understand Project("x", file("x")).dependsOn(x) because they are all plain Scala with no secret mutable state, implicit parameters, or thread locals or whatever it would take to implement the above.  This is a semantic difference that would count against this approach.

Next, this would change := to mutate something as well, which would complicate compatibility.  Dropping the comma in the external proposal was done because it was easier.  I think in a Scala-base approach that preserving the comma and making settings accept normal, repeated parameters would fix this.

See above, I did not intend mutability.

 

depends, aggregates, and settings now need to be visible without prefix.  They are only valid in the context of the project method and this can only be checked at runtime.

Unless a macro can provide implicit import; that is, convert project { ... } to project { implicit p: Project => ... }. Then the unprefixed depends etc. methods would take an implicit Project.

 

 So, I'm not sure this adds much over:

lazy val x = Project().
  dependsOn(a,c).
  aggregates(a,b,c).
  settings(
    demo := 33,
    organization := "org.example"
  )

Perhaps not. 

 

The trailing . is perhaps unfortunate,

You could instead write

lazy val x = (

  Project()

  dependsOn (a,c)

  settings( ... )

)

since it forces it to see it as infix notation across lines. How about:

lazy val x = project(

  _ dependsOn (a,c)

  settings( ... )

)

So project takes a Project => Project or such.

but it uses the same Scala types as is currently used in a Build.scala file without any semantic tradeoffs or magic.

It seems worth pursuing, though.  Things that need to be addressed for this to move to the next step of being a feasible alternative:

 1) defining new task/setting keys

Not familiar enough with writing tasks. If there's something challenging about how to write them, I can try to look into it more.

 

 2) defining plugins in the same file

Again, not familiar enough with them to have a suggestion, but if it's not straightforward I can try to puzzle about it.

 

 3) compilation/loading model, including how multiple files are merged/imported

Here there's nothing I can plead ignorance about, but it would help me think if you set out the problems this needs to solve.

 

 4) preserving the ability to programmatically manipulate settings in a build.sbt file

Not sure I understand.

 

 5) unified source/binary/internal dependency declarations

Didn't understand that one either. Do you mean unifying subprojects, both root and other, with libraryDependencies? That would be nice. Could  project dependencies actually be a Setting, like libraryDependencies?

 6) whether it is possible to have the equivalent of:

  settings in x,a,b
     scalaVersion := "2.8.1"

settings(x, a, b)(

  scalaVersion := "2.8.1"

)

def settings(projects: Project*)(settings: SettingDefinition*)

The problem is how x will dereference to the result of the project it defines after it's passed through the Project=>Project in the common block.

One possible solution: settings is a macro that does not actually evaluate the projects, but instead associates the Seq[SettingDefinition] with those projects by their symbol name. Statements, like settings invocations are invoked before lazy vals are initialized, so when project is invoked, it already has access to the common settings associated with its name.

An interesting idea: maybe we can unify the above. Maybe, just like we have Settings, which are "immutable mutations" -- that is, they alter State when run but they themselves are immutable --- we can have a similar scenario for Projects. So instead of a .sbt file being a collection of SettingDefinition, it now includes "ProjectDefinitions" --- expressions that can transform one or more projects --- too.

val x = project()

x dependsOn y

settings(x, y)(

  libraryDependencies += ...

)

scalaVersion := "2.10" // global SettingDefinition

I think that this approach would imply mutability, but not to settings, only to something like x, which is perhaps a ProjectVar. Personally I prefer the previous way. The only advantage to this way is that there's only one way to define settings, and it doesn't require a macro.

[Mark Harrah]

> > > Regarding the syntax in general, is there no way to get this much
> > > information in scala code, as a dsl, using macros? Perhaps it would
> > require
> > > compiling more at a time (IIRC currently .sbt files only recompile
> > changed
> > > lines), but is there any reason it's not possible?
> >
> > It is indeed a good idea to consider a syntax that is still 99% Scala.
> >
> > In my experience, macros are a huge amount of work. There aren't really
> > that many advantages for a more complicated macro and there are plenty of
> > disadvantages. As examples of disadvantages, they effectively define their
> > own rules and the implementation is going to be opaque to more people than
> > non-macro code. Someone would have to show otherwise with a prototype.
> >
>
> IIRC what I had in mind as the reason for using a macro, was to convert the
> coma-less format to a sequence of immutable expressions.

Ok.

> > depends, aggregates, and settings now need to be visible without prefix.
> > They are only valid in the context of the project method and this can only
> > be checked at runtime.
>
>
> Unless a macro can provide implicit import; that is, convert project { ...
> } to project { implicit p: Project => ... }. Then the unprefixed depends
> etc. methods would take an implicit Project.

The code has to typecheck before and after the macro call using just the signature of the macro (like a normal method call). I believe it would require untyped macros as described in the discussion of a related issue:

https://issues.scala-lang.org/browse/SI-5774

> > So, I'm not sure this adds much over:
> >
> > lazy val x = Project().
> > dependsOn(a,c).
> > aggregates(a,b,c).
> > settings(
> > demo := 33,
> > organization := "org.example"
> > )
> >
> >
> Perhaps not.
>
>
> > The trailing . is perhaps unfortunate,
>
>
> You could instead write
>
> lazy val x = (
> Project()
> dependsOn (a,c)
> settings( ... )
> )
>
> since it forces it to see it as infix notation across lines.

That's true.

> How about:
>
> lazy val x = project(
> _ dependsOn (a,c)
> settings( ... )
> )
>
> So project takes a Project => Project or such.

That is also a possibility.

> but it uses the same Scala types as is currently used in a Build.scala file
> > without any semantic tradeoffs or magic.
> >
> > It seems worth pursuing, though. Things that need to be addressed for
> > this to move to the next step of being a feasible alternative:
> >
> > 1) defining new task/setting keys
> >
>
> Not familiar enough with writing tasks. If there's something challenging
> about how to write them, I can try to look into it more.

All I mean here is that when you define a task/setting, you have to create a key:

val myTask = TaskKey[SomeType]("my-task", "A description")

The original proposal adds a syntax for this:

task myTask: SomeType
A description

> > 2) defining plugins in the same file
> >
>
> Again, not familiar enough with them to have a suggestion, but if it's not
> straightforward I can try to puzzle about it.

Plugins are basically libraries used for the build definition. They need to be configured before the main build definition is loaded. This is why plugins are configured in project/plugins.sbt (or project/project/Build.scala). That project/ is built first and its classpath used for build.sbt and project/Build.scala.

So, the original proposal was to have:

... normal build configuration ...

project plugins
... plugin project configuration ...

but the project plugins section is compiled and loaded first and separately from everything else.

> > 3) compilation/loading model, including how multiple files are
> > merged/imported
> >
>
> Here there's nothing I can plead ignorance about, but it would help me
> think if you set out the problems this needs to solve.

Basically, what gets fed to the compiler when? The existing code can take either a Scala expression or a Scala source file and pass it to the compiler. It can do expressions (as in the settings in a build.sbt) rather efficiently when the build definition classpath and imports haven't changed (basically, plugins haven't changed), but a full source file gets the normal compilation.

> > 4) preserving the ability to programmatically manipulate settings in a
> > build.sbt file
> >
> Not sure I understand.

As an example of what you can do right now with .sbt,

$ sbt
> set libraryDependencies += "net.databinder.dispatch" %% "dispatch-core" % "0.9.2"
> session save

'session save' can safely write the new setting to build.sbt because of the way build.sbt is structured. It might be reasonable to say "unsupported", but it would be something lost from the current .sbt format.

> > 5) unified source/binary/internal dependency declarations
> >
>
> Didn't understand that one either. Do you mean unifying subprojects, both
> root and other, with libraryDependencies? That would be nice. Could
> project dependencies actually be a Setting, like libraryDependencies?

The original proposal has this example:

project a
dependsOn
b
uri("...")

"org.example" % "demo" % "1.0"

project b

The dependency list includes an internal dependency on b, a source dependency on some remote project, and a managed binary dependency on 'demo'.

Project dependencies can't be a setting because certain information is needed from projects before loading settings. (If you define all of your projects up front, you can technically do this now, but it isn't really an improvement and only helps if you need the flexibility.)

I don't think it is impossible to support all dependencies in a setting, but it would take time/effort and I'm not sure if it is an improvement. I was just planning to put the declared binary dependencies in libraryDependencies automatically.

> 6) whether it is possible to have the equivalent of:
> >
> > settings in x,a,b
> > scalaVersion := "2.8.1"
> >
>
> settings(x, a, b)(
> scalaVersion := "2.8.1"
> )
>
> def settings(projects: Project*)(settings: SettingDefinition*)
>
> The problem is how x will dereference to the result of the project it
> defines after it's passed through the Project=>Project in the common block.
> One possible solution: settings is a macro that does not actually evaluate
> the projects, but instead associates the Seq[SettingDefinition] with those
> projects by their symbol name. Statements, like settings invocations are
> invoked before lazy vals are initialized, so when project is invoked, it
> already has access to the common settings associated with its name.

The basic idea would work. You don't actually need the full Project when you do

scalaVersion in x := "2.8.1"

just the String id. The above is a convenience for:

scalaVersion in LocalProject(x.id) := "2.8.1"

It would fail on something like:

val x = project(...)
val y = x

settings(y, ...)

> An interesting idea: maybe we can unify the above. Maybe, just like we have
> Settings, which are "immutable mutations" -- that is, they alter State when
> run but they themselves are immutable ---

Note that you don't need to think of settings as mutable. They are a DAG, so there are no cycles and bindings to the map of setting->computed value are only added as settings are evaluated. So, settings are no more mutable than a lazy val.

> we can have a similar scenario
> for Projects. So instead of a .sbt file being a collection of
> SettingDefinition, it now includes "ProjectDefinitions" --- expressions
> that can transform one or more projects --- too.
>
> val x = project()
>
> x dependsOn y
>
> settings(x, y)(
> libraryDependencies += ...
> )
>
> scalaVersion := "2.10" // global SettingDefinition
>
> I think that this approach would imply mutability, but not to settings,
> only to something like x, which is perhaps a ProjectVar. Personally I
> prefer the previous way. The only advantage to this way is that there's
> only one way to define settings, and it doesn't require a macro.

One of the reasons I used a non-Scala container syntax in the proposal is that you stop worrying about how to hack Scala syntax to get the right combination of syntax and semantics. Regardless, I think you can keep immutability if so desired by having some BuildConfiguration type that represents the build definition in progress and have each line be a transformation on that BuildConfiguration. So,

x dependsOn y

would generate a function BuildConfiguration => BuildConfiguration, which would be the expected type of all top-level expressions. (not sure how that interacts with top-level settings, though)

[Mark Harrah]

Hi Phil,

On Mon, 15 Oct 2012 15:19:10 -0700 (PDT)
Philip Wills <othe...@gmail.com> wrote:

> Sorry if this is a bit of non-sequitur from the conversation up to now, but
> I wanted to provide some feedback on the ideas in
> https://github.com/harrah/xsbt/wiki/Definition-Format-Enhancement.md

Feedback is certainly welcome.

> Combining build.properties with a build definition and a plugin definition
> would be a boon, but I don't think it would be worth it if it meant using
> something that was similar to, but not quite Scala with it's own fairly
> involved parsing rules. Mark mentioned this earlier in the thread and I
> think it's a real concern.

The build.properties change is one thing that can probably be done regardless of whether .sbt is otherwise enhanced, but sbt.version would have to go in build.sbt and not someothername.sbt (the launcher needs a single name) and not in Build.scala (it wouldn't be a Scala file anymore). So, if you only had a Build.scala, you now need a build.sbt just for sbt.version and it doesn't improve your situation over project/build.properties.

Having a plugin definition in the same file as the build rules out that file as being a pure Scala file*. The plugin definition has to be compiled and loaded and then used to compile and load the build definition. So, there has to be some non-Scala syntax somewhere that separates the two.

If the parsing rules are difficult to understand, let's improve them. Elsewhere in this thread is a proposal to have it more Scala-based, for example. Consider the original proposal along the lines of a Scala template. Most things are Scala expressions glued together by non-Scala syntax. The point is to be easier to document, easier to present to new users, and address the problems in the beginning of the proposal. If the current proposal doesn't work out, fine, but I think it is worthwhile to consider alternatives to the current situation.

> Whilst the variety of files can cause some confusion, in general the
> problems that we have with SBT can be broadly arranged into 3 groups:
>
> * Oh my .ivy2 directory is screwed again - great to see the recent work on
> this

Yes, you have probably seen this, but I'll repeat the link anyway:

www.scala-sbt.org/release/docs/Detailed-Topics/Dependency-Management-Flow.html

> * Global plugins conflicting with build specific versions of SBT and even
> even project specific plugins clashing, c.f. Play and the IntelliJ plugin

I have no objection to versioned global directories by default. A pull request is welcome:

https://groups.google.com/d/msg/simple-build-tool/T2_Jk5kLdRM/zP4-RYYbKSEJ

There is also:

https://github.com/harrah/xsbt/issues/528

Help here is welcome as well.

> * The difficulty of understanding SBT's model of Setting/Task
> transformations. It took me a long time to get my head round how map and
> apply are used in this context and we have a lot of developers who have
> been coding primarily in Scala for a decent length of time, but recoil at
> the thought of doing anything not completely trivial in SBT.

I don't know if you mean the syntax, but if so, there is a branch with an improved syntax for defining tasks and settings that doesn't need the map/apply distinction or the <-prefixed methods like <<=. The branch is feature/task-syntax, it requires 2.10.0-RC1, and you can see Defaults.scala for a comparison with the current syntax:

https://github.com/harrah/xsbt/commit/fa0c303f679c15e346fdfcb4

> I don't know if there's a way in future versions that SBT could check
> whether a global plugin is incompatible and skip it with a warning if so
> and even if it were possible it might just confuse things further.

The situation with Play/IDE plugins doesn't sound good, but I don't know if there is/should be a technical solution or if it is a social problem.

> With regards to the settings/tasks model, I don't have any helpful
> suggestions, as the docs are much improved and I don't expect such a core
> concept to change.

It won't change quickly or in a way that causes excessive issues for existing builds, but it can still help to identify better approaches.

-Mark

> Hope that's helpful,
>
> Phil Wills, software architect, guardian.co.uk

>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/65aefPwRj0UJ.

[Jason Zaugg]

On Thursday, October 18, 2012 4:10:45 PM UTC+2, Mark Harrah wrote:

Having a plugin definition in the same file as the build rules out that file as being a pure Scala file*.  The plugin definition has to be compiled and loaded and then used to compile and load the build definition.  So, there has to be some non-Scala syntax somewhere that separates the two.

Right now, I can edit .sbt files in IntelliJ with autocompletion / navigation. This is a big benefit (albeit for a small percentage of SBT users.)

 

If the parsing rules are difficult to understand, let's improve them.  Elsewhere in this thread is a proposal to have it more Scala-based, for example.  Consider the original proposal along the lines of a Scala template.  Most things are Scala expressions glued together by non-Scala syntax.  The point is to be easier to document, easier to present to new users, and address the problems in the beginning of the proposal.  If the current proposal doesn't work out, fine, but I think it is worthwhile to consider alternatives to the current situation.

If there is really such an objection to having the three files (build.properties / plugins.sbt, and build.scala), you could embed the first two in a comment in a magic comment at the top of the third, in the same spirit as the shebang of a unix script.

 

I don't know if you mean the syntax, but if so, there is a branch with an improved syntax for defining tasks and settings that doesn't need the map/apply distinction or the <-prefixed methods like <<=.  The branch is feature/task-syntax, it requires 2.10.0-RC1, and you can see Defaults.scala for a comparison with the current syntax:

 https://github.com/harrah/xsbt/commit/fa0c303f679c15e346fdfcb4

This is going to be a killer feature of 0.13, if you can emit decent error messages when things are wrongly defined. (e.g. "you can't use the value of a Task to initialize a Setting".

-jason

[Mark Harrah]

On Thu, 18 Oct 2012 09:40:03 -0700 (PDT)
Jason Zaugg <jza...@gmail.com> wrote:

> On Thursday, October 18, 2012 4:10:45 PM UTC+2, Mark Harrah wrote:
> >
> > Having a plugin definition in the same file as the build rules out that
> > file as being a pure Scala file*. The plugin definition has to be compiled
> > and loaded and then used to compile and load the build definition. So,
> > there has to be some non-Scala syntax somewhere that separates the two.
> >
>
> Right now, I can edit .sbt files in IntelliJ with autocompletion /
> navigation. This is a big benefit (albeit for a small percentage of SBT
> users.)

If it isn't it in the problems section already (maybe we only discussed it in this thread), you can add it. See also Naftoli's proposal, which is still Scala fragments and which I think would retain the ability to edit .scala files. The problem will always be plugin definitions, as you bring up.

Note that I personally have no intention of implementing the proposal as is at this time, at least not without a lot of help. At this stage, I wouldn't want to exclude a solution just because I can't implement it myself, though. It might be appealing to enough people that it can actually be implemented by them with IDE support as well. This is all hypothetical and I don't really expect it to happen for the current proposal as it stands, but I'd still like to see alternatives like it considered/discussed.

> > If the parsing rules are difficult to understand, let's improve them.
> > Elsewhere in this thread is a proposal to have it more Scala-based, for
> > example. Consider the original proposal along the lines of a Scala
> > template. Most things are Scala expressions glued together by non-Scala
> > syntax. The point is to be easier to document, easier to present to new
> > users, and address the problems in the beginning of the proposal. If the
> > current proposal doesn't work out, fine, but I think it is worthwhile to
> > consider alternatives to the current situation.
> >
>
> If there is really such an objection to having the three files
> (build.properties / plugins.sbt, and build.scala), you could embed the
> first two in a comment in a magic comment at the top of the third, in the
> same spirit as the shebang of a unix script.

This is what I meant when I referenced the Scripts page, but it has other problems. I seem to remember scalariform reformats comments, for example. You still have to modify tools to understand it, including IDEs. Yet another possibility is to allow plugins.sbt in the root.

> > I don't know if you mean the syntax, but if so, there is a branch with an
> > improved syntax for defining tasks and settings that doesn't need the
> > map/apply distinction or the <-prefixed methods like <<=. The branch is
> > feature/task-syntax, it requires 2.10.0-RC1, and you can see Defaults.scala
> > for a comparison with the current syntax:
> >
> > https://github.com/harrah/xsbt/commit/fa0c303f679c15e346fdfcb4
> >
>
> This is going to be a killer feature of 0.13, if you can emit decent error
> messages when things are wrongly defined. (e.g. "you can't use the value of
> a Task to initialize a Setting".

Yes, one nice thing about this macro approach is the ability to be more specific with some error messages. This is what feature/task-syntax does now:

> set version := javacOptions.value.head
<set>:1: error: A setting cannot depend on a task
version := javacOptions.value.head
^

-Mark

> -jason

>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/simple-build-tool/-/oNAZBXyJKqAJ.

[Philip Wills]

Hi Mark,

Syntax isn't the whole thing, but first impressions of that task-syntax branch look like it's a positive step to addressing some of those issues.

I agree that some issues are social, rather than technical and was trying to express that one reason build tools often get stick is because they are often where the clashes between other dependencies manifest.

I'm still a bit nervous about how the boundaries between .sbt file specific glue and normal Scala might be made obvious, but am a fan of the Play templates which take a conceptually similar approach.

Phil

 https://github.com/harrah/xsbt/ecific plugins clashing, c.f. Play anissues/528