Various inconsistencies going forward...

[Ken Scambler]

Hi SBTers,

I'm belatedly taking a deeper interest in the post-0.7 sbt, and I'm simultaneously excited and impressed by it's great flexibility and power, and repelled by the amount of implementation complexity that has leaked into the user interface.  Anyway, you've heard all that before.  

I see that 0.13 entails huge improvements for the user -- hooray!   In particular a source of inconsistency between the command line and Scala has been wiped out: the case/camelCase distinction.  I was wondering whether there's any trains of thought ongoing around some further inconsistencies:

- Scopes and ScopedKeys have a DSL in Scala code like settingKey in (myBuild, myConfig, my Task), but are referred to on the prompt in a concise but cryptic format: {build}project/config:task::key.    Is it worth considering making the canonical command-line format look like 'foo in (a,b,c)', or making the Scala one become 'scope"{build}projectid/config:task::key"', where "scope" is a string format macro enforced at compile time?

- Dependencies are expressed with % and %%-separated strings in scala, but have a different canonical format on the prompt, something like 'com.foo#someproduct;1.6.1'.   Is it worth considering making the command-line one look like 'author%product%version', or making the Scala one like dep"author#product;version", which might also be a "dep" string-format macro?

- .sbt files get placed in the directory of the project they describe, yet .scala build files go into the meta-project in "project".  When you get to a couple of meta-layers (like when you need to declare a github source plugin dependency in /project/project/PluginDefs.scala), then it gets very confusing to keep track of which file is configuring which layer of project.

- You can change the settings of all your projects (including addSbtPlugin() settings), by editing build.sbt in ~/.sbt, which is supposedly a project like any other.  But if you want to do this through Scala files, then the meta-project is called "plugins" instead of "project".


I suspect these points would come up on the list all the time, but I was unable to find them in my brief search.  I apologise in advance if that's the case.

Cheers,
Ken

[Mark Harrah]

Hi Ken,

On Tue, 29 Jan 2013 14:19:42 +1100
Ken Scambler <ken.sc...@gmail.com> wrote:

> Hi SBTers,
>
> I'm belatedly taking a deeper interest in the post-0.7 sbt, and I'm
> simultaneously excited and impressed by it's great flexibility and power,
> and repelled by the amount of implementation complexity that has leaked
> into the user interface. Anyway, you've heard all that before.
>
> I see that 0.13 entails huge improvements for the user -- hooray! In
> particular a source of inconsistency between the command line and Scala has
> been wiped out: the case/camelCase distinction. I was wondering whether
> there's any trains of thought ongoing around some further inconsistencies:

Thanks for taking the time to provide feedback. I've tried to provide background in the replies below. The comments aren't intended to kill discussion, but provide the history/rationale behind things. It is not too late to consider things like you suggest for 0.13.0. I don't think I will personally have time to implement them, even as prototypes, so pull requests are welcome and appreciated.

> - Scopes and ScopedKeys have a DSL in Scala code like settingKey in
> (myBuild, myConfig, my Task), but are referred to on the prompt in a
> concise but cryptic format: {build}project/config:task::key. Is it worth
> considering making the canonical command-line format look like 'foo in
> (a,b,c)', or making the Scala one become
> 'scope"{build}projectid/config:task::key"', where "scope" is a string
> format macro enforced at compile time?

The {build} part is not shown except when source dependencies are used and task:: is fairly rare. In the single project case, you don't see project/ and so it is usually just key or config:key. The config:key was intended to be familiar to Maven users.

The command line format was designed without spaces because commands typically separate arguments by spaces. It was also intended to be unambiguous to assist with tab completion, hence the :: to identify a task scope. The Scala syntax is rather heavily overloaded and I'm not sure it would work too well in the command line environment. Some day it might be possible to use Dynamic+macros to get subProject.Test.testOnly as the Scala syntax so that is another option and one that could merge the syntaxes.

I'm not opposed to trying out the scope string format macro as an alternative. It seems to me that one rough edge would be with scoped keys with only an implied scope. For example,

libraryDependencies += ...

v.

scope"libraryDependencies" += ...

Perhaps it will work out better overall, though. I'd certainly be interested in a pull request with this implemented so that it could be evaluated in practice.

> - Dependencies are expressed with % and %%-separated strings in scala, but
> have a different canonical format on the prompt, something like
> 'com.foo#someproduct;1.6.1'. Is it worth considering making the
> command-line one look like 'author%product%version', or making the Scala
> one like dep"author#product;version", which might also be a "dep"
> string-format macro?

I'd vote for making them consistent as well. Unfortunately, the output format comes from Ivy. It might be possible to make most output strings be uniform, not all. Perhaps that would be enough of an improvement.

In my opinion, the ideal syntax is colon separated. Those don't work as Scala method names, which is why I went with %. It is common for some parts of the dependency to be defined in Scala and I think it would be awkward to use those in an interpolated string.

Now is a good time for experiments, though, and I'd be happy to include pull requests in milestones. For the first beta, they can be moved to a plugin if not ready, removed if they don't work out, or kept if they do.

> - .sbt files get placed in the directory of the project they describe, yet
> .scala build files go into the meta-project in "project". When you get to
> a couple of meta-layers (like when you need to declare a github source
> plugin dependency in /project/project/PluginDefs.scala), then it gets very
> confusing to keep track of which file is configuring which layer of project.

With the enhancements to the .sbt file format, you can declare the source plugin dependency in project/plugins.sbt. I don't think there will be too many uses of project/project/ with that ability. Suggestions are welcome in any case.

> - You can change the settings of all your projects (including
> addSbtPlugin() settings), by editing build.sbt in ~/.sbt, which is
> supposedly a project like any other. But if you want to do this through
> Scala files, then the meta-project is called "plugins" instead of "project".

I don't think ~/.sbt is described as a normal project, at least not on Detailed-Topics/Global-Settings. If it is or if I've said that somewhere, that should be corrected. ~/.sbt/plugins/ is a real project and is loaded as such. ~/.sbt is never loaded like a normal project though. Only the ~/.sbt/*.sbt files are loaded as settings. src/main/scala, lib, etc... are not even considered. (I'm not sure what would be done with them if they were.)

So, I don't know that ~/.sbt is usefully considered as a normal project. It is probably true that if ~/.sbt/plugins/ were called ~/.sbt/project/, you could start sbt in ~/.sbt and things would work as expected. If this is actually useful, the renaming might make sense. Perhaps it could be used for debugging.

> I suspect these points would come up on the list all the time, but I was
> unable to find them in my brief search. I apologise in advance if that's
> the case.

Not a problem. Google Groups search is mediocre and I often have trouble finding things myself. I don't think anyone has made these specific suggestions and in particular, I don't think potential applications of string interpolation have been considered on this list yet.

-Mark

> Cheers,
> Ken
>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
> To post to this group, send email to simple-b...@googlegroups.com.
> Visit this group at http://groups.google.com/group/simple-build-tool?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Ken Scambler]

Thanks very much for your generous response Mark. 

I certainly don't want to put more stuff on your plate -- I'm very appreciative of the huge amount of effort going to 0.13, and I'm more than happy to contribute to the extent I'm capable of doing so, which is modest so far.  I'll have a play with String formats for dependencies and scopes next time I'm near a compiler. 

More comments inline:

The {build} part is not shown except when source dependencies are used and task:: is fairly rare.  In the single project case, you don't see project/ and so it is usually just key or config:key.  The config:key was intended to be familiar to Maven users.

While I used the word "cryptic", the format is actually not hard to learn; I don't really have a problem with it.  It's more the double-learning that has to happen between the prompt and the scala files that I was concerned about.  It'll take some more thinking for me to make more constructive suggestions/contributions though. :)
 

The command line format was designed without spaces because commands typically separate arguments by spaces.  It was also intended to be unambiguous to assist with tab completion, hence the :: to identify a task scope.  The Scala syntax is rather heavily overloaded and I'm not sure it would work too well in the command line environment.  Some day it might be possible to use Dynamic+macros to get subProject.Test.testOnly as the Scala syntax so that is another option and one that could merge the syntaxes.

I'm not opposed to trying out the scope string format macro as an alternative.  It seems to me that one rough edge would be with scoped keys with only an implied scope.  For example,

  libraryDependencies += ...

v.

  scope"libraryDependencies" += ...

Perhaps it will work out better overall, though.  I'd certainly be interested in a pull request with this implemented so that it could be evaluated in practice.

I see your point, the Scala DSL with spaces wouldn't really work on the command line.  My mental model of the 3-dimensional cascading scope model still has some cobwebs, but I'll have a think about the implied scope thing.
 

> - Dependencies are expressed with % and %%-separated strings in scala, but
> have a different canonical format on the prompt, something like
> 'com.foo#someproduct;1.6.1'.   Is it worth considering making the
> command-line one look like 'author%product%version', or making the Scala
> one like dep"author#product;version", which might also be a "dep"
> string-format macro?

I'd vote for making them consistent as well.  Unfortunately, the output format comes from Ivy.  It might be possible to make most output strings be uniform, not all.  Perhaps that would be enough of an improvement.

I guess it doesn't matter, I like the more concise syntax better anyhow.
 

In my opinion, the ideal syntax is colon separated.  

 

 I completely agree with this; string formats are ideal for this kind of thing IMHO.  Not sure how to reproduce the %% feature though.

Those don't work as Scala method names, which is why I went with %.  It is common for some parts of the dependency to be defined in Scala and I think it would be awkward to use those in an interpolated string.

Isn't that what interpolated strings are for, to make it not awkward?  I might look weird for the scope syntax though, since it uses {} in non-interpolated bits.

With the enhancements to the .sbt file format, you can declare the source plugin dependency in project/plugins.sbt.  I don't think there will be too many uses of project/project/ with that ability.  Suggestions are welcome in any case.

I actually can't think of any better suggestion.  Moving *.sbt files to project would solve the inconsistency, but then you would lose the convenience of having an immediately visible config file.

I don't think ~/.sbt is described as a normal project, at least not on Detailed-Topics/Global-Settings.  If it is or if I've said that somewhere, that should be corrected.  ~/.sbt/plugins/ is a real project and is loaded as such.  ~/.sbt is never loaded like a normal project though.  Only the ~/.sbt/*.sbt files are loaded as settings.  src/main/scala, lib, etc... are not even considered.  (I'm not sure what would be done with them if they were.)

So, I don't know that ~/.sbt is usefully considered as a normal project.  It is probably true that if ~/.sbt/plugins/ were called ~/.sbt/project/, you could start sbt in ~/.sbt and things would work as expected.  If this is actually useful, the renaming might make sense.  Perhaps it could be used for debugging.

Yes, I think I just assumed that, because plugin definitions went in the subdirectory, and regular settings went in ~/.sbt/build.sbt.   It quacks a little bit like a duck, and I had the turtles-all-the-way-down thing in my head and was pattern-matching for things to plug into that mental model.

Cheers,
Ken

[Mark Harrah]

On Tue, 29 Jan 2013 18:53:38 -0800 (PST)
Ken Scambler <ken.sc...@gmail.com> wrote:

> Thanks very much for your generous response Mark.
>
> I certainly don't want to put more stuff on your plate -- I'm very
> appreciative of the huge amount of effort going to 0.13, and I'm more than
> happy to contribute to the extent I'm capable of doing so, which is modest
> so far. I'll have a play with String formats for dependencies and scopes
> next time I'm near a compiler.
>
> More comments inline:
>
> >
> > The {build} part is not shown except when source dependencies are used and
> > task:: is fairly rare. In the single project case, you don't see project/
> > and so it is usually just key or config:key. The config:key was intended
> > to be familiar to Maven users.
> >
> > While I used the word "cryptic", the format is actually not hard to learn;
> I don't really have a problem with it. It's more the double-learning that
> has to happen between the prompt and the scala files that I was concerned
> about. It'll take some more thinking for me to make more constructive
> suggestions/contributions though. :)

I'd be happier with a single syntax and one thing to explain as well. I think it is mainly the {} that I find questionable, but it is a bit hard to embed a URI.

> > The command line format was designed without spaces because commands
> > typically separate arguments by spaces. It was also intended to be
> > unambiguous to assist with tab completion, hence the :: to identify a task
> > scope. The Scala syntax is rather heavily overloaded and I'm not sure it
> > would work too well in the command line environment. Some day it might be
> > possible to use Dynamic+macros to get subProject.Test.testOnly as the Scala
> > syntax so that is another option and one that could merge the syntaxes.
> >
> > I'm not opposed to trying out the scope string format macro as an
> > alternative. It seems to me that one rough edge would be with scoped keys
> > with only an implied scope. For example,
> >
> > libraryDependencies += ...
> >
> > v.
> >
> > scope"libraryDependencies" += ...
> >
> > Perhaps it will work out better overall, though. I'd certainly be
> > interested in a pull request with this implemented so that it could be
> > evaluated in practice.
> >
>
> I see your point, the Scala DSL with spaces wouldn't really work on the
> command line. My mental model of the 3-dimensional cascading scope model
> still has some cobwebs, but I'll have a think about the implied scope thing.

Here I don't mean delegation, only parsing. If you were to be fully explicit, you'd write:

libraryDependencies in (ThisProject, Global, Global) += ...

That's a bit verbose, so unspecified axes have a default. It is like an implied 'this' but with three parts to it.

(Tangent: if you do 'eval name.scopedKey.toString' at the command prompt, you'll see the This objects that represent unspecified axes. These get properly resolved before evaluation. Change it to (name in Compile)... and you'll see that the configuration axis is now explicit.)

> > > - Dependencies are expressed with % and %%-separated strings in scala,
> > but
> > > have a different canonical format on the prompt, something like
> > > 'com.foo#someproduct;1.6.1'. Is it worth considering making the
> > > command-line one look like 'author%product%version', or making the Scala
> > > one like dep"author#product;version", which might also be a "dep"
> > > string-format macro?
> >
> > I'd vote for making them consistent as well. Unfortunately, the output
> > format comes from Ivy. It might be possible to make most output strings be
> > uniform, not all. Perhaps that would be enough of an improvement.
> >
> > I guess it doesn't matter, I like the more concise syntax better anyhow.
>
>
> > In my opinion, the ideal syntax is colon separated.
>
>
> I completely agree with this; string formats are ideal for this kind of
> thing IMHO. Not sure how to reproduce the %% feature though.

I'm not sure it needs to be reproduced in the string syntax. You might just stick with the main four (including configuration) and then the cross method:

dep"org.example:demo:1.0:compile" cross(CrossVersion.Binary)

and probably make CrossVersion.Binary the default so it could be just:

dep"org.example:demo:1.0:compile" cross()

> Those don't work as Scala method names, which is why I went with %. It is
> > common for some parts of the dependency to be defined in Scala and I think
> > it would be awkward to use those in an interpolated string.
> >
> > Isn't that what interpolated strings are for, to make it not awkward? I
> might look weird for the scope syntax though, since it uses {} in
> non-interpolated bits.

Yeah, it might be fine for dependencies:

dep"org.example:demo:$demoVersion:compile"

> > With the enhancements to the .sbt file format, you can declare the source
> > plugin dependency in project/plugins.sbt. I don't think there will be too
> > many uses of project/project/ with that ability. Suggestions are welcome
> > in any case.
> >
> > I actually can't think of any better suggestion. Moving *.sbt files to
> project would solve the inconsistency, but then you would lose the
> convenience of having an immediately visible config file.

Yes and another disadvantage would be that plugins would go in project/project/plugins.sbt. (This doesn't consider how to change it in a compatible way either.)

> > I don't think ~/.sbt is described as a normal project, at least not on
> > Detailed-Topics/Global-Settings. If it is or if I've said that somewhere,
> > that should be corrected. ~/.sbt/plugins/ is a real project and is loaded
> > as such. ~/.sbt is never loaded like a normal project though. Only the
> > ~/.sbt/*.sbt files are loaded as settings. src/main/scala, lib, etc... are
> > not even considered. (I'm not sure what would be done with them if they
> > were.)
> >
> > So, I don't know that ~/.sbt is usefully considered as a normal project.
> > It is probably true that if ~/.sbt/plugins/ were called ~/.sbt/project/,
> > you could start sbt in ~/.sbt and things would work as expected. If this
> > is actually useful, the renaming might make sense. Perhaps it could be
> > used for debugging.
> >
> > Yes, I think I just assumed that, because plugin definitions went in the
> subdirectory, and regular settings went in ~/.sbt/build.sbt. It quacks a
> little bit like a duck, and I had the turtles-all-the-way-down thing in my
> head and was pattern-matching for things to plug into that mental model.

Conceptually, I think you can think of it that way, as long as there is a caveat that it isn't quite a normal project.

[Ken Scambler]

Thanks again Mark.

Here I don't mean delegation, only parsing.  If you were to be fully explicit, you'd write:

  libraryDependencies in (ThisProject, Global, Global) += ...

That's a bit verbose, so unspecified axes have a default.  It is like an implied 'this' but with three parts to it.

(Tangent: if you do 'eval name.scopedKey.toString' at the command prompt, you'll see the This objects that represent unspecified axes.  These get properly resolved before evaluation.  Change it to (name in Compile)... and you'll see that the configuration axis is now explicit.)

IIUC, the inspect command's output already does this with * symbols, and just leaving stuff out.  Having different delimiters for each part of the scope makes this pretty straightforward... I think.

Now that I think about it, it might also be possible to unify the syntax for */Global and {.}/ThisBuild in some way. 

I'm not sure it needs to be reproduced in the string syntax.  You might just stick with the main four (including configuration) and then the cross method:

  dep"org.example:demo:1.0:compile" cross(CrossVersion.Binary)

and probably make CrossVersion.Binary the default so it could be just:

  dep"org.example:demo:1.0:compile" cross()

Sure.
 

Conceptually, I think you can think of it that way, as long as there is a caveat that it isn't quite a normal project.

 

 I don't think the "it looks very much like this familiar thing but is actually different" thing is very conducive to learning -- that said, I don't really have a better idea either.

Ken

[eugene yokota]

On Tuesday, January 29, 2013 10:43:28 AM UTC-5, Mark Harrah wrote:

The command line format was designed without spaces because commands typically separate arguments by spaces.  It was also intended to be unambiguous to assist with tab completion, hence the :: to identify a task scope.  The Scala syntax is rather heavily overloaded and I'm not sure it would work too well in the command line environment.  Some day it might be possible to use Dynamic+macros to get subProject.Test.testOnly as the Scala syntax so that is another option and one that could merge the syntaxes. 

I always thought it would be an interesting experiment to implement sbt shell as Scala REPL + implicit execution context.

If tab completion is the blocking issue, can REPL expose tab completion extension point similar to Eugune B's [inference-driving macro][1]?

For a unified syntax, I wouldn't mind typing extra parens occasionally.

-eugene

  [1]: http://docs.scala-lang.org/overviews/macros/inference.html