Def.taskDyn

[Rüdiger Klaehn]

Hi all,

I just saw on a question on stackoverflow that there is a new way to declare dependencies in SBT 0.13.

http://stackoverflow.com/questions/17888316/sbt-0-13-task-macro-equivalent-of-flatmap

1. can somebody point me to a document explaining the rationale for this? What can you do with the new syntax that is not possible with the old one that did not require macros?

2. will the old mechanism that does not use macros eventually be replaced by the macro mechanism, or will it be retained indefinitely?

I must confess that I am slightly irritated about this. Is the expressive power of "vanilla" scala really so low that we need macros for absolutely everything?

best regards,

Rüdiger

[Mark Harrah]

Hi Rüdiger,

Thanks for the questions...

On Fri, 26 Jul 2013 13:54:10 -0700 (PDT)
Rüdiger Klaehn <rkl...@gmail.com> wrote:

> Hi all,
>
> I just saw on a question on stackoverflow that there is a new way to
> declare dependencies in SBT 0.13.
>
> http://stackoverflow.com/questions/17888316/sbt-0-13-task-macro-equivalent-of-flatmap
>
> 1. can somebody point me to a document explaining the rationale for this?
> What can you do with the new syntax that is not possible with the old one
> that did not require macros?

There isn't a document. The rationale was mainly that many people didn't like the old syntax. Here are some details:

a. The old way required a bunch of implicits. This makes builds slower to compile. Much of this makes the user-visible API unnecessarily larger due to intermediate forms or implementation artifacts.

b. The old way had a finite number of allowed inputs (originally 9, raised to 15). The new way is in theory unlimited. It is possibly limited by compiler performance on heterogeneous lists in practice, although the macros give explicit types so maybe not. (I haven't checked too high.)

c. The old way gave poor error messages when using a task as an input to a setting or similar cases. In general, error messages are or can be improved by using a macro.

d. See the new input task syntax, which is built on the same new macro system. The improvement here is more than that for normal tasks and settings. Input tasks are a combination of settings, tasks, parsers, and State. It's hard to stack things in Scala and provide both the ability to work at one level for flexbility or work at the full stack for convenience and provide somewhat readable error messages. I think the new macro-based syntax is much improved here, although it will still need work for some more advanced cases and issues people see in practice.

e. Note that the old flatMap was really an apply followed by a flatMap. The current one is as well, but it is relevant to note that we are mainly talking about applicative functors when talking about tasks/settings and only rarely about monads.

f. macros can capture the source location of the definition site. If you look at 'inspect' in 0.13, you will see that it tells you where the setting/task was defined.

g. There were seven basic methods before (:=, +=, ++=, ~=, <+=, <++=, <<=). Of those, the three beginning with '<' are no longer needed (or are close to it). The new syntax allows tasks with and without dependencies to be defined in the same way (with just the first three). I haven't decided about ~=, but it isn't as necessary as before and you could reasonably get away with just those first three. The last three symbols were a big complaint.

h. A specific example of an improvement is that a task can take an Initialize[Task[T]] as a task input now and not just a TaskKey[T]. (So, no need to declare intermediate, named tasks.) This wasn't possible with standard approaches (I tried). This may not seem like a big deal (and maybe most users don't deal with that type), but it makes it possible to provide more flexible APIs.

i. No need for an intermediate name for the value computed by a setting/task in simple cases.

j. Overall, the documentation was simplified. It is easier to provide snippets and to introduce concepts. (For example, it is no longer necessary to differentiate tasks with and without dependencies as described above.) I have only incrementally updated the documentation, but I think it has made things easier to explain for Josh in sbt in Action as well.

There are some disadvantages to the new approach and to macros in general, but you only asked about the advantages ;)

> 2. will the old mechanism that does not use macros eventually be replaced
> by the macro mechanism, or will it be retained indefinitely?

There isn't currently plan to remove it, although if macros work well, it would make sense to remove the old way. If removed, I doubt it would be a normal removal cycle. I'd guess it would be deprecated, then behind an import, then perhaps still available as a plugin. It depends on demand. There are a lot of builds using the old way. Also, although the macros have been well-received so far, 0.13.0 is still not final and widely used.

> I must confess that I am slightly irritated about this. Is the expressive
> power of "vanilla" scala really so low that we need macros for absolutely
> everything?

The glib answer is yes. I wouldn't say that it is "so low" though, just that macros can make a substantial difference for users. So far, the feedback I have seen on the new macros has been overwhelmingly positive. Certainly there will be things to improve, but I think the benefits are there.

For an additional discussion beyond the points listed above, consider the for comprehension. It is special Scala syntax for monadic computation that you couldn't do yourself using the tools Scala provides. (Even as it is, it is limited.)

What is the equivalent for applicative functors? There isn't one in Scala. People have tried various syntaxes using Scala pre-macros and they have various limitations or are unpopular for various reasons. There are various projects now using macros to provide a better syntax and sbt is one of them.

Next, note that a macro doesn't actually change Scala's syntax. Scala code involving macros is parsed normally and it has to typecheck before and after a macro is run. In particular, the types are the same before and after the macro runs (or "expands"). You could possibly come up with a system that has a similar syntax to 0.13 without using macros, but it almost certainly would be mutable, with less compile-time checking, and less useful and less powerful.

Traditionally, features like default and named arguments, inference, implicits, overloading, and infix methods are used to define a DSL. It quickly becomes hard to provide good error messages in all cases. The error message is often removed from what the user is trying to do. This is a standard internal DSL drawback of course.

Also, it is hard to evolve implicits and overloading without breaking code and it is rather hard to know in advance how code will break, especially due to inference. I don't think anyone writing a DSL sets out to push the boundaries of these things. You start with some requirements and then you come up against a choice is between syntax and semantics and I'd sacrifice syntax almost every time. (I find the use of ThreadLocals to improve syntax such a bad tradeoff, but that's another story.) Macros somewhat mitigate the need in some cases with the usual tradeoff being in implementation complexity.

Lastly, it is hard to document everything involved in making the DSL. A lot isn't fundamentally necessary to know, but it still comes out in error messages or API docs. Macros ideally hide that and provide only what the user needs (ideally of course). 0.13 doesn't fully benefit here because it still supports the old style and other reasons.

At least for now, I don't think you'll find macros being that pervasive just because non-trivial ones are so hard and time consuming to write. That might change with quasiquotes, but I haven't tried them for the macros in sbt yet. Speaking only for myself, I would only resort to macros when there isn't a standard way to do something. As just one example, this new 0.13 feature uses only standard constructs:

http://www.scala-sbt.org/0.13.0/docs/Detailed-Topics/Tasks.html#getting-values-from-multiple-scopes

-Mark

> best regards,
>
> Rüdiger
>
> --
> You received this message because you are subscribed to the Google Groups "simple-build-tool" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to simple-build-t...@googlegroups.com.
> To post to this group, send email to simple-b...@googlegroups.com.
> Visit this group at http://groups.google.com/group/simple-build-tool.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Eugene Burmako]

>> There are some disadvantages to the new approach and to macros in general, but you only asked about the advantages ;)

I would be interested in learning about these.

[Daniel Peebles]

I may be biased, but I wouldn't mind an answer to the SO question either :)

I'll admit that in principle I'm not a huge fan of macros everywhere. They do have their uses though and I think sbt's use of them is "tasteful" and enough of an improvement over the old to justify switching to them. The one thing I'd be tempted to change is to make the syntax more like the spores proposal, where you must declare a bunch of dependency vals up front and are then allowed to use them later in the block. It's uglier but enforces a less "insidious" dependency style. 

--

[Mark Harrah]

Hey Eugene,

On Sat, 27 Jul 2013 01:08:51 -0700 (PDT)
Eugene Burmako <xen...@gmail.com> wrote:

> >> There are some disadvantages to the new approach and to macros in
> general, but you only asked about the advantages ;)
> I would be interested in learning about these.

Thanks for asking. I think you know about most of them and have worked to address them with things like quasiquotes.

1. Issues with reset*Attrs means certain constructs cannot used in the expression passed to a macro. The macro author has to detect these constructs in order to provide a good error message for the user.

2. When constructing trees manually, knowing where to set positions. When a position is forgotten, error messages come out with no position information, which is hard to track down.

3. Type mismatches and other errors in generated trees can be difficult to debug since the position information in the error doesn't help much.

4. General difficulty of working with the compiler API. (I think this has been discussed plenty elsewhere, but I can elaborate if necessary.) This makes macro implementations hard to maintain and understand and increases the barrier to contribution.

5. Having to drop to full Global to get at certain APIs. I think constructing synthetic type constructors is one example.

6. I'm unsure of the right term for this situation, but if I want to wrap a macro call, I have to write another macro. This significantly raises the barrier for extension. As an example, consider a macro `task` that constructs a value of type Task[T]:

def task[T](body: T): Task[T] = macro taskImpl

If I want to wrap this to provide some additional processing afterwards, I can't just write:

def namedTask[T](name: String, body: T): Task[T] = task(body).named(name)

because `task` is just going to get passed a reference to the variable body and not the full tree.

-Mark

> On Friday, July 26, 2013 10:54:10 PM UTC+2, Rüdiger Klaehn wrote:
> >
> > Hi all,
> >
> > I just saw on a question on stackoverflow that there is a new way to
> > declare dependencies in SBT 0.13.
> >
> >
> > http://stackoverflow.com/questions/17888316/sbt-0-13-task-macro-equivalent-of-flatmap
> >
> > 1. can somebody point me to a document explaining the rationale for this?
> > What can you do with the new syntax that is not possible with the old one
> > that did not require macros?
> >
> > 2. will the old mechanism that does not use macros eventually be replaced
> > by the macro mechanism, or will it be retained indefinitely?
> >
> > I must confess that I am slightly irritated about this. Is the expressive
> > power of "vanilla" scala really so low that we need macros for absolutely
> > everything?
> >
> > best regards,
> >
> > Rüdiger
> >
>

[Mark Harrah]

On Sat, 27 Jul 2013 09:13:41 -0400
Daniel Peebles <pumpk...@gmail.com> wrote:

> I may be biased, but I wouldn't mind an answer to the SO question either :)
>
> I'll admit that in principle I'm not a huge fan of macros everywhere. They do have their uses though and I think sbt's use of them is "tasteful" and enough of an improvement over the old to justify switching to them. The one thing I'd be tempted to change is to make the syntax more like the spores proposal, where you must declare a bunch of dependency vals up front and are then allowed to use them later in the block. It's uglier but enforces a less "insidious" dependency style.

Can you explain what you mean by "insidious" dependency style? Do you mean that the dependencies are buried in the body instead of right at the top?

-Mark

[Daniel Peebles]

Yep, that's all I meant. Perhaps spore style isn't ideal but I'm reluctant to have some AST nodes buried deep in my code be more equal than others.

[Mark Harrah]

On Tue, 30 Jul 2013 07:40:39 -0400
Daniel Peebles <pumpk...@gmail.com> wrote:

> Yep, that's all I meant. Perhaps spore style isn't ideal but I'm reluctant to have some AST nodes buried deep in my code be more equal than others.

I agree this is a potential drawback. It was easier with the previous syntax to do a quick scan of task definition and get the dependencies.

That said, some mitigating factors:

1. longer task implementations should be a separate method anyway:

myTask := myTaskImpl(a.value, b.value, c.value)

2. if you don't do that for whatever reason, using vals becomes natural anyway, especially with longer key names and with scopes:

myTask := {
val av = (aLongerKeyName in Compile in run).value
val bv = b.value
... av ... bv ...
}

3. The .value should help as a visual clue. This was one reason 'apply' (and thus the opportunity for a '()' shorthand) wasn't chosen as the name. This is mainly for shorter implementations (maybe 5 lines max?), though.

My current opinion is that enforcing 1 or 2 via technical means increases the learning curve, which the syntax was meant to reduce, without too much benefit. It could be opt-in as a start if someone wants to submit a pull request implementing it.

[Daniel Peebles]

Fair enough, that makes sense :)

Or how about .THIS_IS_A_DEPENDENCY :P

[Rüdiger Klaehn]

Hi Mark,

thanks a lot for explaining the rationale behind the change.

I think I will have to use the new syntax and (perhaps more importantly) try to explain it to somebody without a scala background before I can really form a well-founded opinion on it.

I did not feel limited in any way by the old syntax, so I was just a bit surprised that all the stuff I learned just a few months ago when I set up my first complex sbt build is already kind of deprecated.

cheers,

Rüdiger

[Mark Harrah]

Hi Rüdiger,

I wouldn't look at it as deprecated. The concepts are the same and the new syntax translates to the old syntax (essentially). For example,

aTask := {
f(bTask.value, cTask.value)
}

translates to:

aTask <<= (bTask, cTask) map { (b,c) =>
f(b,c)
}

That reminds me of an additional advantage of the macro approach, which is that you don't need to remember whether to use apply (settings) or map (tasks), which was a problem for some people.

-Mark

> cheers,