[SIP-16] Self-cleaning macros

[Eugene Burmako]

Today we're glad to present our new SIP: Self-cleaning macros:
https://docs.google.com/document/d/1O879Iz-567FzVb8kw6N5OBpei9dnbW0ZaT7-XNSa6Cs/edit.

In comparison with the previous prototype, this one decouples macro
definitions and their implementations and also provides a way to
conveniently compose ASTs from statically-typed parts. All in all,
things became simpler and more type-safe.

You can clone the reference implementation from https://github.com/scalamacros/kepler,
build it according to the readme and experiment. Looking forward to
your comments!

[Daniel Sobral]

This hasn't shown up on http://docs.scala-lang.org/sips/ yet. I don't
know if this reminder is necessary or not, but I decided to remind
just in case. :-)

--
Daniel C. Sobral

I travel to the future all the time.

[Heather Miller]

The pull request was accepted, and it's merged into the source of http://docs.scala-lang.org.

Github has a tendency to take its sweet time generating pages-- we'll see when Github comes through with a live page. :)

[Simon Ochsenreither]

So as far as I understand, moving from T to Expr[T] is now intended to happen with the help of Context.reify and not with quasiquotations (or is there something like code"class Foo" still supported)? It feels like the macro proposal was a bit unbundled from the quasiquotations proposal. Is that right?

[martin odersky]

That's right. The fundamental discovery we made was that reify is
type-safe quasi-quote, so it's not clear one would still have a need
the latter (maybe for pattern matching). So we decided to drop quasi
quotes from the proposal.

Cheers

-- Martin

[Bernd Johannes]

Sorry if I got something wrong (so please ignore it if I am to weak minded to
get the macro thing right) but I dislike the following part:

"However, macro selection always depends on the static type of the receiver
object, not the dynamic type"

The example given:

class C { def m() = 1 }
class D extends C { override def m() = macro mI }

val d = new D; d.m()
val c = d; c.m()

with d.m() calling the macro implementation and c.m() calling C.m() is [in my
eyes] a slap in the face of the principle of least surprise (and to be frank -
I fail to understand why this should be the case).

This assignment could happen everywhere in the source code (and not side by
side like in this example) and I would be very surprised to see this
behaviour. How would you (as a human) reason about this?

I hope I misread or misunderstood something entirely as the example and
behaviour explained above seems... not desireable.

Can you restore my faith?

astonished greetings
Bernd

[Daniel Sobral]

On Fri, Mar 9, 2012 at 17:28, Bernd Johannes <bjoh...@bacon.de> wrote:
>
> Sorry if I got something wrong (so please ignore it if I am to weak minded to
> get the macro thing right) but I dislike the following part:

But you read it right.

>
> "However, macro selection always depends on the static type of the receiver
> object, not the dynamic type"
>
> The example given:
>
> class C { def m() = 1 }
> class D extends C { override def m() = macro mI }
>
> val d = new D; d.m()
> val c = d; c.m()

that should be "val c: C = d".

>
> with d.m() calling the macro implementation and c.m() calling C.m() is [in my
> eyes] a slap in the face of the principle of least surprise (and to be frank -
> I fail to understand why this should be the case).

Consider this simple function:

def f(c: C) = c.m()

Now, a macro is something that changes the bytecode. The bytecode for
calling method m on C is an invokevirtual or something similar. The
bytecode for calling m on D is something completely different -- in
fact, it might not even generate bytecode at all (see the assert macro
for such an example). How can the compile possibly contemplate this
when compiling f? Hell, it might not even know anything about D at
this point -- that might get compiled elsewhere!

So, while it is obvious in "val c: C = d; c.m()" that c actually
contains a D, that is an exception case, and there's no practical way
of specifying this exception, and much space for confusion.

By the way, this is the behavior of collection methods as well -- the
static type defines what happens, not the dynamic type. Some people do
get surprised by it, so some people will get surprised by this. It
can't be helped, though.

[Bernd]

On 9 Mrz., 22:50, Daniel Sobral <dcsob...@gmail.com> wrote:

> On Fri, Mar 9, 2012 at 17:28, Bernd Johannes <bjoha...@bacon.de> wrote:
>
> > Sorry if I got something wrong (so please ignore it if I am to weak minded to
> > get the macro thing right) but I dislike the following part:
>
> But you read it right.
>
>
>
> > "However, macro selection always depends on the static type of the receiver
> > object, not the dynamic type"
>
> > The example given:
>
> > class C { def m() = 1 }
> > class D extends C { override def m() = macro mI }
>
> > val d = new D; d.m()
> > val c = d; c.m()
>
> that should be "val c: C = d".
>

That's a game changer [faith restored]. I took the example literal and
couldn't follow why c and d should behave differently.
When I explicitly fix the type of c: C it is more obvious and
understandable (but might also cause some headache).

I am thinking about something like this:
def do_m(p: C) = p.m()
do_m(d)

It's not so obvious - may be there is a way to help to catch those
potential surprises.

Or think about:
d match { case x: C => x.m() }

I would like to see optional compiler warnings if a macro override
containing type instance is coerced into its super type.

Thanks for the explanations.

Greetings
Bernd

[Roland Kuhn]

Hi Bernd,

Am Freitag, 9. März 2012 23:49:50 UTC+1 schrieb Bernd:

On 9 Mrz., 22:50, Daniel Sobral <dcsob...@gmail.com> wrote:
> On Fri, Mar 9, 2012 at 17:28, Bernd Johannes <bjoha...@bacon.de> wrote:
>
> > Sorry if I got something wrong (so please ignore it if I am to weak minded to
> > get the macro thing right) but I dislike the following part:
>
> But you read it right.
>
>
>
> > "However, macro selection always depends on the static type of the receiver
> > object, not the dynamic type"
>
> > The example given:
>
> > class C { def m() = 1 }
> > class D extends C { override def m() = macro mI }
>
> > val d = new D; d.m()
> > val c = d; c.m()
>
> that should be "val c: C = d".
>

That's a game changer [faith restored]. I took the example literal and
couldn't follow why c and d should behave differently.
When I explicitly fix the type of c: C it is more obvious and
understandable (but might also cause some headache).

I am thinking about something like this:
def do_m(p: C) = p.m()

This is an interesting one: at this call site, p.m() is a virtual method call. So, all subclasses of C must implement this, otherwise NoSuchMethodError ensues. If D#m is a macro, how is the “non-specialized” method generated? I.e. in the SIP’s second example, what byte-code would I find in Queryable.class?

do_m(d)

It's not so obvious - may be there is a way to help to catch those
potential surprises.

Or think about:
  d match { case x: C => x.m() }

I would like to see optional compiler warnings if a macro override
containing type instance is coerced into its super type.

Given a nice answer to my question above, I’d say that this would be pure noise: by the Liskov substitution principle, the macro version in a subclass must have the same semantics as the presumably non-optimized “normal” code. Any surprises which arise must be blamed on the class&macro author.

Taking a step back: the static type of an expression represents the properties which the compiler can prove about it, hence the static type is the only piece of information available in determining which macros to apply, if any. Second-guessing the user (i.e. not respecting explicit type ascriptions) would be rude.

Regards,

Roland

[Roland Kuhn]

Am Samstag, 10. März 2012 17:00:55 UTC+1 schrieb Roland Kuhn:

Hi Bernd,

Am Freitag, 9. März 2012 23:49:50 UTC+1 schrieb Bernd:

On 9 Mrz., 22:50, Daniel Sobral <dcsob...@gmail.com> wrote:
> On Fri, Mar 9, 2012 at 17:28, Bernd Johannes <bjoha...@bacon.de> wrote:
>
> > Sorry if I got something wrong (so please ignore it if I am to weak minded to
> > get the macro thing right) but I dislike the following part:
>
> But you read it right.
>
>
>
> > "However, macro selection always depends on the static type of the receiver
> > object, not the dynamic type"
>
> > The example given:
>
> > class C { def m() = 1 }
> > class D extends C { override def m() = macro mI }
>
> > val d = new D; d.m()
> > val c = d; c.m()
>
> that should be "val c: C = d".
>

That's a game changer [faith restored]. I took the example literal and
couldn't follow why c and d should behave differently.
When I explicitly fix the type of c: C it is more obvious and
understandable (but might also cause some headache).

I am thinking about something like this:
def do_m(p: C) = p.m()

This is an interesting one: at this call site, p.m() is a virtual method call. So, all subclasses of C must implement this, otherwise NoSuchMethodError ensues. If D#m is a macro, how is the “non-specialized” method generated? I.e. in the SIP’s second example, what byte-code would I find in Queryable.class?

Ah, found it: below example 4 in this sentence «will lead to the invocation of method m in C, even though the dynamic receiver type is D», which I take to mean that D#m will in byte-code just call super.m() (i.e. not override). But this could be more clearly spelled out.
 

[argv minus one]

Two questions come to mind:

1. Why are macro implementations decoupled from macro defs?
2. Why is the package called makro instead of macros? Call me obsessive, but that looks kind of ugly.
   

[Simon Ochsenreither]

I guess the same reason why people spell class clazz, type tpe and so on: reserved keywords.

But yes ... I'm trying to think of a better name for it in general, too, but no success so far. :-)

[argv minus one]

On Monday, March 26, 2012 3:50:48 PM UTC-7, Simon Ochsenreither wrote:

I guess the same reason why people spell class clazz, type tpe and so on: reserved keywords.

Yes, and those give me the willies for the same reason. Just say "c" or something.
 

But yes ... I'm trying to think of a better name for it in general, too, but no success so far. :-)

What's wrong with "macros"?

[Stefan Zeiger]

On 2012-03-27 0:04, argv minus one wrote:

1. Why are macro implementations decoupled from macro defs?

Macro implementations have to be reachable through a stable path from a top-level object but the methods which are implemented by macros do not. You can't have a macro implementation in a non-static location because it couldn't see any values from its scope (because the instances comprising that scope do not exist at compile-time). We were at first considering to have a single method for the definition + implementation in any scope you wanted but this leads to all kinds of complications when it comes to dealing with symbols from the scope. It turned out that separating the definition from the implementation makes things much simpler.

-sz

[Jason Zaugg]

Also, from the SIP:

"Relatively less important for us was the convenience of writing
macros, precisely because we anticipate that macros should be used
only in relatively rare cases, and that they should be reserved to
expert programmers who are familiar with language theory and compiler
concepts"

-jason

[Christopher Vogt]

Additionally, decoupling macro def and implementation make the signatures clearer. The signature of a def implemented by a macro reads as relevant to the user of the macro. The signature of the macro implementation reads as relevant to the developer of the macro. E.g.

def assert( cond: Boolean ) = macro assertMacro
def assertMacro( c:Context )( cond: c.Expr[Boolean] ) = { cond.tree match {  ...  }  }

[nafg]

Okay, but what if I want a global macro --- why do I have to write two defs?

Also, what is the grammar of a macro definition? Is it necessary to make such irregular changes to the language syntax? And if it is, should they be in the middle of a def statement? As was pointed out regarding import language._, something that looks like a familiar construct should pull through as a familiar construct. So maybe it would be better to have a special macro statement.

So for instance, rather than

 def id ( args ... ) = macro Context=>args-like  // by that last thing, I mean it's a curried function / multi-parameter-list method, where the first argument is a Context and the second is args lifted into Exprs. If that's a misunderstanding then please correct or adjust.

which to me seems like hijacking the def statement, what about

 macro id ( args ... ) = Context=>args-like

Alternatively, what if macro were an annotation, not a keyword? Maybe,

@macro(assertMacro) def assert(cond: Boolean); // no implementation written out

[Eugene Burmako]

The main reason for splitting macro defs and macro impls is transparency. Previous approach with macro defs and macro impls merged had two deficiencies: 1) awkward interaction with lexical scope, 2) impossibility to specify exact mapping from a macro def to a macro impl.

Speaking of the macro def syntax, both of your proposals have been discussed internally. 

The first one hijacks the "equals" operator, which is also not ideal. 

The second one doesn't scale to type macros. With type macros you need to be able to write "type foo[type parameters](parameters) = ...", in syntax which shouldn't be allowed for regular TypeDefs. It's okay if we disambiguate between TypeDefs and TypeMacroDefs with a keyword. But it's impossible to do so on annotation basis, because parser doesn't know how to map names, such as "macro" in @macro, onto symbols, such as staticClass("scala.annotations.macro").

[Naftoli Gugenheim]

On Monday, April 2, 2012, Eugene Burmako wrote:

The main reason for splitting macro defs and macro impls is transparency. Previous approach with macro defs and macro impls merged had two deficiencies: 1) awkward interaction with lexical scope, 2) impossibility to specify exact mapping from a macro def to a macro impl.

Speaking of the macro def syntax, both of your proposals have been discussed internally. 

The first one hijacks the "equals" operator, which is also not ideal. 

So in hijacking equals vs. hijacking def you prefer the latter?

[Eugene Burmako]

If by "you" you mean Scala team, then yes. If by "you" you mean me, I'm fine with both syntactic schemes.