How to simplify passing macro context around?

[Eugene Burmako]

To put it in a nutshell:
1) Types of most compiler thingies depend on their compiler instances,
referred to as universes. For example, a reified tree has type rm.Tree
(where rm stands for scala.reflect.mirror, a value in scala.reflect
package). To illustrate this fact we can say that rm.Tree belongs to
mirror's universe.
2) This means that everyone who wants to work with compiler guts (here
I refer to macros) needs to be path-dependent.
3) That's why macro definitions get desugared into methods that use
path-dependent types for their arguments and return values:
https://github.com/scalamacros/kepler/blob/e9926a5207aadcfe3831b51b1cd6164757278013/src/compiler/scala/tools/nsc/typechecker/Macros.scala#L29.
4) All these types: trees, symbols and whatnot - depend on a _context,
an entry point into the compiler.
5) _context._ is automatically imported into a macro body, so, when
your macro is a single method, you don't get any problems. By writing,
say, "Apply(fun, args)", you will create an instance of _context.Apply
that belongs to the correct universe.

However, when you want to split your macro into several functions,
things get verbose. You have to carry around the _context to make sure
that the trees you construct will end up in the correct unverse.

Retronym's recent experiments with macros nicely illustrate the
problem:
https://github.com/retronym/macrocosm/blob/95a0505126b4035cf58a379e0dc4375dc2a5243c/src/main/scala/com/github/retronym/macrocosm/Macrocosm.scala.

You might wish to pass _context implicitly, but this hits a wall. On
the one hand, implicit parameters must come after explicit parameters.
But, on the other hand, you need that wannabe-implicit context to
define path-dependent types of your explicit parameters, so it has to
come first in the parameter list. For example, if you have a function
"def foo(context: Context, tree: context.Tree): context.Tree", you
cannot make context implicit, because tree depends on it.

This begs for a question. How do we fight bakery of doom here?

[Adriaan Moors]

On Sun, Feb 19, 2012 at 10:33 AM, Eugene Burmako <eugene....@epfl.ch> wrote:

You might wish to pass _context implicitly, but this hits a wall. On
the one hand, implicit parameters must come after explicit parameters.
But, on the other hand, you need that wannabe-implicit context to
define path-dependent types of your explicit parameters, so it has to
come first in the parameter list.

I'd also like to lift this restriction. (https://github.com/scala/scala/blob/master/src/compiler/scala/tools/nsc/typechecker/Namers.scala#L1454)

In principle, I think it would suffice to enforce the dependence graph for arguments is cycle-free.

This would mean you could write:

def foo(tree: context.Tree)(implicit context: Context): context.Tree

I tend to think of implicit arguments as the value-level counterpart of type parameters,

so you could think of the above as:

def foo[val context: Context](tree: context.Tree): context.Tree

cheers

adriaan

[Miles Sabin]

On Sun, Feb 19, 2012 at 10:20 AM, Adriaan Moors <adriaa...@epfl.ch> wrote:
> On Sun, Feb 19, 2012 at 10:33 AM, Eugene Burmako <eugene....@epfl.ch>
> wrote:
>>
>> You might wish to pass _context implicitly, but this hits a wall. On
>> the one hand, implicit parameters must come after explicit parameters.
>> But, on the other hand, you need that wannabe-implicit context to
>> define path-dependent types of your explicit parameters, so it has to
>> come first in the parameter list.
>
> I'd also like to lift this restriction.
> (https://github.com/scala/scala/blob/master/src/compiler/scala/tools/nsc/typechecker/Namers.scala#L1454)
>
> In principle, I think it would suffice to enforce the dependence graph for
> arguments is cycle-free.
> This would mean you could write:
>
> def foo(tree: context.Tree)(implicit context: Context): context.Tree

Yes! Please! Want! Now!

Cheers,

Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
g+: http://www.milessabin.com
http://twitter.com/milessabin
http://www.chuusai.com/

[martin odersky]

Frankly, I am not super concerned about macros being hard to write. I
am more concerned that they are too easy to write and people will go
overboard writing crazy stuff with them. So, maybe a high bar for
writing macros is even desirable. I certainly would not want to bend
over backwards to give more convenience for macro writers.

-- Martin

--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

[Jason Zaugg]

On Sun, Feb 19, 2012 at 11:24 AM, martin odersky <martin....@epfl.ch> wrote:
> Frankly, I am not super concerned about macros being hard to write. I
> am more concerned that they are too easy to write and people will go
> overboard writing crazy stuff with them. So, maybe a high bar for
> writing macros is even desirable. I certainly would not want to bend
> over backwards to give more convenience for macro writers.
>
>  -- Martin

The difficulty has nothing to do with macros per se -- it's to do with
the cake pattern. The same problems will affect people using the
reflection API, or those who, perhaps after reading PiS, have adopted
this pattern in their own codebase. Surely we want to make it easier
for all of these people to keep their code well factored.

Dependent method types go a long way, but the overly-conservative
restriction on the direction of the dependencies still stands in the
way.

-jason

[Jason Zaugg]

On Sun, Feb 19, 2012 at 10:33 AM, Eugene Burmako <eugene....@epfl.ch> wrote:

> Retronym's recent experiments with macros nicely illustrate the
> problem:
> https://github.com/retronym/macrocosm/blob/95a0505126b4035cf58a379e0dc4375dc2a5243c/src/main/scala/com/github/retronym/macrocosm/Macrocosm.scala.

> This begs for a question. How do we fight bakery of doom here?

I've just checked in a slightly different approach:

https://github.com/retronym/macrocosm/commit/f0197ab2

The important bits:

implicit def Util(context: Context) = new Util[context.type](context)

class Util[C <: Context with Singleton](val context: C) {
import context._

def id(a: Tree): Tree = a
// more method here
}

-jason

[martin odersky]

What else can we do? (without throwing more syntax at the problem).

-- Martin

[martin odersky]

Yes, that looks more like it. We can always refactor by creating
intermediate objects.

Cheers

-- Martin

[Jason Zaugg]

On Sun, Feb 19, 2012 at 11:51 AM, martin odersky <martin....@epfl.ch> wrote:
>> The difficulty has nothing to do with macros per se -- it's to do with
>> the cake pattern. The same problems will affect people using the
>> reflection API, or those who, perhaps after reading PiS, have adopted
>> this pattern in their own codebase. Surely we want to make it easier
>> for all of these people to keep their code well factored.
>>
>> Dependent method types go a long way, but the overly-conservative
>> restriction on the direction of the dependencies still stands in the
>> way.
>>
> What else can we do? (without throwing more syntax at the problem).

Just as Adriaan suggested: loosen the check in Namers [1] to allow for:

def foo(a: b.x)(implicit b: B)

-jason

[1] https://github.com/scala/scala/blob/master/src/compiler/scala/tools/nsc/typechecker/Namers.scala#L1454)

[martin odersky]

On Sun, Feb 19, 2012 at 11:56 AM, Jason Zaugg <jza...@gmail.com> wrote:
> On Sun, Feb 19, 2012 at 11:51 AM, martin odersky <martin....@epfl.ch> wrote:
>>> The difficulty has nothing to do with macros per se -- it's to do with
>>> the cake pattern. The same problems will affect people using the
>>> reflection API, or those who, perhaps after reading PiS, have adopted
>>> this pattern in their own codebase. Surely we want to make it easier
>>> for all of these people to keep their code well factored.
>>>
>>> Dependent method types go a long way, but the overly-conservative
>>> restriction on the direction of the dependencies still stands in the
>>> way.
>>>
>> What else can we do? (without throwing more syntax at the problem).
>
> Just as Adriaan suggested: loosen the check in Namers [1] to allow for:
>
>  def foo(a: b.x)(implicit b: B)
>

And what would the resulting types look like?

MethodType(a: b.x, MethodType(b: B, ...)

I think we'd introduce some serious scoping problems here!

-- Martin

[Jason Zaugg]

On Sun, Feb 19, 2012 at 12:10 PM, martin odersky <martin....@epfl.ch> wrote:
>>> What else can we do? (without throwing more syntax at the problem).
>>
>> Just as Adriaan suggested: loosen the check in Namers [1] to allow for:
>>
>>  def foo(a: b.x)(implicit b: B)
>>
> And what would the resulting types look like?
>
> MethodType(a: b.x, MethodType(b: B, ...)
>
> I think we'd introduce some serious scoping problems here!

I'm going to leave Adriaan to field that one while I swim back to shore :)

-jason

[Adriaan Moors]

yes, the forward references do complicate matters

I think one solution could be to introduce the value-level equivalent of context.undetparams (when you encounter a dependency on an undetermined argument while typing an application, you treat it like an undetermined type parameter and replace it by the type selection on the actual argument, once the latter has been supplied -- eta expansion of partial applications would be tricky, though...)

I have a similar approach in mind for uniformising constraint propagation in implicit coercions and implicit arg application [https://issues.scala-lang.org/browse/SI-4699]

adriaan

ps: for the record, I do not propose we add this to 2.10 -- just speculating on how we could implement it

[Eugene Burmako]

Submitted an improvement to the issue tracker: https://issues.scala-lang.org/browse/SI-5502.

On Feb 19, 12:33 pm, Adriaan Moors <adriaan.mo...@epfl.ch> wrote:

> On Sun, Feb 19, 2012 at 12:10 PM, martin odersky <martin.oder...@epfl.ch>wrote:
>
>
>
>
>
> > On Sun, Feb 19, 2012 at 11:56 AM, Jason Zaugg <jza...@gmail.com> wrote:

> > > On Sun, Feb 19, 2012 at 11:51 AM, martin odersky <martin.oder...@epfl.ch>

[Paul Phillips]

On Sun, Feb 19, 2012 at 2:50 AM, Jason Zaugg <jza...@gmail.com> wrote:

The important bits:

 implicit def Util(context: Context) = new Util[context.type](context)

 class Util[C <: Context with Singleton](val context: C) {
   import context._

   def id(a: Tree): Tree = a
   // more method here
 }

This looks like what I did here a couple months ago:

  https://github.com/scala/scala/commit/d05881e42e

As it says in the commit message, "Amazingly, it seems to work."

The important bits:

  /** Latest attempt to work around the challenge of foo.global.Type

   *  not being seen as the same type as bar.global.Type even though

   *  the globals are the same.  Dependent method types to the rescue.

   */

  def mkManifestToType[T <: Global](global: T) = {

    import global._

    import definitions._

    

    /** We can't use definitions.manifestToType directly because we're passing

     *  it to map and the compiler refuses to perform eta expansion on a method

     *  with a dependent return type.  (Can this be relaxed?) To get around this

     *  I have this forwarder which widens the type and then cast the result back

     *  to the dependent type.

     */

    def manifestToType(m: OptManifest[_]): Global#Type =

      definitions.manifestToType(m)

    

    class AppliedTypeFromManifests(sym: Symbol) {

      def apply[M](implicit m1: Manifest[M]): Type =

        appliedType(sym.typeConstructor, List(m1) map (x => manifestToType(x).asInstanceOf[Type]))

      def apply[M1, M2](implicit m1: Manifest[M1], m2: Manifest[M2]): Type =

        appliedType(sym.typeConstructor, List(m1, m2) map (x => manifestToType(x).asInstanceOf[Type]))

    }

    

    (sym: Symbol) => new AppliedTypeFromManifests(sym)

  }

[Paul Phillips]

Oh yeah, an example of what that's for:

scala> intp("scala.collection.Map")[Int, Int]

res0: $r.global.Type = scala.collection.Map[Int,Int]

Drink that one in for a while.

[Paul Phillips]

You can also do this:

scala> intp.types[Map[_,_]].apply[Int, Int]

res0: $r.global.Type = scala.collection.immutable.Map[Int,Int]

It'd be cool if you could do this, and you almost can because I fixed consecutive type application, but:

scala> intp.types[Map[_,_]][Int, Int]

<console>:32: error: method types: (implicit evidence$3: ClassManifest[Map[_, _]])$r.intp.global.Symbol does not take type parameters.

              intp.types[Map[_,_]][Int, Int]

                                  ^

Those darn implicit parameters again.