Best way to insert a method into a class with a bound variable

[Iñigo Mediavilla]

Hello,

I'm working for the first time intensively with Scala macros and I still lack the more basic concepts. For example I'd like to build the definition of a method that access a free variable and then insert that method into a classdef and bind the free variable to some of the fields of the class. What's is the best way of doing that?

For example:

I build with macros my method:

def sum(num: Int) = freeVar + num

And then I want to add the method to the following class:

class MyClass {

   val freeVar = 3

}

Best,

Inigo

[Eugene Burmako]

There's no easy solution to robust symbol management in the current macro system, so binding manipulations is something that might prove challenging. If you could elaborate on your use case, I will try to help.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Iñigo Mediavilla]

Basically, I'm trying to build a simplified async block similar to what  is explained in:

http://docs.scala-lang.org/sips/pending/async.html

That block is based on a macro that builds a state machine class. The class that I have to build has the following skeleton:

class MyStateMachine extends AnyRef {

      var result$async : scala.concurrent.Promise[Boolean] = scala.concurrent.Promise.apply[Boolean]();

      def resume$async : Unit = try {

      };

      def apply(tr: scala.util.Try[Boolean]) {

        ...

      }

      def apply(): Unit = resume$async

    }

What I'm trying to build apart is the method "apply(tr:scala.util.Try)". This method can call the promise "result$async" to give the result of the computation of the async block. 

I'd like to be able to build that method outside of the construction of the state machine class to then add it to the class and make sure that the "result$async" variable refers to the field of the class.

[Iñigo Mediavilla]

By the way, I'm using Scala 2.11.0-M7 so there's no problem in having a solution that uses quasiquotes. My only fear with quasiquotes is that If I've understood well I have to take care of hygiene of the macros when using them.

[Eugene Burmako]

Hey sorry for taking time to answer - I'm on a vacation at the moment. Today in the evening I'll have some spare time, and I'll try to elaborate :)

--

[Eugene Burmako]

Allright, here goes. Not going to be pretty, but making this better is very high on our priority list. 

=== 1. Manipulating bindings ===

First, back then in my stackoverflow answer I've somewhat outlined how in scalac we use symbols to manipulate bindings: http://stackoverflow.com/questions/11208790/how-can-i-reuse-definition-ast-subtrees-in-a-macro. 

You might find it useful to read through the entire discussion, but in short if we want to force binding of a given identifier (which can be Ident, Select or SelectFromTypeTree, i.e. subtypes of RefTree) to a given definition (which can be PackageDef, ClassDef, ModuleDef, ValDef, DefDef or TypeDef, i.e. subtypes of MemberDef), then we make sure that symbols of both trees are the same. 

This is currently the only way of working with bindings (we plan to introduce something new and shiny in the future, but it's too early to tell at the moment).

=== 2. Problems with using symbols ===

The thing about symbols as they are implemented in our compiler is that they are very capricious. 

First of all, unlike with trees or types, you can't create symbols on a whim. If your MemberDef is already attributed ("attributed" = "assigned with symbols and types") by the typechecker, then you can bind to it using its symbol. That's all good. However, if you've just created a MemberDef from scratch (either using reify or quasquotes), then it doesn't have a symbol yet. A natural reaction to this is to just go and create a symbol for that MemberDef, but it's quite complicated (we'll probably cover this a bit, later in the post).

Secondly, oftentimes you can't take a symbol created somewhere and transplant it somewhere else. E.g. if you already have `def sum(num: Int) = freeVar + num` typechecked (and typechecking among other things creates symbols for MemberDefs) then you can't just put this tree inside MyClass. 

There are no good reasons for this to be complicated - it's just a very unfortunate artifact of scalac's implementation strategy. We have plans to make this dead simple (as it should rightfully be), but it's going to take time to get there, and at the moment we have to deal with what we have.

=== 3. Two ways of dealing with symbol problems ===

In situations like async's, sbt's and yours, you have two options: a simple but not very robust one, and a really complicated but almost bulletproof one. Let's see what we have.

1) The first way is to abandon symbols altogether and rely on a gensym-like approach to binding. To make sure that you bind to `result$async`, you give it a sufficiently unique name (e.g. result$async looks awkward enough to hope that noone's going to use it) and then hope that there won't be inadvertent name collisions. To make sure that there aren't any problems with transplanting symbols, you unattribute all trees that you plan to transplant using `c.resetLocalAttrs`, destroying all symbols and types that have been assigned to them.

This is more or less simple, but is brittle in several ways (however the alternative requires noticeable effort, so people typically go for it anyway - even SBT until very recently used resetAttrs). 

The first problem is that, when names are handled on such a low level, there's never such thing as a sufficiently unique name. Even with obscure names, clashes do happen, and most likely not because of someone's ill intent, but because of unfortunate interaction between your macros. E.g. if you have an async block inside an async block, you might inadvertently bind to the wrong result$async. Such situations are quite rare, but they do happen.

The second problem is much more noticeable. There's a nasty thing about scala's typechecker. In some cases it's not idempotent, which means that if you take a tree of a certain shape, typecheck it, and then reset attrs - you might end up with a tree that's not going to typecheck anymore: https://issues.scala-lang.org/browse/SI-5464. Affected tree shapes are sufficiently rare in order to not become a tragedy, but they are also sufficiently frequent to become a nuisance. So far there's nothing to be done here - it's something that you have to accept if you go for this.

2) The second way is to nurture those symbols manually, which means that you're going to cater for those symbols' needs yourself, without relying on the typechecker. This implies two things: manual symbol creation and owner chain repairs.

Creating a symbol manually requires knowing/creating its owner and flags. The owner comes from the enclosing lexical context (e.g. a class method's owner is the class's symbol), but I don't have an exact recipe for this at the moment - you'd have to consult scalac's sources to see what owner is appropriate in this or that situation. Flags are also context-dependent, and scalac's sources would also be the best documentation here. This covers the first problem with symbols.

The second problem with symbols, the transplantation thing, happens because when you move attributed trees around, you might put symbols defined there in a lexical context that's inconsistent with those symbols' owners. This leads to funny effects that are documented in https://github.com/sbt/sbt/commit/3b213e59caafdc3b28bfbaf124e80a4584caf3aa. I haven't dealt with this myself, but I heard that Jason in async successfully uses https://github.com/scala/scala/blob/97b9b2c06a5f562b749eb34834096118f21ca843/src/reflect/scala/reflect/internal/Trees.scala#L1344 to overcome this problem. Jason, if you're reading this, would be great if you could provide additional details.

=== 4. Summary ===

All in all, we've seen that there are two ways of addressing the challenge that you're facing. Using unique names and resetAttrs is more or less simple to execute, but isn't robust enough to support 100% of Scala features. Manual creation of symbols and symbol repairs work just fine with everything that the users might throw at you, but that's nontrivial to implement.

[Simon Ochsenreither]

Wow, thanks for the detailed description!
This will be very helpful for some of the things I'm planning ... also excited to see the improvements you came up with in the future. :-)

[Iñigo Mediavilla]

Yep, thanks a lot Eugene, that was really useful!

I'll go for the first approach since I'm working in a really simple implementation that doesn't need to be bullet proof. However I'll be really happy to get a deeper understanding of the compiler so I may try to play a bit with manual creation of symbols just for the fun.

Any pointers for where to look to learn some of the details about how to create symbols?

[Eugene Burmako]

Jason's the expert in this topic, so I would start with [1] and [2] that he developed/inspired.

[1] https://github.com/sbt/sbt/commit/3b213e59caafdc3b28bfbaf124e80a4584caf3aa

[2] https://github.com/scala/async

--

[Eugene Burmako]

Also see http://stackoverflow.com/questions/20936509/scala-macros-what-is-the-difference-between-typed-aka-typechecked-an-untyped

To unsubscribe from this group and stop receiving emails from it, send an email to scala-internals+unsubscribe@googlegroups.com.

[Iñigo Mediavilla]

That's perfect, now I understand better what I need to do to be able to bind variables. 

Now I'm fighting with the manipulation of the symbols and I've seen the following code:

case class SymLookup(stateMachineClass: Symbol, applyTrParam: Symbol) {
    def stateMachineMember(name: TermName): Symbol =
      stateMachineClass.info.member(name)
    def memberRef(name: TermName): Tree =
      gen.mkAttributedRef(stateMachineMember(name))
  }

That is used in async to get a symbol and generate a reference for a term name. However this is used by doing quite a complex setup based on Global that seems like a global context for doing transformations. Is there a nicer way to do this or should I resort to the set up that the guys in async have used?

To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.

[Eugene Burmako]

Global is the cake of the compiler itself. As opposed to c.universe, which exposes a public subset of compiler APIs, with Global you can use all functions and helpers defined in the compiler. It might be that you can do everything you need using public APIs, but it will most likely be more convenient to use Global.

[Iñigo Mediavilla]

Hello,

I've ended up playing with global and building the trees and symbols by hand. However I'm running into the following problem:

scala.reflect.internal.Types$TypeError: value <none> is not a member of <type-who-calls-my-macro>

and I can't figure out how to locate where this is happening. I guess some of the symbols is not well assigned but I can't find where even debugging scalac. Anyone knows what is the source of this error and how I can spot what part of the code generated by my macro is producing it?

Regards,

Inigo

[Eugene Burmako]

You could try printing out your macro expansion with showRaw(..., printIds = true). That will tell you which trees have their assigned and which ones don't. printTypes = true will help with looking into types.

[Iñigo Mediavilla]

Hello,

I've tried printing symbols and types but I don't see anything weird.

The error appears when I generate a call to:

I generate a class with an apply with the following type (scala.util.Try[Boolean]) => Unit

then inside the class I make the following call:

myFuture.onComplete(this)(myExecContext)

What gets transformed into:

myFuture.onComplete((result : Try[Boolean]) => MyStateMachine.this.apply(result))(myExecContext)

What seems to make sense.

However when I compile after the expansion of the macro I get the TypeError. And when debugging the compilation I see that the problem is launched with the following term:

$anonfun.this.$outer.$outer().$outer().<none> of type Apply

What do the $outer field contains and why could my anonymous function try to access it?

[Jason Zaugg]

On Fri, Jan 24, 2014 at 1:37 PM, Iñigo Mediavilla <imed...@gmail.com> wrote:

Hello,

I've tried printing symbols and types but I don't see anything weird.

The error appears when I generate a call to:

I generate a class with an apply with the following type (scala.util.Try[Boolean]) => Unit

then inside the class I make the following call:

myFuture.onComplete(this)(myExecContext)

What gets transformed into:

myFuture.onComplete((result : Try[Boolean]) => MyStateMachine.this.apply(result))(myExecContext)

What seems to make sense.

However when I compile after the expansion of the macro I get the TypeError. And when debugging the compilation I see that the problem is launched with the following term:

$anonfun.this.$outer.$outer().$outer().<none> of type Apply

What do the $outer field contains and why could my anonymous function try to access it?

Inner classes (including anonymous functions) contain a "$outer" field that captures the enclosing instance.

I recommend that you compile with -Xprint:typer to show the entire tree after typechecking/macro expansion. You can also try -Ycheck:typer which might find corrupt symbol structures (symbols representing defintions must have the correct "owner" set; when you splice trees around this can become inconsistent).

As the author of scala-async, I can only share that this is super hard to get right. The best way you can get help is to try to boil down a small, self-contained example that exhibits the same problem that you can share with us.

-jason

[Iñigo Mediavilla]

Hello Jason,

This is the really small example that I've prepared for testing:

"deal with future " in {

      import Macros._

      import scala.concurrent.ExecutionContext.Implicits.global

      val result : Future[Boolean] = async {

        val f1 : Future[Boolean] = Future.apply(true)

        var a1 = Macros.await(f1)

        a1

      }

      result.isCompleted must_==(true)

      ok

    }      

And when I run the test, this is what generates after the typing phase:

[[syntax trees at end of                     typer]] // ImplSpec.scala

package es.imediava.cl.async {

  import org.specs2.mutable.Specification;

  import scala.concurrent.ExecutionContext.Implicits.global;

  import scala.util.{Failure, Try};

  import scala.concurrent.{Future, Await};

  class ImplSpec extends org.specs2.mutable.Specification {

    def <init>(): es.imediava.cl.async.ImplSpec = {

      ImplSpec.super.<init>();

      ()

    };

    ImplSpec.this.described("Impl").should(ImplSpec.this.inExample("deal with future that fails ").in[org.specs2.matcher.MatchSuccess[None.type]]({

      val myException: RuntimeException = new scala.`package`.RuntimeException("future that failed");

      import Macros._;

      import scala.concurrent.ExecutionContext.Implicits.global;

      val result: scala.concurrent.Future[Boolean] = ({

        class MyStateMachine extends AnyRef with scala.util.Try[Boolean] => Unit {

          def <init>(): MyStateMachine = {

            MyStateMachine.super.<init>();

            ()

          };

          private[this] val context: scala.concurrent.ExecutionContextExecutor = scala.concurrent.ExecutionContext.Implicits.global;

          <stable> <accessor> def context: scala.concurrent.ExecutionContextExecutor = MyStateMachine.this.context;

          private[this] var result$async: scala.concurrent.Promise[Boolean] = Promise.apply[Boolean]();

          <accessor> def result$async: scala.concurrent.Promise[Boolean] = MyStateMachine.this.result$async;

          <accessor> def result$async_=(x$1: scala.concurrent.Promise[Boolean]): Unit = MyStateMachine.this.result$async = x$1;

          private[this] var state: Int = 0;

          <accessor> def state: Int = MyStateMachine.this.state;

          <accessor> def state_=(x$1: Int): Unit = MyStateMachine.this.state = x$1;

          private[this] var f1: scala.concurrent.Future[Boolean] = null;

          <accessor> def f1: scala.concurrent.Future[Boolean] = MyStateMachine.this.f1;

          <accessor> def f1_=(x$1: scala.concurrent.Future[Boolean]): Unit = MyStateMachine.this.f1 = x$1;

          private[this] var a1: Boolean = false;

          <accessor> def a1: Boolean = MyStateMachine.this.a1;

          <accessor> def a1_=(x$1: Boolean): Unit = MyStateMachine.this.a1 = x$1;

          def apply(result: scala.util.Try[Boolean]): Unit = MyStateMachine.this.state match {

            case 0 => {

              if (result.isFailure)

                {

                  MyStateMachine.this.result$async.complete(result.asInstanceOf[scala.util.Try[Boolean]]);

                  ()

                }

              else

                {

                  MyStateMachine.this.a1_=(result.get.asInstanceOf[Boolean]);

                  MyStateMachine.this.state_=(1);

                  MyStateMachine.this.resume$async()

                };

              ()

            }

          };

          def resume$async(): Unit = MyStateMachine.this.state match {

            case 0 => {

              {

                MyStateMachine.this.f1.onComplete[Unit]({

                  ((result: scala.util.Try[Boolean]) => MyStateMachine.this.apply(result))

                })(MyStateMachine.this.context);

                ()

              };

              ()

            }

          };

          def apply2(): Unit = MyStateMachine.this.resume$async()

        };

        val stateMachine: MyStateMachine = new MyStateMachine();

        Future.apply[Unit](stateMachine.apply2())(ExecutionContext.global);

        stateMachine.result$async.future

      }: scala.concurrent.Future[Boolean]);

      ImplSpec.this.theValue[Boolean](result.isCompleted).must_==(true);

      ImplSpec.this.ok

    })(ImplSpec.this.contextAsResult[None.type, org.specs2.matcher.MatchSuccess](ImplSpec.this.contextAsResult$default$1[None.type, org.specs2.matcher.MatchSuccess])))

  }

}

And this is what I get with check:typer:

info] [Now checking: typer]

[warn] [check: typer] 

[warn] Out of scope symbol reference {

[warn]       tree  ValDef               val stateMachine: MyStateMachine = new MyStateMachine()                         

[warn]   position  OffsetPosition       /Users/zenexity/Desktop/programming/projects/scala/sbt-example 2/macros/src/t...

[warn]   with sym  TermSymbol           val stateMachine: MyStateMachine                                                  in  value result -> value <local ImplSpec> -> class ImplSpec -> package async

[warn]    encl(1)  TermSymbol           val result: scala.concurrent.Future[Boolean]                                      in  value <local ImplSpec> -> class ImplSpec -> package async

[warn]    encl(2)  ClassSymbol          ImplSpec: es.imediava.cl.async.ImplSpec                                           in  package async

[warn]     ref to  ClassSymbol          MyStateMachine ()                                                                 in  value result -> value <local ImplSpec> -> class ImplSpec -> package async

[warn] }

[warn] [check: typer] 

[warn] Out of scope symbol reference {

[warn]       tree  Ident                stateMachine                                                                    

[warn]   position  OffsetPosition       /Users/zenexity/Desktop/programming/projects/scala/sbt-example 2/macros/src/t...

[warn]   with sym  TermSymbol           val stateMachine: MyStateMachine                                                  in  value result -> value <local ImplSpec> -> class ImplSpec -> package async

[warn]    and tpe  UniqueSingleType     stateMachine.type                                                               

[warn]    encl(1)  TermSymbol           val result: scala.concurrent.Future[Boolean]                                      in  value <local ImplSpec> -> class ImplSpec -> package async

[warn]    encl(2)  ClassSymbol          ImplSpec: es.imediava.cl.async.ImplSpec                                           in  package async

[warn]     ref to  ClassSymbol          MyStateMachine ()                                                                 in  value result -> value <local ImplSpec> -> class ImplSpec -> package async

[warn] }

[warn] [check: typer] 

[warn] Out of scope symbol reference {

[warn]       tree  Ident                stateMachine                                                                    

[warn]   position  OffsetPosition       /Users/zenexity/Desktop/programming/projects/scala/sbt-example 2/macros/src/t...

[warn]   with sym  TermSymbol           val stateMachine: MyStateMachine                                                  in  value result -> value <local ImplSpec> -> class ImplSpec -> package async

[warn]    and tpe  UniqueSingleType     stateMachine.type                                                               

[warn]    encl(1)  TermSymbol           val result: scala.concurrent.Future[Boolean]                                      in  value <local ImplSpec> -> class ImplSpec -> package async

[warn]    encl(2)  ClassSymbol          ImplSpec: es.imediava.cl.async.ImplSpec                                           in  package async

[warn]     ref to  ClassSymbol          MyStateMachine ()                                                                 in  value result -> value <local ImplSpec> -> class ImplSpec -> package async

[warn] }

Cheers,

Inigo

[Jason Zaugg]

On Fri, Jan 24, 2014 at 2:58 PM, Iñigo Mediavilla <imed...@gmail.com> wrote:

Hello Jason,

This is the really small example that I've prepared for testing:

The "really small" part that counts is actually the macro implementation. If you can prepare something that I can clone, run, and witness the crash, I can help. I don't spot the problem from that output alone.

Sorry,

Jason

[Iñigo Mediavilla]

My bad, I misunderstood. The project I'm working on is a project that I'm developing for a course on Conception of Languages that is available at:

https://github.com/txominpelu/cl-async/

As you'll see I've tried to create a simplified version of scala/async using some pieces of your code so I guess that'll make it easy for you to understand what's going on in the code. This being my first project with macros and due to my lack of understanding of the compiler the project is full of code for debugging, and rests of tests that I've done to try to figure out what was going on so I apologize in advance for the crap that you'll find inside : ).

[Jason Zaugg]

On Fri, Jan 24, 2014 at 3:32 PM, Iñigo Mediavilla <imed...@gmail.com> wrote:

My bad, I misunderstood. The project I'm working on is a project that I'm developing for a course on Conception of Languages that is available at:

https://github.com/txominpelu/cl-async/

As you'll see I've tried to create a simplified version of scala/async using some pieces of your code so I guess that'll make it easy for you to understand what's going on in the code. This being my first project with macros and due to my lack of understanding of the compiler the project is full of code for debugging, and rests of tests that I've done to try to figure out what was going on so I apologize in advance for the crap that you'll find inside : ).

I believe the problem is with this tree:

        MyStateMachine#52814.this.f1#52865.onComplete#52747[Unit#2505]({

          ((result#52978: scala#23.util#2628.Try#9229[Boolean#2400]) => MyStateMachine#52814.this.apply#52871(result#52978))#52977

        })(MyStateMachine#52814.this.context#52857);

These symbol ids are printed here because I used the option `-uniqid`, which is handy to disambiguate same-named symbols.

The anonymous function's symbol, `value $anonfun#52977` is owned by `value result#52663`. That's the symbol that logically encloses the macro application (aka the "current owner" in the typer's Context.)

But, the correct owner for that place in the tree is the closest enclosing definition (val/def/function), which is this case should be `method resume$async#52872`.

So, when it comes to expanding the `MyStateMachine.this` in the body of that closure, the backend follows the owner-chain expecting to hit the enclosing class. The chain of `outer$` selects show its ever-hopeful, but ultimately fruitless search.

See https://groups.google.com/forum/#!topic/scala-internals/TtCTPlj_qcQ for some discussion about this general problem, and a hacky description of a "owner repair" utilty you could try in https://issues.scala-lang.org/browse/SI-5797.

-jason

[Iñigo Mediavilla]

Yes, thanks a lot it was exactly that!

I added the following code:

val result = asyncMacro.transformAt(final4) {

      case f@Function(_, _) if f.symbol.owner != newClassSkeleton =>

        (ctx: analyzer.Context) =>

          changeOwner(f, f.symbol.owner, ctx.owner)

    }

What effectively changed the owner of the anonymous function and made the project compile.

[Iñigo Mediavilla]

Hello,

I'm having problems again and I think it's again a problem with the owner of my symbol, event though I've tried all possible ways to assign it the right owner. Basically I'm creating a field for my class by creating the valdef and then adding it to the template of my class. Finally I change the owner of my valdef to the class symbol but even after typechecking the symbol table of my class doesn't contain any reference to the field what makes compilation fail in a phase that generates intermediate code.

Am I missing any step in the creation of a field inside a class?

Regards,

Inigo

[Jason Zaugg]

On Sun, Jan 26, 2014 at 9:52 AM, Iñigo Mediavilla <imed...@gmail.com> wrote:

Hello,

I'm having problems again and I think it's again a problem with the owner of my symbol, event though I've tried all possible ways to assign it the right owner. Basically I'm creating a field for my class by creating the valdef and then adding it to the template of my class. Finally I change the owner of my valdef to the class symbol but even after typechecking the symbol table of my class doesn't contain any reference to the field what makes compilation fail in a phase that generates intermediate code.

Am I missing any step in the creation of a field inside a class?

You might need to enter it into the declarations of the class's type.

   classSymbol.info.decls.enter(memberSymbol)

-jason

[Iñigo Mediavilla]

Spot-on, that worked again!

By the way thank you guys for being so reactive.

[Eugene Burmako]

Two more techniques for dealing with the owner chain corruption problem:

* https://issues.scala-lang.org/browse/SI-5797?focusedCommentId=63969&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-63969 (class OwnerRepair)

* https://gist.github.com/retronym/10640845#file-macro2-scala (the Splicer macro)