Late-Closure-Classes and specialization

[Miguel Garcia]

Background: "Late-Closure-Classes" refers to a code-generation technique internal to the new backend ( http://magarciaepfl.github.io/scala/ ) that results in anon-closure-classes exposing the same API as before. If they're behaviorally identical to "traditional" anon-closure-classes, what's the point? Actually the new scheme sports several advantages (listed below) but this post focuses on their interplay with specialization, to answer recent questions.


  (a) A late-closure-class is specialized when -no-specialization is false, and not specialized otherwise.
      Exactly like traditional anon-closure-classes.

      Compare for the anon-function in:

        class C {
          def m() {
            (1 to 10) foreach { i => println(i) }
          }
        }

     Bytecode for the anon-closure-class above:

• traditional   https://gist.github.com/magarciaEPFL/5622649
• late-closure  https://gist.github.com/magarciaEPFL/5622675

The example also shows where the "closure body" now lives: not in the anon-closure-class but in the original enclosing class. Why would one do that? How about this: laying the groundwork for lambdas-as-MethodHandles. In case a MethodHandle replaces an anon-closure-class, well, then there's no way anymore for a closure-body to remain in a non-existent class.

Although the API is the same, late-closures save a method call: a bridge-apply directly targets the "most specialized" method. In contrast, for traditional closures, an apply invocation initially landing at the "least specialized" method follows a daisy-chain before landing at the most specialized one. That's additional bytecode, too.

This brings us to the other advantages of late-closure-classes:

  (b) smaller constant pools, ie good for mobile apps, faster classloading, shorter startup time.

  (c) completely avoids "cannot inline" situations in the (new) optimizer

  (d) fewer heap objects are retained via "closure state minimization",
      https://github.com/magarciaEPFL/scala/commit/060eaa842a14bf3353ce3f0ed946b05c86e6e9a7


Regarding future work, yes, there are further improvements waiting to be realized. For the snippet below the new backend falls back to the traditional scheme:

    def abc[@specialized(Int, Char) T](xs: Array[T]) {
      xs foreach { x => println(x) }
    }

"Falling back to traditional anon-closure-classes" means, we aren't any worse off than before. But yes, that's an example where late-closure-classes could do better. Handling that case requires teaching the specialize phase about Late-Closure-Classes, definitely possible but for the moment nowhere near essential.


And not to be forgotten, late-closure-classes contribute to the 15% speedup of the new backend over the current one.


Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

[Grzegorz Kossakowski]

On 21 May 2013 22:38, Miguel Garcia <miguel...@tuhh.de> wrote:

  (a) A late-closure-class is specialized when -no-specialization is false, and not specialized otherwise.
      Exactly like traditional anon-closure-classes.

      Compare for the anon-function in:

        class C {
          def m() {
            (1 to 10) foreach { i => println(i) }
          }
        }

     Bytecode for the anon-closure-class above:

• traditional   https://gist.github.com/magarciaEPFL/5622649
• late-closure  https://gist.github.com/magarciaEPFL/5622675

Hi Miguel,

What I'm missing is explanation how class generated with 'late-closure' technique gets specialized. Specifically, how do you know that you need to generate methods like 'apply$mcVI$sp' ?

Since you (in the backend) are in full control of synthesizing that class I'm guessing you have specialization rules baked in into backend, right?

 

The example also shows where the "closure body" now lives: not in the anon-closure-class but in the original enclosing class. Why would one do that? How about this: laying the groundwork for lambdas-as-MethodHandles. In case a MethodHandle replaces an anon-closure-class, well, then there's no way anymore for a closure-body to remain in a non-existent class.

Well, there's a problem of passing MethodHandle as a lambda: all higher order methods require FunctionN as argument and not MethodHandle. This makes it fairly hard to migrate to purely MethodHandle-based encoding for lambdas, right?

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

[Miguel Garcia]

Yes, my previous post covered only the "what" and not the "how" of Late-Closure-Classes.

All things code-gen of late-closures is part of the new bytecode emitter, specifically `genLateClosure()`

  https://github.com/magarciaEPFL/scala/blob/GenRefactored56/src/compiler/scala/tools/nsc/backend/jvm/BCodeLateClosuBuilder.scala#L106

Initially there's only a callsite to the method holding the closure-body, the signature of that method reflects non-erased type information (unless, well, the anon-function's args were generic to start with and not specialized). Based on that signature, the "right" (preferably specialized) subclass of FunctionX (respectively AbstractFunctionX) can be built, that's the responsibility of `takingIntoAccountSpecialization()`

  https://github.com/magarciaEPFL/scala/blob/GenRefactored56/src/compiler/scala/tools/nsc/backend/jvm/BCodeLateClosuBuilder.scala#L273

The only thing hardcoded is the arity beyond which no specialized functions can be expected (for efficiency only, and can be removed). At or below that arity, the list of valid function-specializations (for that particular compiler run, ie depending on the Scala library in use) is queried with the help of `specializeTypes.specializedFunctionX`

  that information is initially queried in `initBCodeTypes()`
    https://github.com/magarciaEPFL/scala/blob/GenRefactored56/src/compiler/scala/tools/nsc/backend/jvm/BCodeTypes.scala#L79

  based on what the specialize phase "itself" knows about specialized functions (that had to be cached before specialize forgets about it)
    https://github.com/magarciaEPFL/scala/commit/94491117a31c8fcab1bd789fb231a4a144973f13

Changing topic. Regarding using MethodHandles rather than FunctionX, yes, that's looking into the future and ignoring for the moment backwards compatibility. What I was trying to say is that late-closures offer a better starting point for a more efficient closure-conversion approach than the current scheme.


Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

[Sébastien Doeraene]

Hi all,

I just wanted to point out another advantage of late-closure-classes, irrespective of the backend being used (ASM or BCode). This translation scheme would be a blessing to alternative backends targeting platforms that do offer closures (e.g., JavaScript, of course). The size of the generated code can be improved a lot in certain circumstances.

Cheers,

Sébastien

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Grzegorz Kossakowski]

On 22 May 2013 00:03, Miguel Garcia <miguel...@tuhh.de> wrote:

Yes, my previous post covered only the "what" and not the "how" of Late-Closure-Classes.

All things code-gen of late-closures is part of the new bytecode emitter, specifically `genLateClosure()`

  https://github.com/magarciaEPFL/scala/blob/GenRefactored56/src/compiler/scala/tools/nsc/backend/jvm/BCodeLateClosuBuilder.scala#L106

Initially there's only a callsite to the method holding the closure-body, the signature of that method reflects non-erased type information (unless, well, the anon-function's args were generic to start with and not specialized). Based on that signature, the "right" (preferably specialized) subclass of FunctionX (respectively AbstractFunctionX) can be built, that's the responsibility of `takingIntoAccountSpecialization()`

  https://github.com/magarciaEPFL/scala/blob/GenRefactored56/src/compiler/scala/tools/nsc/backend/jvm/BCodeLateClosuBuilder.scala#L273

Thanks Miguel for explanation. This made me realize that if you compile the following code 

class Foo {

  def f: Int => Int = x => x +1

}

then after uncurry you get:

package <empty> {

  class Foo extends Object {

    def <init>(): Foo = {

      Foo.super.<init>();

      ()

    };

    def f(): Int => Int = {

      final def dlgt$114a61(x: Int): Int = x.+(1);

      lccDisguiserMethod(dlgt$114a61(0)).asInstanceOf[scala.runtime.AbstractFunction1[Int,Int]]()

    }

  }

}

Which means lambda body is moved to local method and you get strange placeholder for lambda creation.

This is a bit surprising to me because I thought that late-closures means that Function tree nodes survive as-is the uncurry phase and get expanded somewhere later. In particular, I don't really like this magical lccDisguiserMethod method call.

Migeul, did you consider keeping Function node all the way until your new backend?

Also, Sebastien, is this approach ok for your alternative, javascript backend? I thought you need Function nodes that you would translate directly to Javascript, right?

[Sébastien Doeraene]

Hi,

Yes, I've been aware of Miguel's translation scheme for a while, now. I agree the disguised thing is not superbly clean. But essentially it is a Function node. It is just encoded in a weird form. Keeping vanilla Function nodes would be cleaner, but one would have to review all the phases between Uncurry and the backend to support Function nodes, which is likely to be very error-prone.

It's OK for me. Just like Miguel recognizes such Tree patterns, I can do the same. The difference is that, for me, that wouldn't be the most complex pattern I have to recognize ^^ (I deal with much worse for while loops, pattern matches, tail calls, etc.!)

Cheers,

Sébastien

[Jason Zaugg]

The advantage of representing the lambda as a (wrapped) Apply node is
that later compiler phases (e.g. Explicitouter) need not be modified.

The risk is that something transforms away the special structure. When
I reviewed an earlier version of this work, I noted that erasure might
eliminate a redundant cast, which would destroy the lambda. I think
that's what the 'lccDisguiser' apply prevents.

In any case, it probably isn't a massive job to introduce a purpose
built tree node to encode this if we decide that the hack is too
fragile.

-jason

> Scalac hacker at Typesafe <http://www.typesafe.com/>
> twitter: @gkossakowski <http://twitter.com/gkossakowski>
> github: @gkossakowski <http://github.com/gkossakowski>

[Grzegorz Kossakowski]

On 23 May 2013 11:59, Jason Zaugg <jza...@gmail.com> wrote:

The advantage of representing the lambda as a (wrapped) Apply node is
that later compiler phases (e.g. Explicitouter) need not be modified.

The biggest advantage comes from the fact that we move body of the lambda to local def. The body of a lambda is potentially very complicated tree (it may contain lazy vals, class definitions, etc.) and later phases handle all complicated trees inside of local defs already.

The node that forwards to that local def has a very simple structure in comparison.

 

The risk is that something transforms away the special structure. When
I reviewed an earlier version of this work, I noted that erasure might
eliminate a redundant cast, which would destroy the lambda. I think
that's what the 'lccDisguiser' apply prevents.

In any case, it probably isn't a massive job to introduce a purpose
built tree node to encode this if we decide that the hack is too
fragile.

How about we keep after uncurry very simple Function node that just forwards to local def?

This way we would have the following tree as a result:

package <empty> {

  class Foo extends Object {

    def <init>(): Foo = {

      Foo.super.<init>();

      ()

    };

    def f(): Int => Int = {

      final def dlgt$114a61(x: Int): Int = x.+(1);

      (x: Int) => dlgt$114a61(x)

    }

  }

}

This way, it's a lot easier and nicer to describe what uncurry does:

1. Take body of a lambda and move it to a local def defined next to Function node
2. New body of a lambda becomes forwarder to that local def

Since the body of a Function will be very simple, the risk that it won't be transformed by later phases is rather low.

If we ever decide to keep body in a Function, the change will be also smaller.

WDYT?

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

[Miguel Garcia]

Besides the risk of introducing another kind of tree node (burden of special-casing all the way to the bytecode emitter) there's the hard evidence that the current scheme:

  (a) speeds up the compiler 15%

  (b) lccDisguiserMethod is a Symbol nobody else can possibly use, thus guaranteeing it won't be mistaken for anything else (in particular, this scheme overcomes a deficiency that Jason discovered in a previous version)

  (c) the nightly build has been passing at all optimization levels with the new backend for months now (just look at the history of http://magarciaepfl.github.io/scala/ , checkout any version mentioned there, and run the nightly)

Finally, I don't understand what the example "class Foo { def f: Int => Int = x => x +1 }" is supposed to suggest. As already mentioned, initially the lambda-body is turned into a local method. Is that a disadvantage? It can't be: the lambdalift phase won't have an easier time lifting anything but a local method, and any spin on the Function-node idea will have to be lifted, sooner or later.

I really don't see
  (1) what disadvantages the current scheme is supposed to have; nor
  (2) what advantages another scheme would have.

I mean, any proposed scheme should at least match what we have now, ie (a) to (c) above. *That* would provide evidence.



Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

[Grzegorz Kossakowski]

Miguel,

I think you misunderstood what I propose. I proposed in my last email small tweak to your scheme and not any major change.

Please read my email again and let me know what you think.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[martin odersky]

I am not sure about comprehensibility of lccDisguiserMethod either, even though it's true that it does make good use of the existing LambdaLift logic. But I believe Functions are not what we want either. Abstractly, what you are creating is not a Function node, but a Closure node, which would look something like that:

  case class Closure(env: List[Tree], methodRef: Select) extends Tree

Where the `env` is a list of closed over values and the `methodRef` node would be a Select with the receiver and method symbol. One potentially risky aspect here is that it would require an adaptation of the environment lifting logic to this new kind of node. So I understand the attractiveness of using a closure class like this one instead:

  case class Closure(apply: Apply) extends Tree

where the Apply would receive dummy arguments for missing arguments, as in Miguel's version. That second version is very close to what Miguel has; essentially we exchange the lccDisuiserMethod call followed by the cast  with the Closure node. I believe it would not be very risky. 

For me, either of the two Closure representations would be fine.

Cheers

 - Martin

On Thu, May 23, 2013 at 11:10 AM, Grzegorz Kossakowski <grzegorz.k...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

[Miguel Garcia]

The difference wrt to Late-Closure-Classes boils down to:

> How about we keep after uncurry very simple Function node that just forwards to local def?

> Since the body of a Function will be very simple, the risk that it won't be transformed by later phases is rather low.

The above assumes:

  (a) the proposed Function nodes won't demand more processing than a plain method call
      (that Function node plays the same role as lccDisguiserMethod in the new backend)

  (b) regarding risk of being transformed when it shouldn't,
      perhaps we should start by settling the issue whether an Apply with lccDisguiserMethod symbol can be transformed "when it shouldn't"
      (ie transformed in a way that prevents the correct Late-Closure-Class from being emitted).
      I say that risk is zero. Counter-example, anyone?
      Whether a Function node (handled in a special way) would also arrive in pristine state to the bytecode emitter, is open to debate.
      For starters, because we don't have *evidence* in the form of a working implementation about that.

In fact, "one can almost see" that at any point in the pipeline after uncurry, a bidirectional conversion is possible, as many times as desired between:

  case class Closure(env: List[Tree], methodRef: Select) extends Tree

and

  lccDisguiserMethod( delegateMethod ( <apply-args> <captured-values ) )

where the latter is precisely the AST shape used in the new backend. If each can be exchanged for the other without information loss, then other reasons should inform which one to pick. We're back to the topic that an alternative should at least match the capabilities of the new backend / optimizer.


Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

[Grzegorz Kossakowski]

On 23 May 2013 13:37, martin odersky <martin....@epfl.ch> wrote:

I am not sure about comprehensibility of lccDisguiserMethod either, even though it's true that it does make good use of the existing LambdaLift logic. But I believe Functions are not what we want either. Abstractly, what you are creating is not a Function node, but a Closure node, which would look something like that:

  case class Closure(env: List[Tree], methodRef: Select) extends Tree

Where the `env` is a list of closed over values and the `methodRef` node would be a Select with the receiver and method symbol. One potentially risky aspect here is that it would require an adaptation of the environment lifting logic to this new kind of node. So I understand the attractiveness of using a closure class like this one instead:

  case class Closure(apply: Apply) extends Tree

where the Apply would receive dummy arguments for missing arguments, as in Miguel's version. That second version is very close to what Miguel has; essentially we exchange the lccDisuiserMethod call followed by the cast  with the Closure node. I believe it would not be very risky. 

For me, either of the two Closure representations would be fine.

Hi Martin,

I agree that if we were able to introduce a new tree node then 

case class Closure(env: List[Tree], methodRef: Select) extends Tree

is the best and most straightforward node we could use in this case. However, I always heard that introducing a new tree node is essentially impossible at this stage of Scala development.

Therefore I'm trying to find the closest approximation we can get using existing nodes. I believe that Apply node with dummy arguments wrapped in lccDisguiserMethod method and followed by cast is modelled more nicely by Function node which has just Apply node with function arguments provided.

The advantages are:

• there are no magic symbols and tree shapes
• after lambdalift you get a tree shape representing a closure that is a valid Scala program that user could write

The main disadvantage is:

• You need to make sure that Apply node inside of Function node is properly processed by lambdalift logic so additional arguments to Apply are supplied if needed but this seems to be fairly easy to achieve

[Grzegorz Kossakowski]

On 23 May 2013 14:02, Miguel Garcia <miguel...@tuhh.de> wrote:

The difference wrt to Late-Closure-Classes boils down to:

> How about we keep after uncurry very simple Function node that just forwards to local def?

> Since the body of a Function will be very simple, the risk that it won't be transformed by later phases is rather low.

The above assumes:

  (a) the proposed Function nodes won't demand more processing than a plain method call
      (that Function node plays the same role as lccDisguiserMethod in the new backend)

  (b) regarding risk of being transformed when it shouldn't,
      perhaps we should start by settling the issue whether an Apply with lccDisguiserMethod symbol can be transformed "when it shouldn't"
      (ie transformed in a way that prevents the correct Late-Closure-Class from being emitted).
      I say that risk is zero. Counter-example, anyone?
      Whether a Function node (handled in a special way) would also arrive in pristine state to the bytecode emitter, is open to debate.
      For starters, because we don't have *evidence* in the form of a working implementation about that.

My main concern is long-term maintainability of this solution. It's not a question whether it's being transformed incorrectly right now but what will happen in the future when somebody modifies the code and misses special case for the magic tree.

As I alluded in response to Martin if we have two choices where one is to emit a tree that represent a valid Scala program and another which doesn't and involved magic symbols then I'd pick the former.

 

In fact, "one can almost see" that at any point in the pipeline after uncurry, a bidirectional conversion is possible, as many times as desired between:

  case class Closure(env: List[Tree], methodRef: Select) extends Tree

and

  lccDisguiserMethod( delegateMethod ( <apply-args> <captured-values ) )

where the latter is precisely the AST shape used in the new backend. If each can be exchanged for the other without information loss, then other reasons should inform which one to pick. We're back to the topic that an alternative should at least match the capabilities of the new backend / optimizer.

It's true that there's bidirectional mapping between all of those representations and that's why we can discuss which one of those we can pick because all of them contain the same information. :-)

The fact that there's bidirectional mapping doesn't help to address my concern with maintainability of the code.

[martin odersky]

On Sun, May 26, 2013 at 12:11 PM, Grzegorz Kossakowski <grzegorz.k...@gmail.com> wrote:

On 23 May 2013 13:37, martin odersky <martin....@epfl.ch> wrote:

I am not sure about comprehensibility of lccDisguiserMethod either, even though it's true that it does make good use of the existing LambdaLift logic. But I believe Functions are not what we want either. Abstractly, what you are creating is not a Function node, but a Closure node, which would look something like that:

  case class Closure(env: List[Tree], methodRef: Select) extends Tree

Where the `env` is a list of closed over values and the `methodRef` node would be a Select with the receiver and method symbol. One potentially risky aspect here is that it would require an adaptation of the environment lifting logic to this new kind of node. So I understand the attractiveness of using a closure class like this one instead:

  case class Closure(apply: Apply) extends Tree

where the Apply would receive dummy arguments for missing arguments, as in Miguel's version. That second version is very close to what Miguel has; essentially we exchange the lccDisuiserMethod call followed by the cast  with the Closure node. I believe it would not be very risky. 

For me, either of the two Closure representations would be fine.

Hi Martin,

I agree that if we were able to introduce a new tree node then 

case class Closure(env: List[Tree], methodRef: Select) extends Tree

is the best and most straightforward node we could use in this case. However, I always heard that introducing a new tree node is essentially impossible at this stage of Scala development.

I don't think that's true. In particular if the class is not used as an interchange format in pickling or reflection it should be not hard, and in fact it should be better to make things explicit via a new node rather than hiding them with special symbols and implicit assumptions.

 

Cheers

 - Martin

Therefore I'm trying to find the closest approximation we can get using existing nodes. I believe that Apply node with dummy arguments wrapped in lccDisguiserMethod method and followed by cast is modelled more nicely by Function node which has just Apply node with function arguments provided.

The advantages are:

• there are no magic symbols and tree shapes
• after lambdalift you get a tree shape representing a closure that is a valid Scala program that user could write

The main disadvantage is:

• You need to make sure that Apply node inside of Function node is properly processed by lambdalift logic so additional arguments to Apply are supplied if needed but this seems to be fairly easy to achieve

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Paul Phillips]

On Sun, May 26, 2013 at 3:55 AM, martin odersky <martin....@epfl.ch> wrote:

and in fact it should be better to make things explicit via a new node rather than hiding them with special symbols and implicit assumptions.

Oh god yes.

See also TypeRef. What else can we encode in a TypeRef? I have a map of the surface of the moon which needs a home. I'm thinking, TypeRef. We can pattern match it right out of there.

[Miguel Garcia]

> My main concern is long-term maintainability of this solution. It's not a question whether it's being transformed incorrectly right now
> but what will happen in the future when somebody modifies the code and misses special case for the magic tree.

That's precisely the whole point: the current kinds of tree-node are expressive enough to denote a closure that delegates via method-ref. It's not being transformed incorrectly because no special casing is needed to express that.

It's almost as if the compiler should handle that "magic" Apply as any other node that returns a Late-Closure-Class.

Which by the way, it does.


Miguel
http://lampwww.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

[Miguel Garcia]

Paul,

It would be great to hear your perspective on the concrete matter, rather than TypeRef.

The alternatives are:

  (a) case class Closure(env: List[Tree], methodRef: Select) extends Tree

  (b) Function node which has just Apply node with function arguments provided

  (c) Apply node that is processed like Apply nodes are

Can you comment on maintainability?


Miguel
http://magarciaepfl.github.io/scala/

[Paul Phillips]

On Sun, May 26, 2013 at 6:05 AM, Miguel Garcia <miguel...@tuhh.de> wrote:

  (b) Function node which has just Apply node with function arguments provided

Utilizing Function in some capacity looks miles ahead to me. Whatever effort it may take to clear a path beyond uncurry is still going to be way less trouble than saddling us with more tree decodings. Anything which goes by the name "xxxDisguiserMethod" is a huge code smell. Adding more semantics to Apply should be the last of all last resorts. "Apply" is the "TypeRef of Trees" already.

I trust the above concurrently generated notes can be linearized into something meaningful.

[Paul Phillips]

And I would be all for adding a new node if I thought it would pay off enough. But Function is both close enough and barely utilized. If there is will to add tree nodes then I would much prefer to keep that powder dry.

[Miguel Garcia]

We have yet to see a single example of the "special handling" being painted on the Late-Closure-Class approach.

That "special handling" is at the core of the argument about un-maintainability of the approach.

Greg, can you mention where you've seen an lccDisguiserMethod-Apply-node being special-cased? Your argument is "in case we forget to special-case it, we're in trouble"

Can you post a URL showing at least one example of the "special handling" you keep talking and talking about? I mean, there should be plenty of them in the new backend.


Miguel
http://magarciaepfl.github.io/scala/

[Miguel Garcia]

> Adding more semantics to Apply should be the last of all last resorts.

That's why an lccDisguiserMethod (just a name) has the same contract as any other method.

More semantics? You've been watching too much TypeRef lately. Go and get some fresh air: https://github.com/magarciaEPFL/scala/compare/master...GenRefactored58


Miguel

[Paul Phillips]

I should also point out that the tasteful application of a few extractors would go a long ways to alleviate the pain of node overloading. (Something which should happen retroactively even more urgently.) So using Apply isn't necessarily a deal killer, even if it did add to this brand of pain, which I understand you are saying it won't anyway.

[Grzegorz Kossakowski]

On 26 May 2013 17:12, Miguel Garcia <miguel...@tuhh.de> wrote:

> Adding more semantics to Apply should be the last of all last resorts.

That's why an lccDisguiserMethod (just a name) has the same contract as any other method.

More semantics? You've been watching too much TypeRef lately. Go and get some fresh air: https://github.com/magarciaEPFL/scala/compare/master...GenRefactored58

Miguel,

Let's have a look:

• The definition is of lccDisguiserMethod is:
  enterNewMethod(ScalaRunTimeModule, newTermName("lccDisguiserMethod"), anyparam, ObjectClass.tpe, flags)
  but if you inspect bytecode of ScalaRunTime that method is nowhere to be found
• If you check how it's being used you see:
  if (lccDisguised) {
    val Apply(_, fakeCallsiteWrapper :: Nil) = obj
    val fakeCallsite = fakeCallsiteExtractor(fakeCallsiteWrapper)
    genLateClosure(fakeCallsite, tpeTK(app))
  } else genTypeApply()
  
  This is a special case and you even have (appropriate) names like "fake" which suggest a code smell. It's hard to defend claim that lccDisguiserMethod is like any other method.
• lccDisguiserMethod behaves more like a compiler macro because it inspects its arguments (to extract a reference to a method) which no other method call does
  

Each of those three points are warning signs to me. Normally, if you run into a case like this where a special case is need you should look into refactoring so things are more explicit.

Also, don't get me wrong. I think what you got was fine for experimenting and it's impressive that it got you that far but I don't think this should be considered a final design.

Now, I'll sum up proposals we consider for encoding a closure:

• an Apply node with dummy arguments wrapped in lccDisguiserMethod followed by a cast
• a Function node with Apply node as a body; arguments to Apply node are Function parameters
• a new node Closure defined as follows:

• 
  
  case class Closure(env: List[Tree], methodRef: Select) extends Tree
  

I'm in favor of introducing Closure node if it's not too much work. Martin has a point that this node doesn't need to be exposed through reflection api and pickler doesn't need to care about it so it might not be too much work to introduce it.

[Miguel Garcia]

I agree with the following two facts you state, but not with the conclusion the new backend is an impressive "experiment" based on a "code smell".

  (a) The definition is of lccDisguiserMethod is:
      https://github.com/magarciaEPFL/scala/compare/master...GenRefactored58#L65R379
      that's what a synthetic method for compiler-internal use looks like!

  (b) where lccDisguiserMethod is being used:
      https://github.com/magarciaEPFL/scala/compare/master...GenRefactored58#L33R576
      that's in the bytecode emitter, in order to materialize the Late-Closure-Class.
      I hope you're not suggesting we should omit that expansion.

Finally, the last item you list as "fact", your opinion the expansion in (b) "looks like" a macro. All you're talking about is an AST lowering performed by the compiler.

Summing up, you haven't been able to pinpoint any special-handling of lccDisguiserMethod, as you originally claimed.

Ironically, this discussion highlights precisely the strengths of my approach: (a) and (b).


Regarding having a dedicated Closure node, of course it's always possible to have an additional node-type for something the existing ones can already express. That doesn't increase expressive power, nor improve compiler speed, nor help with maintenance, nor enable more powerful optimizations. It's just running in circles.


Miguel
http://magarciaepfl.github.io/scala/

[Grzegorz Kossakowski]

On 27 May 2013 13:28, Miguel Garcia <miguel...@tuhh.de> wrote:

I agree with the following two facts you state, but not with the conclusion the new backend is an impressive "experiment" based on a "code smell".

Miguel,

I'd like you to stop putting words in my mouth I didn't say. We've been discussing lccDisguiserMethod in this thread and I was making comments about this particular design decision. Nowhere in this thread I was making comments on the design of the whole backend.

[Miguel Garcia]

It's great the discussion covered the main points from our respective perspectives.

I don't have much to add.

For those wanting to join the discussion, please do so. Taking a good look at the pointers to source code discussed so far more than suffices to get you up to speed. After all, it's not rocket science :)


Miguel
http://magarciaepfl.github.io/scala/

[Grzegorz Kossakowski]

On 27 May 2013 15:39, Miguel Garcia <miguel...@tuhh.de> wrote:

It's great the discussion covered the main points from our respective perspectives.

I agree.

 

I don't have much to add. 

Same here.