scala compiler plugin - how to generate and insert some new code into a ClassDef

[kgd]

i'm writing a relatively simple compiler plugin. the goal is to create
a trait that when extended by a class, automatically generates a val
on that class that returns an array of the default constructor
parameters used to instantiate the class.

for example:

trait AutoArgs {
val autoArgs: Array[Any]
}

class A(x: Int, y: Double) extends AutoArgs

then class A would automatically generate:

lazy val autoArgs = Array(x, y)


i've successfully created a plugin that uses constrParamAccessors to
determine the symbols for default parameters, and i use symtab
definitions' function getClass on the trait i made to see if the
current ClassDef is a subclass of AutoArgs.

the problem i'm running into is writing the new ValDef. it's extremely
straightforward to do so if i want to run my phase after "parser" -
the code looks like this:

val params = cd.symbol.constrParamAccessors
val finalParams: List[Ident] = params filter (x => !(x hasFlag
Flags.IMPLICIT)) map (x => Ident(x.name))
val newVal = ValDef(Modifiers(Flags.LAZY | Flags.OVERRIDE),
"autoArgs",
TypeTree(),
Apply(Ident("Array"), finalParams))


the problem is that running after "parser" seems too early, because
the symbol information about whether something is a trait or a class,
using getClass, etc, is missing, so none of those checks work. it
seems the earliest i can run the phase is after "namer" but then the
code i generate must deal with managing its own types. so instead of
the above code, i have to write a tree that generates something like:

lazy private[this] var AutoArgs: Array[Any] = scala.Array.apply[Any]
(D.this.x, D.this.y)(reflect.this.Manifest.Any);

which i'm having much more trouble doing.

is there a simple way to run the "namer" phase on just the new ValDef
i created, or otherwise tackle this problem in general? can someone
perhaps help with writing the ValDef that has all that type
information? thanks!

[Chris Twiner]

Unfortunately its not straightforward. Try searching for emails from Kevin Wright about plugins, you can run typers directly but there are other issues that follow.

[Kevin Wright]

Same old problem...

• You need to run after namer so that you have symbol information
• You need to run before typer, because the compiler will fail and report a missing method if you don't
• You can't run between the two phases, because they're inexorably entwined

If you're willing to never call your generated method, then you can happily run the thing after typer.  I'd recommend that you use TreeDSL to build the AST fragment, and wrap it in a `localTyper.typed` block via the TypingTransformer trait.

As per line 69 here: https://github.com/kevinwright/Autoproxy-Lite/blob/master/plugin/src/main/scala/autoproxy/plugin/GenerateSynthetics.scala

There's also another consideration here... If you're using eclipse, then the presentation compiler only runs up to the typer phase.  So if you want your method to be shown in code outlines and available for code-completion then it had *better* exist by that point!

Right now, I'm working on a patch to scalac that provides the necessary hooks for a plugin to inject synthesized methods during the namer & typer phases.  Watch this space for news of further progress.

--
Kevin Wright

gtalk / msn : kev.lee...@gmail.com

mail: kevin....@scalatechnology.com

vibe / skype: kev.lee.wright

quora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

[Paul Butcher]

On 24 May 2011, at 17:04, Kevin Wright wrote:
> If you're willing to never call your generated method, then you can happily run the thing after typer.

Ah - what does this mean, exactly? Never call it within what context?

--
paul.butcher->msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: pa...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher

[Kevin Wright]

On 24 May 2011 17:10, Paul Butcher <pa...@paulbutcher.com> wrote:

On 24 May 2011, at 17:04, Kevin Wright wrote:
> If you're willing to never call your generated method, then you can happily run the thing after typer.

Ah - what does this mean, exactly? Never call it within what context?

Never call it within the code base that's currently being compiled.

If memory serves, methods synthesised after typer don't appear in the ScalaSig either.  But you should be just fine & dandy if you're happy to call it from Java!

 

--
paul.butcher->msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: pa...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher

[Paul Butcher]

On 24 May 2011, at 17:13, Kevin Wright wrote:
> Never call it within the code base that's currently being compiled.

Hmm - does that mean the file that's currently being compiled? Or all code within the entire codebase?

If the former, I might just be able to get away with it...

[Kevin Wright]

If using something like SBT, you could try splitting it into a separate module.

Good luck, may the source be with you!

[kgd]

Thanks Kevin, is this an accurate summary of the discussion:

- it's extremely hard to generate new code in a compiler plugin
- using TreeDSL will definitely(?) or maybe(?) work if the code
calling the method is in a different module
- in the future a plugin will allow this (any idea on ETA?)

since i can override something on a trait, it doesn't matter to me
code completion isn't as important to me

- kgd

On May 24, 9:20 am, Kevin Wright <kev.lee.wri...@gmail.com> wrote:
> If using something like SBT, you could try splitting it into a separate
> module.
>
> Good luck, may the source be with you!
>

> On 24 May 2011 17:19, Paul Butcher <p...@paulbutcher.com> wrote:
>
>
>
>
>
>
>
>
>
> > On 24 May 2011, at 17:13, Kevin Wright wrote:
> > > Never call it within the code base that's currently being compiled.
>
> > Hmm - does that mean the file that's currently being compiled? Or all code
> > within the entire codebase?
>
> > If the former, I might just be able to get away with it...
>
> > --
> > paul.butcher->msgCount++
>
> > Snetterton, Castle Combe, Cadwell Park...
> > Who says I have a one track mind?
>
> >http://www.paulbutcher.com/
> > LinkedIn:http://www.linkedin.com/in/paulbutcher

> > MSN: p...@paulbutcher.com

> > AIM: paulrabutcher
> > Skype: paulrabutcher
>
> --
> Kevin Wright
>

> gtalk / msn : kev.lee.wri...@gmail.com
> <kev.lee.wri...@gmail.com>mail: kevin.wri...@scalatechnology.com

[Kevin Wright]

On 24 May 2011 18:43, kgd <kevi...@gmail.com> wrote:

Thanks Kevin, is this an accurate summary of the discussion:

- it's extremely hard to generate new code in a compiler plugin
- using TreeDSL will definitely(?) or maybe(?) work if the code
calling the method is in a different module
- in the future a plugin will allow this (any idea on ETA?)

since i can override something on a trait, it doesn't matter to me
code completion isn't as important to me

In your specific case, this is already possible, because the method is already inherited in abstract form and so the thing should type-check cleanly even before any synthesis takes place.

Take a look at the autoproxy-lite plugin, where I've relied on exactly this "trick"

https://github.com/kevinwright/Autoproxy-Lite/tree/master/plugin/src/main/scala/autoproxy/plugin

(incidentally, the fact that only already-inherited members can be proxied is what makes it "lite")

--
Kevin Wright

gtalk / msn : kev.lee...@gmail.com

mail: kevin....@scalatechnology.com

vibe / skype: kev.lee.wright

quora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

[Alex Cruise]

On Tue, May 24, 2011 at 12:39 AM, kgd <kevi...@gmail.com> wrote:

create a trait that when extended by a class, automatically generates a val
on that class that returns an array of the default constructor
parameters used to instantiate the class.

for example:

trait AutoArgs {
 val autoArgs: Array[Any]
}

Well, if you can restrict yourself to immutable case classes, you don't need a compiler plugin.

case class Foo(x: Int, y: String)

val foo = Foo(1,"one")

foo.productIterator.toArray // : Array[Any] => Array(1, one)

-0xe1a

[kgd]

hi kevin, i went through your AutoProxy example and adapted it to fit
my needs.

the plugin compiles ok, but when i use it to compile a test case, the
error i get is that no override method is detected on the test class -
but if i print out the tree after i add the new def, it shows the new
method is there. any obvious problems in what i'm doing? one issue i
saw was that the tree doesn't show the variables as scoped to the
class, e.g. it's "x" not "D.this.x", but i couldn't figure out how to
change that. thanks -

here's the test case code:

package test.plugin {
trait AutoArgs {
def autoArgs: Array[Any] // should be
overridden
}
}

import test.plugin.AutoArgs

object Test {
class D(x: Int, y: Double, v: Boolean, p: String) extends AutoArgs
}


and the error i get is:

class D needs to be abstract, since method autoArgs is not defined


and the tree after adding the method looks like this:

class D extends java.lang.Object with test.plugin.AutoArgs with
ScalaObject {
override <synthetic> def autoArgs: Array[Any] =
scala.Array.apply[Any](x, y, v, p)(reflect.this.Manifest.Any)
}


and the plugin code:

// this is all set to run after "typer"

val autoArgsClass = definitions.getClass("test.plugin.AutoArgs")
val autoArgsMethod = getMember(autoArgsClass, "autoArgs")

def generateArgs(tree: Tree): Tree = {
var params = tree.symbol.constrParamAccessors
val filteredParams = params filter (x => !(x hasFlag IMPLICIT))
val finalParams = filteredParams map (x => Ident(x))

val newMethod = autoArgsMethod.cloneSymbol(tree.symbol)
newMethod setFlag SYNTHETIC
newMethod setFlag OVERRIDE

tree.symbol.info.decls.enter(newMethod)
if (autoArgsMethod.isStable)
newMethod setFlag STABLE

val rhs: Tree = Apply(Ident("Array"), finalParams)

localTyper.typed {
DEF(newMethod) === rhs
}
}


override def transform(tree: Tree): Tree = {

def shouldGenerateArgs(sym: Symbol) = {
if (sym != null)
(sym isSubClass autoArgsClass) && sym.isClass && !
sym.isTrait
else
false
}

val newTree = tree match {
case ClassDef(mods, name, tparams, impl) if
shouldGenerateArgs(tree.sym\
bol) => {
val newDefDef = generateArgs(tree)
val newImpl = treeCopy.Template(impl,
impl.parents,
impl.self,
impl.body :::
List(newDefDef))
treeCopy.ClassDef(tree, mods, name, tparams, newImpl)
}
case _ => tree
}
super.transform(newTree)

}



On May 24, 10:52 am, Kevin Wright <kev.lee.wri...@gmail.com> wrote:
> In your specific case, this is already possible, because the method is
> already inherited in abstract form and so the thing should type-check
> cleanly even before any synthesis takes place.
>
> Take a look at the autoproxy-lite plugin, where I've relied on exactly this

> "trick"https://github.com/kevinwright/Autoproxy-Lite/tree/master/plugin/src/...

>
> (incidentally, the fact that only already-inherited members can be proxied
> is what makes it "lite")

[Kevin Wright]

On 25 May 2011 00:51, kgd <kevi...@gmail.com> wrote:

hi kevin, i went through your AutoProxy example and adapted it to fit
my needs.

the plugin compiles ok, but when i use it to compile a test case, the
error i get is that no override method is detected on the test class -
but if i print out the tree after i add the new def, it shows the new
method is there. any obvious problems in what i'm doing? one issue i
saw was that the tree doesn't show the variables as scoped to the
class, e.g. it's "x" not "D.this.x", but i couldn't figure out how to
change that. thanks -

typing should take care of that.  You also don't need to override an abstract method, so you might want to remove the OVERRIDE modifier.

Have you got the source in a public repository anywhere?

--
Kevin Wright

gtalk / msn : kev.lee...@gmail.com

mail: kevin....@scalatechnology.com

vibe / skype: kev.lee.wright

quora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

[kgd]

i hope this isn't a double post, i posted earlier and just now
discovered it never went through

still haven't figured out the problem with the method still being
deemed abstract. here's a link to my code: https://github.com/kevinder/autoArgs


On May 24, 5:27 pm, Kevin Wright <kev.lee.wri...@gmail.com> wrote:

[kgd]

so it turns out my previous issue was due to cloning the abstract
symbol without removing the abstract flag. i just assumed this would
happen if it was assigned in a def, but that's not the case.

the issue i'm stuck on now seems similar to the params issue i
mentioned earlier -


the method is generated but there's a compiler error around the
params:

<synthetic> def autoArgs: Array[Any] = scala.Array.apply[Any](x, y, v,
p)(reflect.this.Manifest.Any)

Error during sbt execution: java.lang.Error: symbol value x does not
exist in Test$D.autoArgs


i think the problem lies in the way i'm generating those:

var params = tree.symbol.constrParamAccessors
val finalParams = params map (x => Ident(x))

val rhs: Tree = Apply(Ident("Array"), finalParams)

is there a correct syntax that involves using "This" or something like
that? thanks!

On May 24, 5:27 pm, Kevin Wright <kev.lee.wri...@gmail.com> wrote:

[Kevin Wright]

On 25 May 2011 22:20, kgd <kevi...@gmail.com> wrote:

so it turns out my previous issue was due to cloning the abstract
symbol without removing the abstract flag. i just assumed this would
happen if it was assigned in a def, but that's not the case.

the issue i'm stuck on now seems similar to the params issue i
mentioned earlier -


the method is generated but there's a compiler error around the
params:

<synthetic> def autoArgs: Array[Any] = scala.Array.apply[Any](x, y, v,
p)(reflect.this.Manifest.Any)

Error during sbt execution: java.lang.Error: symbol value x does not
exist in Test$D.autoArgs


i think the problem lies in the way i'm generating those:

var params = tree.symbol.constrParamAccessors

val finalParams = params map (x => Ident(x))

val rhs: Tree = Apply(Ident("Array"), finalParams)

is there a correct syntax that involves using "This" or something like
that? thanks!

My favourite approach in these situations is to manually write the code that I would expect to be synthesized, then browse the resulting AST and use that as a template.

-Ybrowse:typer can be very useful here.

--
Kevin Wright

gtalk / msn : kev.lee...@gmail.com

mail: kevin....@scalatechnology.com

vibe / skype: kev.lee.wright

quora: http://www.quora.com/Kevin-Wright
twitter: @thecoda

[Paul Butcher]

On 24 May 2011, at 17:04, Kevin Wright wrote:

Right now, I'm working on a patch to scalac that provides the necessary hooks for a plugin to inject synthesized methods during the namer & typer phases.  Watch this space for news of further progress.

Kevin,

Not sure if you spotted, but I've managed to get my compiler plugin that works around this issue by generating source files working. It's not the nicest solution ever, but it does work:

http://www.paulbutcher.com/2011/06/towards-typesafe-native-scala-mocks/

If you want a guinea pig to test things out on, or if you think that there's anything else that I could do to help, please do let me know.

--
paul.butcher->msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher

MSN: pa...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher