variable in macro implementation

[antoine]

Hello,

I would just quickly try macro, so i write:

object MacroRegex {
def build(text: String): Regex = macro buildImp
def buildImp(c: Context)(text: c.Expr[String]): c.Expr[Regex] = {
var reg: Regex = null
try { reg = text.eval.r }
catch {
case e: java.util.regex.PatternSyntaxException =>
c.error(c.enclosingPosition, e.getMessage())
}
c.reify(reg)
}
}

The aim is to get regexp syntax error at compile time.
But i get this cryptic error:
" error: macro expansion contains free term variable reg defined by
buildImp in regex.scala:10:9. have you forgot to use eval when
splicing this variable into a reifee? if you have troubles tracking
free term variables, consider using -Xlog-free-terms "

I remove the reg variable and get a correct code.

object MacroRegex {
def build(text: String): Regex = macro buildImp
def buildImp(c: Context)(text: c.Expr[String]): c.Expr[Regex] = {
try { text.eval.r }
catch {
case e: java.util.regex.PatternSyntaxException =>
c.error(c.enclosingPosition, e.getMessage())
}
c.reify(text.eval.r)
}
}


My question is how to use variable in macro implementation?
(i use scala 2.10-snapshot)

Thanks you.

[Johannes Rudolph]

This works for me:

def mregex(c: Context)(t: c.Expr[String]): c.Expr[Regex] = try {
// test regex
t.eval.r

c.reify(t.eval.r)

} catch {
case e: java.util.regex.PatternSyntaxException =>
c.error(c.enclosingPosition, e.getMessage())

throw e
}

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Johannes Rudolph]

Ah ok, I see that's exactly what you did. So ignore what I sent.

[antoine no]

do you know project that use scala macro where i can look at it?

[Eugene Burmako]

Hey antoine,

The problem you have stems from the fact that reify works in an unexpected way. Let's study the result of c.reify(reg) (we will take a look at a simplified snippet, you can get the full code dumped to the console if you enable -Yreify-copypaste):

{

  val $mr: scala.reflect.`package`.mirror.type = scala.reflect.`package`.mirror;
  val free$reg1 = $mr.newFreeTerm("reg", $mr.TypeRef($mr.thisModuleType("scala.runtime"), $mr.staticClass("scala.runtime.ObjectRef"), scala.collection.immutable.List.apply($mr.NoType)), reg, 17592186114048L, "defined by res1 in <console>:10:21");
  $mr.Expr[Regex]($mr.Ident(free$reg1))

} 

Here we go. When reifying reg, the compiler creates a reference to the value reg from the scope. To express that, it makes up a free term and refers to it from the Ident node. Note that this free term captures the value of the reg variable (the 3rd parameter to newFreeTerm function). This is done to ensure that reified exprs don't lose the connections to their scopes.

What happens next? This tree is returned as a result of macro expansion, and it tells to the compiler: "hey, I want you to generate the code that (during the runtime) will refer to my variable reg". But that is impossible, because reg exists only during the compile-time. Hence the problem.

Bottom line. Macro expansions cannot refer to the values defined in a macro unless you splice.

Cheers,

Eugene

[Johannes Rudolph]

On Fri, Apr 20, 2012 at 11:51 AM, antoine <antoi...@gmail.com> wrote:

>    var reg: Regex = null
>    try { reg = text.eval.r }
>    catch {
>      case e: java.util.regex.PatternSyntaxException =>
> c.error(c.enclosingPosition, e.getMessage())
>    }
>    c.reify(reg)
>  }

>

> My question is how to use variable in macro implementation?

I think you are confusing stages here: The reg variable contains here
the value of evaluating the expression at macro expansion time. What
you are then trying to do when using `reify` is to transfer this value
to runtime which isn't possible per se. In the example which works you
are doing something else: You are generating code which at runtime
generates the regular expression.

[Eugene Burmako]

Type tag generation in Scala: https://github.com/scala/scala/blob/master/src/library/scala/reflect/makro/internal/Utils.scala

Macrocosm: https://github.com/retronym/macrocosm

Expecty: https://github.com/pniederw/expecty

[Johannes Rudolph]

On Fri, Apr 20, 2012 at 12:08 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Bottom line. Macro expansions cannot refer to the values defined in a macro
> unless you splice.

An additional note: The important thing you have to always remember
when dealing with macros is that you are really mainly operating on
expression trees, not on runtime values. So, if you splice (using
`.eval` inside of `reify`) you have to splice in trees not values.

Incidentally: I think the surprising thing here is that evaluating
expression trees (using `.eval` outside of `reify`) at macro expansion
time works for some expressions. Eugene, is there some effort to
improve error messages in cases this fails? For example, calling the
example macro from this thread with `MacroRegex.build("abc"*3)`
results in a

[error] /home/johannes/git/self/scala-days/src/test/scala/MacroTest.scala:2:
exception during macro expansion:
[error] scala.tools.nsc.ToolBoxes$ToolBox$ToolBoxError: reflective
toolbox has failed: cannot operate on trees that are already typed
[error] at scala.tools.nsc.ToolBoxes$ToolBox.runExpr(ToolBoxes.scala:50)
[error] at scala.reflect.api.Exprs$Expr.eval(Exprs.scala:15)

at compilation of user code which may be confusing. Probably it's the
responsibility of the macro to catch those problems, are there any
helpers to do that effectively?

[Eugene Burmako]

This error happens because eval uses reflective compiler under the covers. Due to an implementation restriction, the reflective compiler cannot deal with trees that are already typechecked, hence the error. OTOH, we could automatically untype the trees that are evaluated with eval, but that might create misleading errors of a different flavor. 

All in all, using eval outside of reify in a macro isn't a very good idea, because it might easily create cross-stage problems. It's more robust to parse the AST manually. Sure, c.reify("a").eval works. c.resetAllAttrs(c.reify("a" * 3)).eval will most likely work as well. But c.reify(f("a")).eval won't work if f is defined in the current compilation run.

[antoine no]

Ok i get it, i miss undertood between with the life of reg, which live in macro-expension, and the fact that i return AstTree and not computed state of objects(in that case a regex object).

To make sure i understand well, i will setup a little use case and try to answer it:
- I want to make a long computation at compile time instead of runtime.
- I have a class A { def compute(/*some arg*/) //change internal state of object
                            def serialize: String //return the current state of object
                            def deSerialize(state: String) //mutate the object with state
 }
- i will have a macro:
def compileCompute(/*some arg*/): A = macro compileComputeImpl

def compileComputeImpl(c: Context)(/*some arg*/): c.Expr[A] = {
  val a = A().compute(/*some arg*/.eval)
  val s = a.serialize
  c.reify(A().deSerialize(s))
}

writting that i relize the output of compileComputeImpl will be:
<[ A().deSerialize(a) ]> when i expected <[ A().deSerialize("some text") ]>
Need i to write AstTree by hand?
Or is there a way to extract value of "a" inside reify?

[Eugene Burmako]

you can use eval:

<[ A().deSerialize(Expr[String](Literal(Constant(s))).eval) ]>  

or a shorter version:

<[ A().deSerialize(c.literal(s).eval) ]>

except that A() won't compile, so you'll have to use a factory or a typeclass for that, but that's orthogonal to the idea with reify. to learn more about eval, take a look at the SIP:  https://docs.google.com/document/d/1O879Iz-567FzVb8kw6N5OBpei9dnbW0ZaT7-XNSa6Cs/edit (sections "Reify" and "Splicing").

[Eugene Burmako]

oops, thought A in your example was a type parameter. sorry, everything will work fine if A is some class defined externally.

[antoine no]

I have write a concrete example of the Serialilze/Deserialize stuff.
I didn't manage to deal with the second parameter of Regex the "groupNames: String*"  in macro.
I am not sure of the second parameter of ArrayValue().

import scala.language.experimental.macros
import scala.reflect.makro.Context
import scala.util.matching.Regex
import java.io.{ObjectOutputStream, ObjectInputStream, ByteArrayOutputStream, ByteArrayInputStream}

class Reg(regex: String, groupNames: Seq[String]) extends Regex(regex, groupNames:_*) with Serializable {
  //transform a Reg into an Array[Byte]
  def write = {
    val b = new ByteArrayOutputStream()
    val out = new ObjectOutputStream(b)
    out.writeObject(this)
    out.close()
    b.toByteArray
  }
}

object Reg {
  //create a Reg from an Array[Byte]
  def read(b : Array[Byte]): Reg = {
    val r = new ObjectInputStream(new ByteArrayInputStream(b))
    val o = r.readObject()
    r.close()
    o.asInstanceOf[Reg]
  }
}

object RegMacro {
 
  def go(regex: String): Array[Byte] = macro goImpl
  def goImpl(c: Context)(regex: c.Expr[String]): c.Expr[Array[Byte]] = {
    val r = new Reg(regex.eval, List().toSeq)
    val a = r.write
    import c.mirror._
    val e = Expr[Array[Byte]](ArrayValue(TypeTree(ByteTpe), a.toList.map(e => Literal(Constant(e)))))
    c.reify(e.eval)
  }
}

val arr = RegMacro.go("pa[tT]ern") //executed at compile time
//if you expand the previous line: val arr = Array(/*number*/, /*number*/, ...)
val r = Reg.read(arr) //executed at run time

[Daniel Sobral]

On Fri, Apr 20, 2012 at 06:51, antoine <antoi...@gmail.com> wrote:
> Hello,
>
> I would just quickly try macro, so i write:
>
> object MacroRegex {
>  def build(text: String): Regex = macro buildImp
>  def buildImp(c: Context)(text: c.Expr[String]): c.Expr[Regex] = {
>    var reg: Regex = null
>    try { reg = text.eval.r }

So, I don't have a recent Scala here to try, but wouldn't this work?

val regExpr = reify(text.eval.r)
try regExpr.eval catch { ... }
regExpr

>    catch {
>      case e: java.util.regex.PatternSyntaxException =>
> c.error(c.enclosingPosition, e.getMessage())
>    }
>    c.reify(reg)
>  }
> }
>
> The aim is to get regexp syntax error at compile time.
> But i get this cryptic error:
> " error: macro expansion contains free term variable reg defined by
> buildImp in regex.scala:10:9. have you forgot to use eval when
> splicing this variable into a reifee? if you have troubles tracking
> free term variables, consider using -Xlog-free-terms "
>
> I remove the reg variable and get a correct code.
>
> object MacroRegex {
>  def build(text: String): Regex = macro buildImp
>  def buildImp(c: Context)(text: c.Expr[String]): c.Expr[Regex] = {
>    try { text.eval.r }
>    catch {
>      case e: java.util.regex.PatternSyntaxException =>
> c.error(c.enclosingPosition, e.getMessage())
>    }
>    c.reify(text.eval.r)
>  }
> }
>
>
> My question is how to use variable in macro implementation?
> (i use scala 2.10-snapshot)
>
> Thanks you.

--
Daniel C. Sobral

I travel to the future all the time.

[Johannes Rudolph]

On Sat, Apr 21, 2012 at 6:11 PM, antoine no <antoi...@gmail.com> wrote:
>     c.reify(e.eval)

This is the same as just `e`. Apart from that you can include
deserialization into the macro as well:

object RegMacro {

def go(regex: String): Regex = macro goImpl
def goImpl(c: Context)(regex: c.Expr[String]): c.Expr[Regex] = {

val r = new Reg(regex.eval, List().toSeq)
val a = r.write
import c.mirror._
val e = Expr[Array[Byte]](ArrayValue(TypeTree(ByteTpe),
a.toList.map(e => Literal(Constant(e)))))

c.reify(Reg.read(e.eval))

[antoine no]

Daniel: it works but it don't generate a PatternSyntaxException, it say NullPointerException during macro expension but i can't catch it
 def buildImp(c: Context)(text: c.Expr[String]): c.Expr[Regex] = {
    val exp = c.reify(text.eval.r)
    try  exp.eval

    catch {
      case e: java.util.regex.PatternSyntaxException => c.error(c.enclosingPosition, e.getMessage())

      case a => println("case a") ;  c.error(c.enclosingPosition, a.getMessage())
    }
    exp