unable to enforce tree properties with macros

[Roland Kuhn]

Hi folks,

inspired by Simon’s keynote at ScalaDays I started implementing a “static” macro, the purpose of which is to ensure that a function is not capturing any free variables (i.e. it is not actually a closure). While the tree transformation seems doable, I think I’m hitting a road-block when it comes to actually guaranteeing what I want: the macro shall be the only way to construct an instance of type Static[T]. What I tried is the following:

---[macro.scala]---

import scala.reflect.makro.Context

import language.experimental.macros

final case class Static[T] private (body: Product => T, args: Product) {

  def apply(): T = body(args)

}

object Static {

  def static[T](body: T): Static[T] = macro staticImpl[T]

  def staticImpl[T: c.TypeTag](c: Context)(body: c.Expr[T]): c.Expr[Static[T]] = {

    c.reify(makeStatic[T](x => body.eval))

  }

  private def makeStatic[T](x: Product => T) = new Static(x, null)

}

---[Main.scala]---

object Main extends App {

  println(Static.static("hallo" + 42))

}

---[running it … (nightly from yesterday)]---

src/test/Main.scala:7: error: failed to perform typecheck1: method makeStatic in object Static cannot be accessed in object test.Static at source-/Users/rkuhn/comp/scala/testmacro/src/test/Main.scala,line-7,offset=142

  println(Static.static("hallo" + 42))

                       ^

The issue basically is that the tree resulting from a macro must be a legal tree wherever it is inserted, making it impossible to restrict capabilities of any kind to a macro. Am I wrong with this conclusion? Other ideas?

Regards,

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

[Eugene Burmako]

Yeah, that's right. The result of a macro expansion is inlined into the call site, hence it is typechecked within the call site's lexical context. This includes visibility, implicits, whatever.

[Roland Kuhn]

So what about allowing a macro to emit a tree with special “privileges”, i.e. a macro expansion would “live” at the place it is inserted as well as at the package/class it was constructed? (only for the purpose of access modifiers)

Regards,

Roland Kuhn

Typesafe — The software stack for applications that scale

twitter: @rolandkuhn

[Eugene Burmako]

this is a part of a more general problem. how do we allow something to be typechecked in the macro definition context (because c.typecheck also works in the call site context)?

it's essential for hygiene of hand-crafted trees, so this is on our todo list, but no idea when we'll actually have time for this. can you think of a use case where that'd bump our priority?

as a hackaround, you can manually typecheck the tree you're building (by manually assigning symbols and types to it). macro expansion is typechecked by normal typer, and all typers bail when see already typed trees. hence you won't undergo visibility checks, and most likely it will work. ahhh, it won't!! now I remember, why we had troubles when discussed this. JVM visibility checks. you need a way to reliably unprivate/unprotect something by a macro, which is not easy.

[Alex Wilson]

Perhaps my understanding of macros is insufficiently complete, but could you not re-work the code so the AST check macro is called from the body of the constructor of the Static class (now made public) rather than in Static.static?

Thus any constructions of the Static class would include the check...

Alex

[Eugene Burmako]

Wow, nice one!

[Alex Wilson]

I imagine you could use a similar trick to mark off pure functions just using the macro system. I've not been following the effects system dialog in great detail - presumably this is already being considered. Macros look like a huge amount of fun... :-)

Alex

[Roland Kuhn]

Hi Alex,

unfortunately this does not help: the way you describe it you either lose the Tree (by passing the closure at runtime into a factory) or you have the user call reify, in which case the check would again only be performed at runtime (that was the original problem underlying the one in the first post). Even if the constructor IS the macro (which probably does not even compile), its expansion would have the very same problem as described below.

So, it seems that pre-checking the tree would be the only way to “make it work” right now, although I must say that that is a brittle hack in every sense.

Regards,

Roland

[Roland Kuhn]

As long as Martin does not prove otherwise (hint, hint ;-) ) I am inclined to believe that the current macro proposal is not powerful enough in order to enforce properties which are not already enforceable by the compiler itself.

Regards,

Roland

[Eugene Burmako]

Yeah, I was wrong about the constructor idea. Indeed it won't work due to the reasons that Roland described.

Speaking of the solution, even if we made the private ctor work, it'd still be possible to circumvent it using reflection. Hence, I think, making the constructor public would do no harm.

As to enforcing properties, it depends on the definition. E.g. with macros we can validate string literals for conformance with domain-specific rules. Or by wrapping a body of a function in pure {...}, we can have the "pure" macro verify that the body is actually pure. I'm not Martin, of course :)

[Roland Kuhn]

On Apr 21, 2012, at 16:50 , Eugene Burmako wrote:

Yeah, I was wrong about the constructor idea. Indeed it won't work due to the reasons that Roland described.

Speaking of the solution, even if we made the private ctor work, it'd still be possible to circumvent it using reflection. Hence, I think, making the constructor public would do no harm.

Well, .asInstanceOf[] and reflection are universally accepted ways of breaking all kinds of guarantees, whereas calling a public constructor is not. Maybe something like

def `whoever calls this method declare themselves smarter than this macro`(...) = ...

will be enough to signal the intent. (This will make the implementation of said macro recursively funny, though :-) )

As to enforcing properties, it depends on the definition. E.g. with macros we can validate string literals for conformance with domain-specific rules. Or by wrapping a body of a function in pure {...}, we can have the "pure" macro verify that the body is actually pure. I'm not Martin, of course :)

Doing this is a voluntary action of the user of some API, hence I would not call it enforcement.

Regards,

Roland

[Eugene Burmako]

Hmm, what do you mean by enforcement then? Could you give some examples?

[Roland Kuhn]

I mean enforcement at the same level that the type system is enforced: unless you cast yourself, you will not get ClassCastExceptions during runtime. My intended use for the “static” macro is to ensure that you will not get serialization exceptions at runtime (amongst others). The equivalent of a cast would in this case be

Props(Static(<my evil function>, ...)) // might be named differently as suggested below

instead of the safe

Props(static(<my safe function>))

Regards,

Roland

[Oleg Galako]

On Saturday, April 21, 2012 10:42:39 PM UTC+7, rkuhn wrote:

Well, .asInstanceOf[] and reflection are universally accepted ways of breaking all kinds of guarantees, whereas calling a public constructor is not.

So maybe then it should be you who calls .asInstanceOf[]? Not too beatiful, but still better than nothing:

import scala.reflect.makro.Context

import language.experimental.macros

final case class Static[T] private (body: Product => T, args: Product) {

  def apply(): T = body(args)

}

object Static {

  def static[T](body: T): Static[T] = macro staticImpl[T]

  def staticImpl[T: c.TypeTag](c: Context)(body: c.Expr[T]): c.Expr[Static[T]] = {

    c.reify(makeStatic[T](x => body.eval).asInstanceOf[Static[T]])

  }

  def makeStatic[T](x: Product => T): Any = new Static(x, null)

} 

[martin odersky]

Another idea would be to have a public factory method with '$' characters in it. You are not supposed to use '$' in user code.

 - Martin

--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

[Roland Kuhn]

A combination sounds good: have the factory name contain $ and require a cast, that should make it sufficiently clear.

Now that I consider this issue solved, I’m left with the little technicality that lambdaLift comes way after typer … (for easily spotting free term names)

But what would life be without the occasional challenge?

Regards,

Roland

[Eugene Burmako]

You can reify the body in a macro (with c.reifyTree) and look for free terms in the result.

[Roland Kuhn]

I tried that, but my Tree-fu is obviously not good enough: how is a free term different from anything else in that result? c.isLocatable(tree.symbol) did not give the result I was looking for; basically I want to make sure that in the end the generated Function1 subclass does not take any constructor arguments (where the elimination of the $outer reference will depend on teaching the compiler a new trick).

[Paul Phillips]

On Sun, Apr 22, 2012 at 2:35 AM, martin odersky <martin....@epfl.ch> wrote:
> Another idea would be to have a public factory method with '$' characters in
> it. You are not supposed to use '$' in user code.

Please don't use $ this way! It is a guarantee that when you use $
now, or devise any new ad hoc mangling at all, you create new bugs.
There is a chance we can make most of the name mangling problem go
away by taking advantage of symbolic freedom at the vm level, but even
if we do that, it will still be important that people not casually use
'$' for nonessential purposes. Java makes assumptions about '$' which
we cannot affect, and we have to honor those assumptions.

[martin odersky]

I agree. But I would argue that "static" is an essential purpose. It's the foundation of serializing closures, judged important enough to require a language extension for Haskell. 

Still, it seems we are probably good enough with just the cast, so no dollar mangling would be needed.

Cheers

 - Martin

[Paul Phillips]

On Sun, Apr 22, 2012 at 4:28 AM, martin odersky <martin....@epfl.ch> wrote:
> I agree. But I would argue that "static" is an essential purpose. It's the
> foundation of serializing closures, judged important enough to require a
> language extension for Haskell.

I wasn't suggesting the method is nonessential, but that the
application of '$' as a way to signal "don't use this, users" is
nonessential.

[Eugene Burmako]

scala> import scala.reflect.makro.Context
import scala.reflect.makro.Context

scala> def impl(c: Context)(x: c.Expr[Any]) = { println(c.reifyTree(c.reflectMirrorPrefix, x.tree));
c.literalUnit }
impl: (c: scala.reflect.makro.Context)(x: c.Expr[Any])c.Expr[Unit]

scala> def foo(x: Any) = macro impl
foo: (x: Any)Unit

scala> { val x = 2; foo(x) }
{
  val $mr: `package`.this.mirror.type = `package`.this.mirror;
  val free$x1 = $mr.newFreeTerm("x", $mr.staticClass("scala.Int").asTypeConstructor, x, 17592186044416L, "defined by res4 in <console>:14:21");
  $mr.Expr[Int]($mr.Ident(free$x1))($mr.ConcreteTypeTag[Int]($mr.staticClass("scala.Int").asTypeConstructor, Predef.classOf[Int]))
}

Look for newFreeTerm calls in the result of c.reifyTree. If they're not there, you're good.

You should also watch out newFreeType calls. These are created every time a reifee refers to a non-locatable and non-spliceable type (e.g. to a local class, or to a non-tagged type parameter). We can serialize those and expect that the receiver will substitute them with real types, but I'm not sure whether this goes along the lines of your vision of serialization.

[Stefan Zeiger]

On 2012-04-21 09:12, Eugene Burmako wrote:
> ahhh, it won't!! now I remember, why we had troubles when discussed
> this. JVM visibility checks. you need a way to reliably
> unprivate/unprotect something by a macro, which is not easy.

At the JVM level, the constructor is public in this case. I don't
remember the exact rules but I think anything other than private[this]
is not really made private in the bytecode.

C:\Users\szeiger\Desktop\stest>javap Static
Compiled from "Static.scala"
public final class Static extends java.lang.Object implements
scala.ScalaObject,scala.Product,scala.Serializable{
...
public Static(scala.Function1, scala.Product);
}

[√iktor Ҡlang]

Sprinkle some magic on it: 

case class Static[T](f: Product => T, a: Product, magic: String) {

  if(magic != "I am not trying to cheat, because that would be illegal")

    assert(false, "Attempt to create a Static with the wrong kind of magic. HERESY!!!")

}

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Roland Kuhn]

:-)

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn