Exception in ClassfileParser during macro expansion due to TypeTree?

[Som Snytt]

While working to nuance a certain fast-track macro, adding some polish to handle

f"${null}%#s"    // DNC!

it suddenly blew up.

The fun part is that it worked "normally" (in REPL), but failed under partest, while building the class path for expansion.

macro expansion has failed: null

It's using context.eval to evaluate constants, ignoring errors.

The problem was the ValDef with a TypeTree as shown in the reduction. Maybe someone can tell me why that was wrong.

I mention it because the failure mode was catastrophic. Maybe it's a class loader or class path interaction.

The symptom was a  slew of:

error: error while loading HashMap$EntryIterator, class file '/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar(java/util/HashMap$EntryIterator.class)' is broken
(class java.util.NoSuchElementException/key not found: K)

Sources named for partest:

// Macros_1.scala

package bakery

import scala.language.experimental.macros
import scala.reflect.macros.blackbox.Context

object Bakery {

  def f: Any = macro fImpl

  def fImpl(c: Context): c.Expr[Any] = {
    import c.universe._

    val n = c.freshName("x")
    val d = ValDef(Modifiers(), n, TypeTree(typeOf[java.util.Formattable]), Literal(Constant(1)))
    val expr = Block(d, Ident(n))
    val expr0 = q"{ val x: java.util.Formattable = 1 ; x }"

    //Block(List(ValDef(Modifiers(), TermName("x$macro$1"), TypeTree(), Literal(Constant(1)))), Ident(TermName("x$macro$1")))
    //Block(List(ValDef(Modifiers(), TermName("x"), Select(Select(Ident(TermName("java")), TermName("util")), TypeName("Formattable")), Literal(Constant(1)))), Ident(TermName("x")))
    Console println showRaw(expr)
    Console println showRaw(expr0)

    //try c.eval(c.Expr[Any](expr0)) catch { case t: Throwable => t.printStackTrace() }
    try c.eval(c.Expr[Any](expr)) catch { case t: Throwable => t.printStackTrace() }

    c.Expr[Any](Literal(Constant(1)))
  }
}

// Test_2.scala

object Test extends App {
  val v = bakery.Bakery.f
  Console println v
}

Throws under partest:

java.lang.AssertionError: assertion failed: class WeakHashMap: <none>
    at scala.Predef$.assert(Predef.scala:165)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parseMethod(ClassfileParser.scala:574)
    at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$scala$tools$nsc$symtab$classfile$ClassfileParser$$queueLoad$1$2.apply$mcVI$sp(ClassfileParser.scala:477)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:166)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.scala$tools$nsc$symtab$classfile$ClassfileParser$$queueLoad$1(ClassfileParser.scala:477)
    at scala.tools.nsc.symtab.classfile.ClassfileParser$$anonfun$parseClass$1.apply$mcV$sp(ClassfileParser.scala:487)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:492)

Compiling by hand catches the ordinary error:

scala.tools.reflect.ToolBoxError: reflective compilation has failed:

type mismatch;
 found   : Int(1)
 required: java.util.Formattable

[Eugene Burmako]

I think that the error is caused by a strange classloader configuration that is used to create the evaluating toolbox. I vaguely remember that partest uses a very non-trivial scheme of creating classloaders, so that might be too much for the toolbox.

Just wondering, does it also crash, when an expression being evaluated doesn't have type errors?

All in all, I'd probably recommend avoiding c.eval when there are alternative ways of doing things. Apart from still being brittle (e.g. see SI-6393), toolboxes also take a long time to initialize, so there should be a really good reason to reach for them. Do you think there might be a workaround for your case, or you absolutely need a toolbox?

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Som Snytt]

OK, thanks.

Yes, it shows the same behavior with `val x: java.util.Formattable = null`.

It's not a problem with primitives.

I also wondered if there was an interaction with the recent "flat class path" change, but I haven't pursued that.

I remember seeing the types expressed in output as `util.this.Formattable`, but maybe the missing `java` is normal.

OK, I'll avoid c.eval; it was just a quick way to evaluate constant expressions. I mean quick for me.

[Jason Zaugg]

On Mon, Feb 9, 2015 at 10:11 AM, Som Snytt <som....@gmail.com> wrote:

OK, thanks.

Yes, it shows the same behavior with `val x: java.util.Formattable = null`.

It's not a problem with primitives.

I also wondered if there was an interaction with the recent "flat class path" change, but I haven't pursued that.

I remember seeing the types expressed in output as `util.this.Formattable`, but maybe the missing `java` is normal.

OK, I'll avoid c.eval; it was just a quick way to evaluate constant expressions. I mean quick for me.

I can reproduce outside of partest and Toolbox, using an Importer from the runtime reflection universe is sufficient.

% tail test/files/run/som/*.{java,scala}
==> test/files/run/som/Java_1.java <==

public class Java_1<K, V> {
    public class Inner extends Java_1<K, V> {

    }
}

class SomeOtherJava {

}
==> test/files/run/som/Macros_1.scala <==
  def fImpl(c: Context): c.Tree = {
    import c.universe._
    import reflect.runtime.{universe => ru}

    val formatterSym = c.universe.symbolOf[bakery.SomeOtherJava]
    ru.internal.createImporter(c.universe).importSymbol(formatterSym)

    q"()"
  }
}

==> test/files/run/som/Test_2.scala <==
object Test {
  bakery.Bakery.f
}

% javac -d /tmp test/files/run/som/*.java && qscalac -classpath /tmp -d /tmp test/files/run/som/Macros_1.scala && qscalac -classpath /tmp -d /tmp test/files/run/som/Test_2.scala
error: error while loading Java_1$Inner, class file '/tmp/bakery/Java_1$Inner.class' is broken
(class java.util.NoSuchElementException/key not found: K)
one error found

Importer forces the info of the owner of SomeOtherJava. This then runs ClassfileParser over Java_1$Inner. References it its type signatures to the type parameters of the enclosing class trigger the NSEE. We’d need to contrast this with a situation where these enclosing type parameters are visible (e.g regular compilation), so as to understand where the problem lies.

​

-jason

[Jason Zaugg]

On Mon, Feb 9, 2015 at 11:10 AM, Jason Zaugg <jza...@gmail.com> wrote:

I can reproduce outside of partest and Toolbox, using an Importer from the runtime reflection universe is sufficient.

Here's how to trigger the same error with the regular compiler.

% tail sandbox/{Java_1.java,test.scala}
==> sandbox/Java_1.java <==
public class Java_1<K, V> {
    public class Inner extends Java_1<K, V> {
    }
}

==> sandbox/test.scala <==
object Test {
  classOf[Java_1$Inner]
}

% javac -d /tmp sandbox/Java_1.java && qscalac -classpath /tmp sandbox/test.scala
error: error while loading Java_1$Inner, class file '/tmp/Java_1$Inner.class' is broken
(class java.util.NoSuchElementException/key not found: K)
sandbox/test.scala:2: warning: a pure expression does nothing in statement position; you may be omitting necessary parentheses
  classOf[Java_1$Inner]
         ^
one warning found
one error found

​

-jason

[Jason Zaugg]

Logged as https://issues.scala-lang.org/browse/SI-9152

-jason

[Som Snytt]

Thanks very much.

I'd spent some time before asking about it, because it was a flaky symptom and I thought I must be doing something obviously wrong. It would succeed, I'd tinker, and it would fail. That can be educational, or what's the opposite of edutainment, but I ran out of free time.

From inside the macro, I couldn't see the trees for the trees. (That's what it says on the back of my new "Lost in Scala" t-shirt.)

Moreover, it's a case of, You're focused on your immediate domain problem and there's no capacity to think about pipeline issues.

You must tire of people saying, I can't believe how quickly you minimized it! And so cleanly!