java.lang.RuntimeException: Method code too large!

[Bjorn Regnell]

I am working on a scala-internal DSL that "serialize" its data structures into scala code that get's interpreted back from files to generate and manipulate big, domain-specific models. Now I'm stuck with the problem of "RuntimeException: Method code too large!" for realistic sized models.
Can I get around this through some compiler option?
Is it the scala compiler that generates too large chunks of byte code for the JVM and is it principally fixable in a future scala version?
Any help or advice appreciated.
A demonstrating test case to illustrate this using Scala REPL 2.10.1 below.
/Bjorn

Welcome to Scala version 2.10.1 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_02).
Type in expressions to have them evaluated.
Type :help for more information.

scala> case class Test(i: Int*)
defined class Test

scala> $intp.interpret((for (i <- 1 to 2436) yield s"Test($i)").mkString("Vector(",",",")") )
error:
     while compiling: <console>
        during phase: jvm
     library version: version 2.10.1
    compiler version: version 2.10.1
  reconstructed args:

  last tree to typer: Apply(method $asInstanceOf)
              symbol: method $asInstanceOf in class Object (flags: <method> final <synthetic> <triedcooking>)
   symbol definition: final def $asInstanceOf[T0 >: ? <: ?](): T0
                 tpe: collection.immutable.Vector
       symbol owners: method $asInstanceOf -> class Object -> package lang
      context owners: object iw -> package $line5

== Enclosing template or block ==

Template( // val <local $iw>: <notype>, tree.tpe=type
  "java.lang.Object" // parents
  ValDef(
    private
    "_"
    <tpt>
    <empty>
  )
  DefDef( // def <init>(): type
    <method>
    "<init>"
    []
    List(Nil)
    <tpt> // tree.tpe=type
    Block( // tree.tpe=Unit
      Apply( // def <init>(): Object in class Object, tree.tpe=Object
        $read$super."<init>" // def <init>(): Object in class Object, tree.tpe=()Object
        Nil
      )
      ()
    )
  )
)

== Expanded type of tree ==

TypeRef(
  TypeSymbol(
    final class Vector extends collection.AbstractSeq with collection.immutable.IndexedSeq with collection.generic.GenericTraversableTemplate with collection.In
dexedSeqLike with collection.immutable.VectorPointer with Serializable with collection.CustomParallelizable

  )
)

uncaught exception during compilation: java.lang.RuntimeException
java.lang.RuntimeException: Method code too large!
        at scala.tools.asm.MethodWriter.getSize(Unknown Source)
        at scala.tools.asm.ClassWriter.toByteArray(Unknown Source)
        at scala.tools.nsc.backend.jvm.GenASM$JBuilder.writeIfNotTooBig(GenASM.scala:457)
        at scala.tools.nsc.backend.jvm.GenASM$JPlainBuilder.genClass(GenASM.scala:1402)
        at scala.tools.nsc.backend.jvm.GenASM$AsmPhase.run(GenASM.scala:119)
        at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1582)
        at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1556)
        at scala.tools.nsc.Global$Run.compileSources(Global.scala:1552)
        at scala.tools.nsc.interpreter.IMain.compileSourcesKeepingRun(IMain.scala:428)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compileAndSaveRun(IMain.scala:801)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.compile(IMain.scala:761)
        at scala.tools.nsc.interpreter.IMain$Request.compile$lzycompute(IMain.scala:936)
        at scala.tools.nsc.interpreter.IMain$Request.compile(IMain.scala:931)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:603)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568)
        at .<init>(<console>:9)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734)
        at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983)
        at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568)
        at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:745)
        at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:790)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:702)
        at scala.tools.nsc.interpreter.ILoop.processLine$1(ILoop.scala:566)
        at scala.tools.nsc.interpreter.ILoop.innerLoop$1(ILoop.scala:573)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:576)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:867)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:822)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:822)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:822)
        at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:83)
        at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:96)
        at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:105)
        at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)


scala>

[Paul Phillips]

On Sat, May 25, 2013 at 10:54 AM, Bjorn Regnell <bjorn....@gmail.com> wrote:

scala> case class Test(i: Int*)
defined class Test

scala> $intp.interpret((for (i <- 1 to 2436) yield s"Test($i)").mkString("Vector(",",",")") )

Heh, I see you drilled down to 2436 with precision, since that generates a method which is 65535 bytes, which is exactly the maximum method size on the jvm. (It will fail given directly in the repl because of additional wrapping, but as given can be compiled within a class.)

Instantiating Test(i) is about 24 bytes (slightly fewer for smaller numbers, but it creeps upward from iconst_N to bipush to sipush to ldc.) Each of them must be stored in an array for the call to Vector.apply. It adds up.

In this example you could shrink it by 1000x by writing "1 to 2436 map (i => Test(i)) toVector". I understand it's only an example, but I point that out because even in the non-example version the same principle will be at work: if you generate lots of literal code, you will generate lots of literal bytecode. If you don't want 65K+ methods, generate code more like a human might write.

So no, there's nothing material scala can do to help you here. But it's relatively easy to help yourself once you know what you're up against. At worst you can always automate the division into smaller methods.

[Bjorn Regnell]

Many thanks, Paul, for this enlightening clarification!

> If you don't want 65K+ methods, generate code more like a human might write.

Well, my problem is that it is actually humans that write such large-sized parameter sequences to the apply-method of my deeply nested case class factory object apply calls.
This seems to be kind of a show stopper for my approach to express models as to-be-interpreted case class-based Scala DSL code strings, so I might have to rethink the whole concept as I realize that models can get too big by the nature of the domain... :-(
This may be a stupid question as I do not know that much about what is going on behind the scenes in the interpreter, but:
Couldn't the interpreter somehow potentially chop things up automatically to come around such problems and avoiding too-large-method-crashes of scala-internal-DSL-with-gigantic-apply-clause-users?
/Bjorn

[Paul Phillips]

On Sat, May 25, 2013 at 12:32 PM, Bjorn Regnell <bjorn....@gmail.com> wrote:

Couldn't the interpreter somehow potentially chop things up automatically to come around such problems and avoiding too-large-method-crashes of scala-internal-DSL-with-gigantic-apply-clause-users?

Well, technically it could. The interpreter is not a robust basis upon which to base production code. It has many weak points even relative to the compiler, which is itself rather a soft target.

So "it could" but there is so much to do of higher priority than catering to people generating 65K+ methods, I think it unlikely that "it will". It might be a good place to make a contribution.

[Bjorn Regnell]

Ah - then at least my Q was not unreasonable :-)

 It might be a good place to make a contribution.

A contribution would be fabulous, but I'm afraid I'm lacking the competence to do it myself...
I have full appreciation for this not getting priority; do you think it might be a good idea anyway if I filed an issue - maybe some virtuous happen to invent a quick-n-easy fix?

[Rex Kerr]

Ugh, this is not quick or easy.  You either have a horrible kludge or you have to cut your dependency tree at every plausible place and find some way to optimize the number of variables that you have to pass forwards or in to reduce them to a sensible level.

It's way, way better if you can use extra knowledge of what you need to accomplish to pick the points yourself than expect someone to do it for the general case.

  --Rex

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.