Something is very wrong with 'private[this] final var'

[Sébastien Bocq]

Here is the problem: somehow the test below fails when TMP is declared
as a 'private[this] final var'.

final class Box {
private[this] final var TMP:Object = null

final def put(o:Object) {
TMP = o
}

final def get() = {
val tmp = TMP
TMP = null
tmp
}
}

class Client(box:Box) {
var count = 0

final def run(cycles:Int) {
if (cycles > 0) {
box.put(new Object)
if (box.get() != null) count += 1
run(cycles - 1)
}
}
}

object Main {

def main(args:Array[String]) {
val c = new Client(new Box)
val N = 1000000000
c.run(N)
assert(c.count == N, c.count)
}

}

$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Server VM (build 20.1-b02, mixed mode)

$ scala -version
Scala code runner version 2.9.1.final -- Copyright 2002-2011, LAMP/EPFL

$ sbt run
[error] (run-main) java.lang.AssertionError: assertion failed: 999982178
java.lang.AssertionError: assertion failed: 999982178
at scala.Predef$.assert(Predef.scala:103)
at Main$.main(Bug.scala:33)
at Main.main(Bug.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)

This issue is reproducible 100% of the time and the test passes if I
omit the 'final' modifier for TMP.

So, who is the culprit? JVM or Scala?

Thanks,
Sébastien

[Roland Kuhn]

Trying your code in the REPL, I only get count incremented 14156 times, and not anymore thereafter. I was always wondering why scalac emits mutable fields which are declared final in the byte code: could it be that Box.get() gets inlined and collapsed into returning the well-known “null” which was in the field during JIT compilation?

The only difference in the byte code (at least as printed by javap) is the final modifier on the field, and removing that fixes it.

Regards,

Roland

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

[Adriaan Moors]

wow, yeah I'd say that's a bug in scalac's backend (most closely related: https://issues.scala-lang.org/browse/SI-3569)

I haven't  checked the JVM spec, but it seems it shouldn't accept bytecode that assigns to a final variable outside of a constructor

I got varying values of c.count when it crashed on several runs -- as Roland said, probably due to when the JIT's inliner kicks in

FWIW, here's the disassembled code, manually reworked to Java, commenting out the final modifier that's rejected by javac, but which is in the bytecode generated by scalac

final class Box

{

  public final void put(Object o)

  {

    TMP = o;

  }

  public final Object get()

  {

    Object tmp = TMP;

    Object _tmp = null;

    TMP = null;

    return tmp;

  }

  public Box()

  {

    Object _tmp = null;

  }

  private /*final*/ Object TMP = null;

}

class Client {

  public int count()

  {

    return count;

  }

  public void count__eq(int i)

  {

    count = i;

  }

  public final void run(int cycles)

  {

    for(; cycles > 0; cycles--)

    {

      box.put(new Object());

      if(box.get() != null)

        count__eq(count() + 1);

    }

  }

  public Client(Box box)

  {

    super();

    this.box = box;

    count = 0;

  }

  private final Box box;

  private int count;

}

public final class Main

{

  public static void main(String args[])

  {

    Client c_1 = new Client(new Box());

    int N = 0x3b9aca00;

    c_1.run(N);

    if(c_1.count() != N) throw new RuntimeException();

  }

}

[Dominik Gruntz]

> I haven't  checked the JVM spec, but it seems it shouldn't accept bytecode that assigns to a final variable outside of a constructor

unfortunately, the JVM does allow that. This is used upon deserialization where no constructor is called and where the final fields have to be set. With generated bytecode it is possible to write to final fields after object construction. See Cliff Click's blog post at [1]. He also mentions that code may "behave differently" when the JIT kicks in.

Dominik

[1] http://www.azulsystems.com/blog/cliff/2011-10-17-writing-to-final-fields-via-reflection

[√iktor Ҡlang]

On Wed, Mar 7, 2012 at 10:26 PM, Dominik Gruntz <dominik...@fhnw.ch> wrote:

> I haven't  checked the JVM spec, but it seems it shouldn't accept bytecode that assigns to a final variable outside of a constructor

unfortunately, the JVM does allow that. This is used upon deserialization where no constructor is called and where the final fields have to be set. With generated bytecode it is possible to write to final fields after object construction. See Cliff Click's blog post at [1]. He also mentions that code may "behave differently" when the JIT kicks in.

Also, never forget: setAccessible(true)

 

Dominik

[1] http://www.azulsystems.com/blog/cliff/2011-10-17-writing-to-final-fields-via-reflection

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Dominik Gruntz]

> Also, never forget: setAccessible(true)
this allows to access private fields. But in the given example, it seems that a *final* field is modified.
In the original problem you can remove the private[this], this is not the problem, it is the final declaration which (according to the decompiled code) is translated into a final field in the byte code and which is modified. Strange.

Dominik

[Roland Kuhn]

What Viktor meant is that reflection also allows writing to final fields; setAccessible(true) enters the picture because all fields generated by scalac are private.

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

[Tony Morris]

See java.lang.System.setOut/In/Err for examples of methods that modify final fields.

[Dominik Gruntz]

but that is implemented using the native methods setIn0/setOut0/setErr0, not using reflection. Is scalac using such a trick?
Dominik

[Kevin Wright]

The whole business can actually get very scary indeed.  There's some great examples, redefining the value of boxed primitives, in this stack overflow question:

http://stackoverflow.com/questions/2481862/how-to-limit-setaccessible-to-only-legitimate-uses

--
Kevin Wright
mail: kevin....@scalatechnology.com
gtalk / msn : kev.lee...@gmail.com

quora: http://www.quora.com/Kevin-Wright

google+: http://gplus.to/thecoda

twitter: @thecoda

vibe / skype: kev.lee.wright

steam: kev_lee_wright

"My point today is that, if we wish to count lines of code, we should not regard them as "lines produced" but as "lines spent": the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger" ~ Dijkstra

[Tony Morris]

Yeah it is not using reflection, but does modify final fields (System.out/in/err). Just another example of the fun that can be had.

[martin odersky]

I just sent a pull request that avoids FINAL on all Scala mutable
fields that are not lazy vals. This should solve the problem at hand.

There seems to be a mismatch between spec and compiler as to what
final means on mutable variables. The compiler does not let you
override a mutable variable, but the spec does not have that
restriction. So we need to clear that up at some point.

But my fix works independently of the resolution of this problem.

Cheers

-- Martin

--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

[Som Snytt]

Historical note on the surprise factor:
"Java 5 - "final" is not final anymore"
http://www.javaspecialists.eu/archive/Issue096.html