Scala 2.9.0.RC1 erasing ParameterizedType details
Wille Faler
Paul Phillips
> parameterized.toString used to result in:
> scala.collection.mutable.LinkedList<java.lang.Integer>
>
> With Scala 2.9.0.RC1:
> scala.collection.mutable.LinkedList<java.lang.Object>
Unfortunately this is a feature.
> Is this "erased" detail for ParameterizedType's using reflection an
> intended thing in Scala 2.9, and if so is there a workaround?
It is an intended thing. If you parameterize it on java.lang.Integer
instead of Int, you'll see java.lang.Integer, which may or may not count
as a workaround.
You don't want to know how much I've suffered for signatures, and for
now there was no way forward except to lose some precision as viewed via
java reflection.
Wille Faler
is there no other workaround? (other ways than parameterizedtype are acceptable).
If not, I would imagine this really pulls the rug from under a lot of key features in a lot of frameworks that try to map some data into arbitrary objects.
Paul Phillips
Kevin Wright
On 4/7/11 4:41 PM, Wille Faler wrote: > is there no other workaround? (other ways than parameterizedtype are acceptable). The information is there, you just can't get it through java reflection. If we had a decent reflection library I'd just point you at a similar function in that. Since we don't yet, I can point you at the raw data. // a program import scala.collection.mutable.LinkedList class A { var x1 = new LinkedList[Int] var x2 = new LinkedList[java.lang.Integer] // javap Compiled from "bip.scala" public class A extends java.lang.Object implements scala.ScalaObject [...] const #35 = Asciz Lscala/reflect/ScalaSignature;; const #36 = Asciz bytes; const #37 = Asciz y2A! \t\t 9usz !Q!! PE*,7\r )1oY1mC& G.Y(cU =S:LGO #I5\taD ;bE2,'BA 7fGRLwN\ T5oW $G*[:u!\tyQ% '!\t 0M0%KF$\"AK \t\rA \tA('F 5!\ri\"% YJ!a %sG/Z4fe\"9 +w!9a &A'A See, it's right there! Here's our command line signature unpickler making it somewhat less opaque. You can see the Int: 37,201: EXTref 2: 38(Int) 16 % /scala/trunk/tools/showPickled -cp . A Version 5.0 0,3: CLASSsym 4: 1(A) 2 0[] 5 1,9: TYPEname 1: A 2,12: EXTMODCLASSref 1: 3(<empty>) 3,15: TERMname 7: <empty> 4,24: NONEsym 0: 5,26: CLASSINFOtpe 3: 0 6 14 6,31: TYPEREFtpe 2: 7 12 7,35: THIStpe 1: 8 8,38: EXTMODCLASSref 2: 9(lang) 10 9,42: TERMname 4: lang 10,48: EXTMODCLASSref 1: 11(java) 11,51: TERMname 4: java 12,57: EXTref 2: 13(Object) 8 13,61: TYPEname 6: Object 14,69: TYPEREFtpe 2: 15 18 15,73: THIStpe 1: 16 16,76: EXTMODCLASSref 1: 17(scala) 17,79: TERMname 5: scala 18,86: EXTref 2: 19(ScalaObject) 16 19,90: TYPEname 11: ScalaObject 20,103: VALsym 5: 21(<init>) 0 200[<method>] 22 21,110: TERMname 6: <init> 22,118: METHODtpe 1: 23 23,121: TYPEREFtpe 2: 24 0 24,125: THIStpe 1: 2 25,128: VALsym 7: 26(x1) 0 8000200[<method> <accessor>] 27 26,137: TERMname 2: x1 27,141: POLYtpe 1: 28 28,144: TYPEREFtpe 3: 29 34 36 29,149: THIStpe 1: 30 30,152: EXTMODCLASSref 2: 31(mutable) 32 31,156: TERMname 7: mutable 32,165: EXTMODCLASSref 2: 33(collection) 16 33,169: TERMname 10: collection 34,181: EXTref 2: 35(LinkedList) 30 35,185: TYPEname 10: LinkedList 36,197: TYPEREFtpe 2: 15 37 37,201: EXTref 2: 38(Int) 16 38,205: TYPEname 3: Int 39,210: VALsym 7: 40(x1_$eq) 0 8000200[<method> <accessor>] 41 40,219: TERMname 6: x1_$eq 41,227: METHODtpe 2: 42 45 42,231: TYPEREFtpe 2: 15 43 43,235: EXTref 2: 44(Unit) 16 44,239: TYPEname 4: Unit 45,245: VALsym 7: 46(x$1) 39 202000[<param> <synthetic>] 28 46,254: TERMname 3: x$1 47,259: VALsym 6: 48(x1 ) 0 81004[private <mutable> <local>] 28 48,267: TERMname 3: x1 49,272: VALsym 7: 50(x2) 0 8000200[<method> <accessor>] 51 50,281: TERMname 2: x2 51,285: POLYtpe 1: 52 52,288: TYPEREFtpe 3: 29 34 53 53,293: TYPEREFtpe 2: 7 54 54,297: EXTref 2: 55(Integer) 8 55,301: TYPEname 7: Integer 56,310: VALsym 7: 57(x2_$eq) 0 8000200[<method> <accessor>] 58 57,319: TERMname 6: x2_$eq 58,327: METHODtpe 2: 42 59 59,331: VALsym 7: 46(x$1) 56 202000[<param> <synthetic>] 52 60,340: VALsym 6: 61(x2 ) 0 81004[private <mutable> <local>] 52 61,348: TERMname 3: x2
--
Kevin Wright
gtalk / msn : kev.lee...@gmail.com
Wille Faler
Wille Faler
Paul Phillips
> This one has me completely stumped right now - is there ANYTHING
> short of inspecting raw bytecode that will give away the generic
> type-parameter where the type is a Java primitive?
I'm not sure what counts as "inspecting raw bytecode". The mechanisms
in java.lang.reflect.Method aren't anything more than a wrapper around
raw bytecode. Here's another. (Example only, not proposing you start
embedding scalap.)
% echo "class B { var f = List(10) }" > b.scala ; \
scalac b.scala ; scalap B | grep 'var f'
var f : scala.collection.immutable.List[scala.Int] = { /* compiled code */ }
> In my case, setting values reflectively on an object and transforming
> to the correct type from another format doesn't work with
> "speculative" setting of primitives if I see a java.lang.Object type
> as I would have thought. On first access of the field from a
> type-safe context during runtime, it will throw a ClassCastException
> (which in theory can be long after the field has been set).
I need more elaboration to understand the problem. You are reflectively
manipulating a field, assuming it not to be (say) an Int in the mind of
scala based on the reflective signature saying nothing more specific
than "Object" and then getting a CCE later. Is that right? Have you
been getting away with that until now? Is this something people commonly do?
> I can see a hole raft of frameworks struggling with this one (JSON
> and web frameworks), and also especially those still relying on old
> Java libraries which will be broken.
In case it's not clear, the present changes were made trying to unbreak
things. It's possible that the present approach can be relaxed: the
exact boundaries of the signature problem are quite elusive, and because
I personally don't use the tools where things break, I do not have my
usual confidence levels in the solution. But the ball is in your court
to assemble the evidence, because the issue has already burned way too
much time and there's no way to get across how much stuff starts coming
my way at this point. Search trac for "signature" to get a sense of what
you are up against.
Wille Faler
Maxime Lévesque
In Squeryl, the library user has to supply a no arg constructor when there are Option[] fields in a "plain old scala object".
Some()'s are stuffed with the appropriate values, so that types can be deduced :
class Student(val name: String, val lastName: String, val age: Option[Int], val gender: Int, val addressId: Option[Int]) {
def this() = this("","",Some(0),0, Some(0), Some(false))
}
Recently someone contributed a patch to "guess" the type inside the Option, so the user only has to declare :
class Student(val name: String, val lastName: String, val age: Option[Int], val gender: Int, val addressId: Option[Int])
This uses the reflection trick that Wille is talking about.
The trick is no longer working.
This new feature of Squeryl is not yet documented, so no one will complain about the loss of this possibility in 2.9.0-RC1,
but it's s still unfortunate that people will be declaring things like :
def this() = this("","",Some(0),0, Some(0), Some(false))
or even more verbose :
class Student(val name: String, val lastName: String, val age: @Column(optionType=classOf[Int]) Option[Int], val gender: Int, @Column(optionType=classOf[Int]) val addressId: Option[Int])
for the forseable future.
ML
Paul Phillips
> Recently someone contributed a patch to "guess" the type inside the
> Option, so the user only has to declare :
>
> class Student(val name: String, val lastName: String, val age:
> Option[Int], val gender: Int, val addressId: Option[Int])
Oh yeah, I just remembered that's how this works.
https://github.com/paulp/optional
Anyway, you guys don't need to convince me that this sucks. What you
need to do is solve the problem better. I'm not a magician, and if it's
a choice between eclipse working and reflection conveniences working,
eclipse is going to win. I simply cannot table every other facet of the
distribution to dive into this issue again. You guys are
enthusiastically encouraged to research the situation and assemble a
convincing argument that the boxed types can be used without breaking
things.
Wille Faler
Dave Whittaker
Paul Phillips
> It seems to me that in most cases when the Java compiler needs to
> indicate a primitive type during reflection it uses a special Class
> rather than the boxed type. For instance, if a method returns int you
> wouldn't find Class[java.lang.Integer] you would see it as
> java.lang.Integer.Type which the javadoc explains as "The Class instance
> representing the primitive type int". Would this be a possible
> alternative?
The class objects have no bearing, except inasmuch as java reflection
will return a class object based on what is in the signature. The issue
is what is the textual content of the generic signatures, which look
something like this:
Lscala/collection/immutable/List<TA;>;
They can look like this:
Lscala/collection/immutable/List<Ljava/lang/Integer;>;
But they cannot look like this, both because the generic signature
specification prohibits it and it breaks java software which doesn't
expect it. ("I" is the signature of primitive Int, which corresponds to
java.lang.Integer.TYPE.)
Lscala/collection/immutable/List<I>;
Here's an email I sent somewhere a few weeks ago. It implies a part two
is coming but I haven't finished it.
Scala's Signature Dish, Chapter I
1) This ends with a puzzle. You like puzzles, don't you?
2) A couple days ago retronym took me to the restaurant "Scala" where I
actually ordered their signature dish. No enlightenments yet but I'm
still hopeful.
I am going to attempt to distill the signature problem and why it isn't
fixed yet, but in case I can draw in more interested parties by
uncharacteristically documenting a little bit, here is a quick overview
of signatures. There are two bytecode constructs which communicate the
key characteristics of the blobs of data you find in there. The older,
pre-generics one is called a "descriptor" and the one tacked on later a
"signature", although the latter term is often misused for both. Fields
and methods get both varieties; class definitions only get signatures.
Descriptors have no knowledge of type parameters, but they are
approximately equivalent to a signature after erasing the types.
Why do we care about signatures? Here is how things look to java, and
everything else in the world beyond the scala compiler (which reads its
own, much richer signature format when parsing scala bytecode, but
relies on java generic signatures for reading java since that's all
there is.)
// ...with signatures
class Cell[T](val x: T) // put a T in, get a T out
// ...without signatures
class Cell(val x: Object) // put whatever you want in, take an Object out
To see some sweet signature action, download a 2.9 nightly and:
// define this in the repl
class Bippy[A <: Function0[_ <: Comparable[_]]] extends Mutable {
var x: List[A] = _
def f[B, C <: A](x: Int, y: B) = ((x, y))
}
scala> :javap -verbose Bippy
// trimmed to relevant constant pool entries
const #3 = Asciz x;
const #4 = Asciz Lscala/collection/immutable/List;;
const #5 = Asciz Lscala/collection/immutable/List<TA;>;;
const #18 = Asciz f;
const #19 = Asciz (ILjava/lang/Object;)Lscala/Tuple2;;
const #38 = Asciz
<B:Ljava/lang/Object;C:TA;>(ITB;)Lscala/Tuple2<Ljava/lang/Object;TB;>;;
const #45 = Asciz
<A::Lscala/Function0<+Ljava/lang/Comparable<*>;>;>Ljava/lang/Object;Lscala/Mutable;Lscala/ScalaObject;;
[4] field descriptor // notice the raw List type
[5] field signature // "TA;" means "type parameter A"
[19] method descriptor // "I" is primitive Int; "L...;" is a class.
[38] method signature // "C:TA" means tparam C is bounded by A.
[45] class signature // "+" is covariance; "*" a wildcard.
Notice there is a lot more information in the generic signatures. That
is the point. There are a number of restrictions on exactly how you
create a signature. The most important for our purposes is that you
cannot parameterize a type on primitives. The one-character primitive
type tags cannot appear as type arguments or type bounds unless you
enjoy the music of breakage.
Before we come to the next chapter wherein we examine the problem,
consider the following scala code. The questions for you are:
1) What signatures should go into classes A and B?
1b) "Eclipse crashes." Pay $50 and go back to square 1.
2) Given those signatures, does the java code compile?
2b) "You broke reflection." Lose a turn.
3) Try again. Would you like to buy some bridge methods?
3b) The troll grabbed you! Pay $100 or be eaten and lose infinity turns.
4) Do you like apples? How do you like these apples?
// it's some scala code
trait A[T] {
def f(): T = Predef.error("")
def g(x: Int) = ()
}
class B extends A[Int] { }
// and some java code
public class J {
public void sigs() {
B b = new B();
b.g(b.f());
}
}
David Whittaker
Ismael Juma
> Thanks for forwarding that on, I think I understand the problem now. Could
> you possibly clarify where the signature that does contain the primitive
> type is stored within the class? I'm thinking about using a library like
> ASM to extract the un-cripled signature and any hint you can give me about
> where to find it would be helpful (I tried to absorb the "unpickled" output
> but I'm afraid I don't speak JVM byte code very well). When I've got it
> worked out I'll try to post an example to Github so it can be used by others
> with the same problem.
A few people have already done this. An example:
https://github.com/codahale/jerkson/commit/7069e58283d17b96d76f632a38e0d6e51c60ced4
Best,
Ismael