Should ClassTag.unapply know about boxing?

[som-snytt]

O__ asked about "ClassTag based pattern matching fails for [boxed] primitives".

http://stackoverflow.com/questions/16825927/classtag-based-pattern-matching-fails-for-primitives

Should this just work?

scala> import scala.PartialFunction._
import scala.PartialFunction._

scala> def f[A: ClassTag](a: Any) = cond(a) {
     | case _: A => true
     | }
f: [A](a: Any)(implicit evidence$1: scala.reflect.ClassTag[A])Boolean

scala> f[Double](2.3)
res7: Boolean = false

scala> f[java.lang.Double](2.3)
res8: Boolean = true

Probably there is a counter-example where an extra unbox/box is incurred. Probably Rex Kerr has an example.  I just saw a :javap with an extra unbox/box, but I forgot the example; and I saw the recent issue where the footnote was that -optimise makes it go away.

My broader question is, is boxing supposed to be invisible, modulo performance?

[Paul Phillips]

On Mon, Jun 3, 2013 at 5:38 PM, som-snytt <som....@gmail.com> wrote:

My broader question is, is boxing supposed to be invisible, modulo performance?

When you refer to scala.Double, boxing is supposed to be invisible, plus or minus.

When you refer to java.lang.Double, boxing should be visible, because it is a box.

The details of boxing are numerous and I haven't seen any sign much thought was given to them in the reflection library, so I don't know what the given example is expected to do. The classtag materialization and unapply method take place during typer or shortly thereafter, but the primitive argument isn't boxed until much later, in erasure. I realize that doesn't offer any useful information to people without an interest in the implementation, but that's all I have.

[Johannes Rudolph]

On Thu, Jun 6, 2013 at 12:32 PM, Paul Phillips <pa...@improving.org> wrote:

The details of boxing are numerous and I haven't seen any sign much thought was given to them in the reflection library, so I don't know what the given example is expected to do. The classtag materialization and unapply method take place during typer or shortly thereafter, but the primitive argument isn't boxed until much later, in erasure. I realize that doesn't offer any useful information to people without an interest in the implementation, but that's all I have.

Isn't this a pure runtime issue? IMO the implementation of `ClassTag.unapply(x: Any)` should know that boxing occurs if you have managed to pass an argument of type `Any` and handle boxed values accordingly.

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Paul Phillips]

On Fri, Jun 7, 2013 at 2:04 AM, Johannes Rudolph <johannes...@googlemail.com> wrote:

Isn't this a pure runtime issue? IMO the implementation of `ClassTag.unapply(x: Any)` should know that boxing occurs if you have managed to pass an argument of type `Any` and handle boxed values accordingly.

ClassTags exist primarily to provide accurate answers to questions like "what kind of array do I need to allocate." They cannot simultaneously provide accurate answers to low-level questions and useful answers to high-level questions at times when the low-level distinction becomes a burden. That's why I wanted to separate ClassTags and ArrayTags.

If ClassTag.unapply(Any) performs runtime discrimination, it becomes the only bit of ClassTag infrastructure which does so. The primitive type ClassTag unapplies are handled via method overloading, i.e. it's completely based on the static type.

scala> 5 match { case reflect.ClassTag.Int(x) => x }

res2: Int = 5

scala> (5: Any) match { case reflect.ClassTag.Int(x) => x }

scala.MatchError: 5 (of class java.lang.Integer)

at .<init>(<console>:8)

at .<clinit>(<console>)

Since this is rarely the behavior you want if you're working with values, the ClassTag unapply is unsuitable for most uses. But because of the way ClassTags are applied in the implementation, they can't be "smart" about this.

[Johannes Rudolph]

On Friday, June 7, 2013, Paul Phillips wrote:

On Fri, Jun 7, 2013 at 2:04 AM, Johannes Rudolph <johannes...@googlemail.com> wrote:

Isn't this a pure runtime issue? IMO the implementation of `ClassTag.unapply(x: Any)` should know that boxing occurs if you have managed to pass an argument of type `Any` and handle boxed values accordingly.

ClassTags exist primarily to provide accurate answers to questions like "what kind of array do I need to allocate." They cannot simultaneously provide accurate answers to low-level questions and useful answers to high-level questions at times when the low-level distinction becomes a burden. That's why I wanted to separate ClassTags and ArrayTags.

Funny, this is the mail I started to write before you answered:

<snip>

I'd say it's a condition that could be prevented statically. It could be that all the parts leading to it are working as specified (if they are), so not sure if you could really call it a bug.

One issue is that it's questionable if there should be `ClassTag` for primitive types. Technically, there are java.lang.Classes for the primitive value types and actually those classes are also needed for array creation (what once was ArrayTag). However, for other purposes those primitive type classes are much less useful than classes for reference types. E.g. The existence of `classOf[Int].isInstance` is basically a lie because it never returns true (because you can't pass an unboxed value to it). Similar for other methods of java.lang.Class.

</snip>

I then remembered the once existing ArrayTag and went to reread the mails regarding its removal but I can't say what the final reason was (it seemed to be something between reducing the overall API surface and not finding a proper name).

If ClassTag.unapply(Any) performs runtime discrimination, it becomes the only bit of ClassTag infrastructure which does so.

ClassTag.unapply calls `x.getClass` on its argument, how is that not runtime discrimination? Isn't that the only purpose of ClassTag.unapply?

 

The primitive type ClassTag unapplies are handled via method overloading, i.e. it's completely based on the static type.

scala> 5 match { case reflect.ClassTag.Int(x) => x }

res2: Int = 5

scala> (5: Any) match { case reflect.ClassTag.Int(x) => x }

scala.MatchError: 5 (of class java.lang.Integer)

at .<init>(<console>:8)

at .<clinit>(<console>)

Since this is rarely the behavior you want if you're working with values, the ClassTag unapply is unsuitable for most uses. But because of the way ClassTags are applied in the implementation, they can't be "smart" about this.

Yeah, I've seen the implementation. Isn't the snippet you've posted seems like the ultimate evidence that unapply(Any) is wrong? Are you saying there is something which relies on this behavior? I fail to imagine how you can build anything reasonable upon this behavior.

I'm not interested in the overloaded implementations of ClassTag.unapply with the primitive parameters which seem to be there to suggest that unapply is working for primitives as well. For ClassTag.unapply(Any) the situation seems pretty clear to me: there's no way you could pass an unboxed primitive into it so IMO it makes no sense to even suggest it would handle it. One argument could be that the boxing behavior is not specific to Scala but more a specific of its runtime platform. I would say: especially then should the runtime library be aware of this fact and provide reasonable behavior.  

Johannes

[Som Snytt]

The SO question is really benign in its naivete.

That is: I made a List[Any](..., 2.3, ...) and now I want to use ClassTag[Double] to find the Double.  You're asking me to use ClassTag[java.lang.Double]?  Because I have a scorecard at my side on which I pencil in whether my values are boxed.  (I guess the performance guys do that.)

Paul is right about the dual-usage issue. My first thought on the SO question was that ClassTags are for making arrays, maybe I can make an empty array and get its component type, but you still wind up just with Array[Int] or Array[Integer], of course, with no way to test that one is the box of the other.

That was the surprise for me, that there is no API (even unsupported) for testing that one type is the boxed form of another.  Unless I missed it?  I think I started to drill into Definitions, but my time-box ran out.  TimeBox is the boxed form of Time.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Paolo G. Giarrusso]

We don't need to change the behavior of ClassTag's existing API to improve the situation. I'm sure at least some of the ideas below are implementable, and maybe somebody else could pick this up (sorry but I don't have time). Further below, answers to some points in the thread.

1. Could one add methods to convert a type to/from its boxed version? It should do nothing on most types, but convert Class<Integer> to/from Class<int> (similarly for other primitive types), and then the same for the corresponding ClassTags. That'd be better than the current situation.

I had to write these methods for Class (and they could easily be extended to ClassTag):

https://github.com/ps-mr/LinqOnSteroids/blob/master/src/main/scala/ivm/expressiontree/Util.scala#L7 (I already posted this link on StackOverflow, but I thought it could be useful here)

I don't have time for a pull request right now, and somebody should figure out the correct place where to stick these methods. But maybe somebody else can? som-snytt (please tell me if I should address you otherwise), would you consider something like that?

For bonus points, one could also work on point 2 and come closer to a (full?) solution.

2.

Usually calls to ClassTag.unapply are generated by the pattern matcher, and one could generate better code. (Not in the StackOverflow question, but I'd write a typed match there). Moreover, one could probably split unapply in different methods for different uses.

Moreover, maybe the pattern matcher could even figure out statically that it needs to call this method. Or it could just always convert box the value to be tested, "box" the ClassTag used for testing, and let the optimizer figure out something faster, when it is clear what is boxed and what isn't? This might even be faster than handling boxing within ClassTag.unapply. And maybe maybe maybe, this optimization could be done in an extra-clever macro by deciding boxedness statically (I'm sure the compiler can do that somewhere).

And if you really want ClassTag.unapply, rename the current one to ClassTag.unapplyLit and call this macro ClassTag.unapply.

How does one decide boxedness statically? My guesses:

Idea 1: a term of static type type Int, Long, etc. should not be boxed, but a term of type Any or AnyVal should always be, and for a term of type X <: AnyRef none of this matters.

Idea 2: in fact, whenever you call `ClassTag.unapply(Any)`, its argument is boxed. If you instead call any specialized overload `ClassTag.unapply(T)` where T is primitive, its argument is unboxed — yes, there are such overloads. Hence, one can statically prove that if `v: AnyVal` and a classtag `c` is for an unboxed type, then `c.unApply(Any)` applied to `v` returns `false`. In standard Scala, that made-up syntax can be written `((c: ClassTag[_], v: Any) => c unapply v) c v == false`.

I did not look at the actual semantics, I just guesstimated, and I feel there's not so much freedom for having a "reasonable" translation.

On Sunday, June 9, 2013 5:48:27 PM UTC+2, som-snytt wrote:

The SO question is really benign in its naivete.

That is: I made a List[Any](..., 2.3, ...) and now I want to use ClassTag[Double] to find the Double.  You're asking me to use ClassTag[java.lang.Double]?  Because I have a scorecard at my side on which I pencil in whether my values are boxed.  (I guess the performance guys do that.)

For a List[Any], the values *have to* be boxed. Nothing to remember.

One issue is that it's questionable if there should be `ClassTag` for primitive types. Technically, there are java.lang.Classes for the primitive value types and actually those classes are also needed for array creation (what once was ArrayTag). However, for other purposes those primitive type classes are much less useful than classes for reference types. E.g. The existence of `classOf[Int].isInstance` is basically a lie because it never returns true (because you can't pass an unboxed value to it). Similar for other methods of java.lang.Class.

Indeed, that's why has to manually map Classes to the boxed form before calling isInstance.

I'm not interested in the overloaded implementations of ClassTag.unapply with the primitive parameters which seem to be there to suggest that unapply is working for primitives as well. For ClassTag.unapply(Any) the situation seems pretty clear to me: there's no way you could pass an unboxed primitive into it so IMO it makes no sense to even suggest it would handle it. One argument could be that the boxing behavior is not specific to Scala but more a specific of its runtime platform. I would say: especially then should the runtime library be aware of this fact and provide reasonable behavior.

I agree they're pretty stupid, just a bit less so. If you manage to call those overloaded implementations, they "work" — but if you manage to call them, at least the compiler knows statically the type of their argument, so at least a non-novice/experienced Scala user knows as well.

Hence, ClassTag.unapply(Int) on ClassTag `c` is equivalent to `(x: Int) => if c == classOf[Int] Some(x) else None`. So in essence, you're testing the classTag, nothing else, and frankly most of the time I'd rather have a method `isClassTagInt`, or arguably nothing at all since I can write `c == classOf[Int]` myself. Demonstration:

scala> import scala.reflect._

import scala.reflect._

scala> Seq(classTag[Int], classTag[Double]) filter (x => (x unapply 1).isDefined)

res7: Seq[scala.reflect.ClassTag[_ >: Double with Int <: AnyVal]] = List(Int)

What's `Int` there? `classTag[Int]`? Indeed:

scala> classTag[Int]

res8: scala.reflect.ClassTag[Int] = Int

Pretty useful, eh?

[Paul Phillips]

On Mon, Jun 10, 2013 at 2:28 PM, Paolo G. Giarrusso <p.gia...@gmail.com> wrote:

How does one decide boxedness statically?

Before there were classtags, before there were manifests, before there was much, I did it this way (this obviously being somewhat simplified to show the mechanism.) Still works. "Implicits are the window into the compiler's soul."

scala> implicit def imp1[T <: AnyVal](x: T) = new { def whatAmI = "something primitive" } ; implicit def imp2[T <: AnyRef](x: T) = new { def whatAmI = "something boxed" }

imp1: [T <: AnyVal](x: T)AnyRef{def whatAmI: String}

imp2: [T <: AnyRef](x: T)AnyRef{def whatAmI: String}

scala> 5.whatAmI

res0: String = something primitive

scala> (5: Integer).whatAmI

res1: String = something boxed

[Som Snytt]

+1 fun factor.

> "Implicits are the window into the compiler's soul."

I wish I was at Scala Days so I could visit the t-shirt booth.

[Paolo G. Giarrusso]

That's certainly cool, but I'm sorry to disagree, I don't think with current Scala it does what we want here. Look:

implicit def imp1[T <: AnyVal](x: T) = new { def whatAmI = s"something primitive: look at >>>>>${x.getClass}<<<<<<" }

implicit def imp2[T <: AnyRef](x: T) = new { def whatAmI = "something boxed" }

scala> 5.whatAmI

res11: String = something primitive: look at >>>>>class java.lang.Integer<<<<<<

Notice that's java.lang.Integer :-). And I think that's the kind of boxing we're after, when a term has static type Int and dynamic type java.lang.Integer. We (or at least I) want to distinguish that from terms with static type Int and dynamic type Int.

My guess is that within imp1, you'll always be in the first situation, since imp1 is polymorphic. If you manually specialize that to primitive types (writing the ~8 overloads you need), you'll always be in the second situation. If you auto-specialize it, I'm less sure but you'll always in the second situation in the types you specialized for.

And I think that's relevant to the original example:

val tag = classOf[Double]

List[Any](..., 2.3, ...).collect {

  case tag(c) => c
}

or its variant, which doesn't work when T is a primitive:

def f[T](implicit val tag: ClassTag[T]) = {

  List[Any](..., 2.3, ...).collect {

    case c: T => c
  }

}

So, within a pattern match, how does one know whether a scrutinee of primitive type (Int) is autoboxed or unboxed? If a macro can compute that, ClassTag.unapply (or some additional method, used by the pattern matcher) could be made into a macro getting that information with optimal performance. Otherwise, you are either stuck with some slow implementation or you need to build this logic in the compiler.

[Paul Phillips]

On Mon, Jun 10, 2013 at 6:21 PM, Paolo G. Giarrusso <p.gia...@gmail.com> wrote:

res11: String = something primitive: look at >>>>>class java.lang.Integer<<<<<<

Well, that's why I didn't call getClass. The information being sought is encoded in which method was called. If the T <: AnyVal method is called, then it was primitive. The actual thing you're holding will of course be boxed, but that's okay, because you know what the unboxed form of anything boxed is, and you know the static type is unboxed.

If you really didn't believe it, I can flesh out that method until you get the thingy you want. The key is that discriminating via variant implicit conversions collects the information at the call site.

[Paolo G. Giarrusso]

On Tuesday, June 11, 2013 12:33:32 AM UTC+2, Paul Phillips wrote:

On Mon, Jun 10, 2013 at 6:21 PM, Paolo G. Giarrusso <p.gia...@gmail.com> wrote:

res11: String = something primitive: look at >>>>>class java.lang.Integer<<<<<<

Well, that's why I didn't call getClass.

Yeah, calling it was a bad way of making my point.

 

The information being sought is encoded in which method was called. If the T <: AnyVal method is called, then it was primitive. The actual thing you're holding will of course be boxed, but that's okay, because you know what the unboxed form of anything boxed is, and you know the static type is unboxed.

Ah, so you propose to figure out whether something is AnyVal and then just unbox it, even though at the call site it maybe was already unboxed? OK, that would work, but that's not how I hoped to do it.

If you really didn't believe it, I can flesh out that method until you get the thingy you want. The key is that discriminating via variant implicit conversions collects the information at the call site.

I suspect I was mislead by your `(5: Integer).whatAmI`  example, and by some details. I'm also not sure whether I guesstimated the ABI correctly.

If the T <: AnyVal method is called, maybe the call-site type was AnyVal and at the caller-site was still boxed. Right?

I think I'm after something more like the following (shown only for Int):

implicit def imp1[T](x: T) = new { def whatAmI = "something boxed" }

implicit def impInt(x: Int) = new { def whatAmI = "something primitive" }

// Other N variants of impInt for all base types

// It would maybe be cool if I could instead write:

//  implicit def imp2[@specialized T](x: T) = new { def whatAmI = "something primitive" }

// and get what I want, but of course specialization tries to not alter the semantics, exactly the opposite of what I'd want here. So I guess I'd need some advanced macros for that.

// Three example call-sites, to call on argument 5.

def f1[T](t: T) = t.whatAmI //t is boxed here

def f2[T <: AnyVal](t: T) = t.whatAmI // also boxed

def f3(t: Int) = t.whatAmI //here it's primitive

The output matches the comments, but I am not yet sure whether erasure actually works like that. That's just my guess.

Now, what I was going after is something horribly complicated like:

  def unboxClassTag(x: ClassTag[_]): ClassTag[_] = //omitted

  def boxClassTag(x: ClassTag[_]): ClassTag[_] = //omitted

  implicit def imp1[T](x: T): Matcher[T] = new Matcher[T] {

      def matchMe[U](ct: ClassTag[U]) = {

        println("imp1")

        (boxClassTag(ct) unapply x: Option[_]).asInstanceOf[Option[T]]

      }

    }

  implicit def impInt(x: Int): Matcher[Int] = new Matcher[Int] {

      def matchMe[U](ct: ClassTag[U]) = {

        println("impInt")

        (unboxClassTag(ct) unapply x: Option[_]).asInstanceOf[Option[Int]]

      }

    }

//All matches succeed

  println(1 matchMe classTag[Int])

  println(1 matchMe classTag[Integer])

  def foo[T](x: T) = {

    println(x matchMe classTag[Int])

    println(x matchMe classTag[Integer])

  }

  foo(1)

Now the idea is that matchMe could be a macro method, so that you decide statically how to convert the macro tag. If the matching is done in a loop and this conversion is lifted outside the loop, that's might be actually faster, isn't it?

But for the non-loop case, I fear I'm optimizing the wrong direction — isn't it faster to simply unbox or box the value, rather than comparing class tags? Well, I don't know, but when the class tag is either unboxed or boxed and you don't know, you need to normalize the class tag, so it doesn't get much better. Or you just call `ct.unapply(box(t)) orElse ct.unapply(unbox(t))` (or the result of aggressive inlining in that fragment) and maybe that's even faster.

[Paul Phillips]

I shouldn't take too much time for boxing while at scala days, but let me disabuse you of one notion which may change your thinking about both boxing and specialization:

scala> object A { def f1[T](x: T) = x.getClass ; def f2[@specialized T](x: T) = x.getClass }

defined object A

scala> A.f1(5)

res0: Class[_ <: Int] = class java.lang.Integer

scala> A.f2(5)

res1: Class[_ <: Int] = int

A related way I've seen this come up before: here's what ArraySeq does right now:

scala> new scala.collection.mutable.ArraySeq[Float](5) mkString

res2: String = nullnullnullnullnull

scala> new scala.collection.mutable.ArraySeq[Float](5) map ("" + _) mkString

res3: String = 0.00.00.00.00.0

However if you specialize ArraySeq, both expressions are the same - no nulls are revealed.

[Scott Carey]

On Monday, June 10, 2013 11:28:07 AM UTC-7, Paolo G. Giarrusso wrote:

On Sunday, June 9, 2013 5:48:27 PM UTC+2, som-snytt wrote:

The SO question is really benign in its naivete.

That is: I made a List[Any](..., 2.3, ...) and now I want to use ClassTag[Double] to find the Double.  You're asking me to use ClassTag[java.lang.Double]?  Because I have a scorecard at my side on which I pencil in whether my values are boxed.  (I guess the performance guys do that.)

For a List[Any], the values *have to* be boxed. Nothing to remember.

Perhaps not -- what if Cons was abstract, and had specialized sub types for AnyRef and each value type?    How would such an implementation interact with ClassTag, specialization, and 'extends AnyVal'?

[Paolo G. Giarrusso]

On Wednesday, June 12, 2013 7:42:45 PM UTC+2, Scott Carey wrote:

On Monday, June 10, 2013 11:28:07 AM UTC-7, Paolo G. Giarrusso wrote:

On Sunday, June 9, 2013 5:48:27 PM UTC+2, som-snytt wrote:

The SO question is really benign in its naivete.

That is: I made a List[Any](..., 2.3, ...) and now I want to use ClassTag[Double] to find the Double.  You're asking me to use ClassTag[java.lang.Double]?  Because I have a scorecard at my side on which I pencil in whether my values are boxed.  (I guess the performance guys do that.)

For a List[Any], the values *have to* be boxed. Nothing to remember.

Perhaps not -- what if Cons was abstract, and had specialized sub types for AnyRef and each value type?

 

Interesting point.

 

I think you can just specialize Cons (and List) and get some appropriate encoding (where maybe Cons won't be abstract).

A specialized collection would indeed contain unboxed values. Hence, functions/data structures etc., if polymorphic *and not specialized*, receive/contain boxed data.

However, that works if you have a List[Int], but a List[Any] is more complicated. In particular, I'd try with List("s", 1) and List(1, "s"): the second case seems hopeless, since a Cons[Int] can't contain a Cons[Any]. In the first case, I'm curious whether specialization would manage to build a Cons[Any]("", Cons[Int](1, Nil)), which is at least type-correct (assuming variant, thus immutable, lists).

However, I fear that doesn't help.

 

   How would such an implementation interact with ClassTag, specialization, and 'extends AnyVal'?

 

I can't answer on 'extends AnyVal', but I'll suggest a partial answer for the rest.

It all depends on as far as specialization goes, with the twist that ClassTag.unapply is not specialized, it simply has overloads for all primitives.

Take this example:

scala> def f[A: ClassTag](a: Any) = cond(a) {
     | case _: A => true
     | }

Now, which overload of ClassTag.unapply should be called here? It should be the generic overload ClassTag.unapply(Any). Suppose now you specialize this function: specialization won't change the results of overloading resolution (last I checked, but please do try), so the specialized functions will still call `ClassTag.unapply(Any)`; since this overload can't accept unboxed values, the caller will need to box its argument. Also, if specialization affected overloading resolution, it'd arguably be a bug, since the compiler can't assume overloads are anyhow semantically related. If the variants are instead the results of specialization (which won't be overloads), the compiler will select the right variant.

So, if I remember wrongly and specialization affects overloading resolution, I think you'd get unpleasant results on something along the lines of:

def f(x: Any) = println("I'm nice!")

def f(x: Int) = { println("I'm the opposite of nice!"); System.tryDestroyingTheComputer(); }

def caller[@specialized(Int) T](t: T) {

  //Print a nice message:

  f(t)

}

and probably you could use (a variant) of that for a bug report.

[Som Snytt]

Hey, I just stubbed my toe on the same problem!

Even after re-reading this whole thread carefully, it is an irksome sharp corner case.  (pun on case)

scala> class C[A: ClassTag] {
     | def f(a: A) = a match {
     | case s: String => "str"
     | case x: A      => "A"
     | }
     | }
defined class C

scala> new C[Int].f(3)
scala.MatchError: 3 (of class java.lang.Integer)

It was in the context of future onSuccess { ... } and it was after midnight and suddenly a promise went silently unfulfilled (because I had blearily added the ClassTag bit).

I wound up sleeping on it, like a pea.

So my suggestion is that -Xlint needs a special mode, -Xlint:12, to catch all those things, never good, which happen after midnight.

On Mon, Jun 3, 2013 at 5:38 PM, som-snytt <som....@gmail.com> wrote: