SIP-15 Value class encoding breaks reflection, structural types

Showing 21-83 of 83 messages
SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/9/12 1:42 PM
The current compilation scheme for methods that accept Value Class
parameters doesn't play nicely with reflection, structural types, or
callers from Java. SI-6336 details the issues [1]. In the comments, I
propose an alternative.

I know it's very late in the release cycle, but we ought to discuss:

- Is the alternative proposal feasible?
- For how long can we live with the current limitations?
  - Can we at least turn some of those limitations into compile,
rather than runtime, errors?
- What's the migration path to the alternative? I feel it would be
messy to change in a minor release.

I couldn't find any discussion of this particular detail on the list;
most of the discussions on this SIP gravitated to FlatArray, and the
overall attention on the list might have been spread too thinly across
the smorgasbord of SIPs on offer at the time (I'm looking at you, 18.)

-jason

[1] https://issues.scala-lang.org/browse/SI-6336
Re: SIP-15 Value class encoding breaks reflection, structural types Rex Kerr 9/9/12 2:11 PM
That looks like a good approach to me.  I agree that the current method is going to be painful to change if we don't do it now.  I also note that there is precedent for exactly this sort of thing with specialization.  This has the potential to create conflict--is it function f$mSc$sp$unboxed or f$unboxed$mSc$sp?

Also, do you specialize everyone or only the unboxed variants?  I would say only specialize on the unboxed ones.  If you have to box the value class, you also have to box everyone else.  Tough luck--but doubling the already absurd number of methods you can get with specialization is painful.  Likewise, if you have multiple value classes, you are either entirely boxed or entirely unboxed, to avoid multiplicative issues.  Your base unspecialized function can have both boxed and unboxed variants, though, as you don't want to unnecessarily penalize the case where you don't have primitives but the objects are already created so you aren't spending an extra object allocation every call.

Also, since specialization is only $sp, I suggest that unboxed be something similarly short that can't be confused.  Maybe $ub?  Or $vc for value class?  As it won't be read by anyone, pushing up against limits on platforms with minimal resources (e.g. Android) for no reason is probably inadvisable.

  --Rex

P.S. Guilty as charged of focusing most of my attention on 18.
Re: SIP-15 Value class encoding breaks reflection, structural types Erik Osheim 9/9/12 3:10 PM
On Sun, Sep 09, 2012 at 10:42:22PM +0200, Jason Zaugg wrote:
> - Is the alternative proposal feasible?

I don't see an obvious reason why this wouldn't work--from what I can
tell the changes to the value class spec would be pretty cosmetic. I
agree with Rex that choosing a naming scheme somewhat compatible with
specialization would make sense; $vc sounds good to me.

Obviously I can't comment on the lateness in the release cycle.

> I couldn't find any discussion of this particular detail on the list;
> most of the discussions on this SIP gravitated to FlatArray, and the
> overall attention on the list might have been spread too thinly across
> the smorgasbord of SIPs on offer at the time (I'm looking at you, 18.)

Reflection was definitely not on my radar, so I think I did probably
focus too much on other areas instead.

-- Erik
Re: SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/9/12 3:22 PM
On Mon, Sep 10, 2012 at 12:10 AM, Erik Osheim <er...@plastic-idolatry.com> wrote:
> On Sun, Sep 09, 2012 at 10:42:22PM +0200, Jason Zaugg wrote:
>> - Is the alternative proposal feasible?
>
> I don't see an obvious reason why this wouldn't work--from what I can
> tell the changes to the value class spec would be pretty cosmetic. I
> agree with Rex that choosing a naming scheme somewhat compatible with
> specialization would make sense; $vc sounds good to me.

Another alternative would be to leave the names of both boxed and
unboxed versions unmangled.

> Obviously I can't comment on the lateness in the release cycle.

I mention that possibility because it might be easier to retrofit onto
the current implementation with a minimum of effort/risk. But that's
just a hunch, and even if it's right, it's not a great way to take
these sort of decisions.

> Reflection was definitely not on my radar, so I think I did probably
> focus too much on other areas instead.

Yeah, we would really profit from checklists for both code and SIP
review. Top of the list: interaction with existing features.

-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/9/12 4:41 PM


On Sun, Sep 9, 2012 at 3:10 PM, Erik Osheim <er...@plastic-idolatry.com> wrote:
Obviously I can't comment on the lateness in the release cycle.

It's not like we'd be protecting some long standing paragon of robustness.  It's all new code.  If the initial incarnation has the "stability high ground", its altitude advantage in that domain should be measured with a micrometer.

Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/9/12 4:48 PM


On Sun, Sep 9, 2012 at 3:22 PM, Jason Zaugg <jza...@gmail.com> wrote:
Yeah, we would really profit from checklists for both code and SIP
review. Top of the list: interaction with existing features.

I'd like to see a chart like this:


But with 1000 rows and 1000 columns, which you will need.

"How lazy vals see objects"
"How lazy vals see themselves"
"How lazy vals see polymorphic methods when defined in a closure itself defined in a try/catch all nested in a trait, with the val being based via by-name argument to a specialized  overloaded method with a structurally defined parameter in its second parameter list"

And etc.

Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/10/12 12:38 AM


On Sun, Sep 9, 2012 at 10:42 PM, Jason Zaugg <jza...@gmail.com> wrote:
The current compilation scheme for methods that accept Value Class
parameters doesn't play nicely with reflection, structural types, or
callers from Java. SI-6336 details the issues [1]. In the comments, I
propose an alternative.

I know it's very late in the release cycle, but we ought to discuss:

- Is the alternative proposal feasible?

I don't know. The proposal only deals with method parameters, what about results? There are many other places where value classes come into the mix and we'd have to be sure we deal with them all. My preference in this case would be to have as simple a semantic model as possible.

The current model is: Value classes erase to the same type as their underlying type.

I think we can keep to it and address the concerns:

 - structural types: Have a restriction that method signatures in structural types may not refer to value classes. We already have a restriction that they may not refer to enclosing class parameters, so this is nothing new. At some point we might want to come back and change the structural type dispatch scheme, which could then hopefully get rid of both restrictions in one fell sweep.

- reflection: not sure yet, but I guess we can find the right rules. In any case, I vote for keeping reflection experimental for the current release so we have time to fix it.

 - Java interop: Have the simple rule that value classes are invisible for Java. Java APIs have to deal with the underlying type instead. This is good, in my opinion. The whole purpose of value classes is to give nice syntactic coating for something that does not need boxing. Having to box from Java is not an improvement, IMO.

Regarding time frame: I think the fixes for value classes can be done this week or next at the outset. Redoing the design will add at least one or two months to the release schedule. I think our time is better spent on extending value classes to multiple parameters. 

Cheers

 - Martin



 
- For how long can we live with the current limitations?
  - Can we at least turn some of those limitations into compile,
rather than runtime, errors?
- What's the migration path to the alternative? I feel it would be
messy to change in a minor release.

I couldn't find any discussion of this particular detail on the list;
most of the discussions on this SIP gravitated to FlatArray, and the
overall attention on the list might have been spread too thinly across
the smorgasbord of SIPs on offer at the time (I'm looking at you, 18.)

-jason

[1] https://issues.scala-lang.org/browse/SI-6336



--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

Re: SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/10/12 1:37 AM
On Mon, Sep 10, 2012 at 9:37 AM, martin odersky <martin....@epfl.ch> wrote:
> I don't know. The proposal only deals with method parameters, what about
> results? There are many other places where value classes come into the mix
> and we'd have to be sure we deal with them all. My preference in this case
> would be to have as simple a semantic model as possible.

The boxed bridge method would both unbox parameters and box results.

> The current model is: Value classes erase to the same type as their
> underlying type.
>
> I think we can keep to it and address the concerns:
>
>  - structural types: Have a restriction that method signatures in structural
> types may not refer to value classes. We already have a restriction that
> they may not refer to enclosing class parameters, so this is nothing new. At
> some point we might want to come back and change the structural type
> dispatch scheme, which could then hopefully get rid of both restrictions in
> one fell sweep.
>
> - reflection: not sure yet, but I guess we can find the right rules. In any
> case, I vote for keeping reflection experimental for the current release so
> we have time to fix it.

Yep, Structural Types and Scala reflection both have enough
information to do the Right Thing, eventually. But I think it's
tempting to install unboxed bridges to avoid touching either of them.

Or declare Value Classes experimental?

>  - Java interop: Have the simple rule that value classes are invisible for
> Java. Java APIs have to deal with the underlying type instead. This is good,
> in my opinion. The whole purpose of value classes is to give nice syntactic
> coating for something that does not need boxing. Having to box from Java is
> not an improvement, IMO.

They also offer type safety. One Java caller might prefer convenience
+ type safety, another might prefer performance. I don't think we can
say one need is greater than the other.

But if that's the goal, it really needs to be explicitly stated, and
well tested. The erasure transformation should also be applied to the
generic signatures; I suspect Eclipse will choke on what is currently
emitted (can someone knowledgeable test this please?):

  scala> class Val[A](val value: Int) extends AnyVal
  defined class Val

  scala> class C2 { def foo[A](in: Val[A]) = in }
  defined class C2

  scala> classOf[C2].getMethods.head
  res4: java.lang.reflect.Method = public int C2.foo(int)

  scala> res4.getGenericParameterTypes
  res5: Array[java.lang.reflect.Type] = Array(.Val<A>)

With a bit of work, IntelliJ can be made to expose the right
signatures to Java; it doesn't use the generic signatures.

> Regarding time frame: I think the fixes for value classes can be done this
> week or next at the outset. Redoing the design will add at least one or two
> months to the release schedule. I think our time is better spent on
> extending value classes to multiple parameters.

I'd favour the walk-before-we-run approach here.

-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/10/12 2:08 AM


On Mon, Sep 10, 2012 at 10:36 AM, Jason Zaugg <jza...@gmail.com> wrote:
On Mon, Sep 10, 2012 at 9:37 AM, martin odersky <martin....@epfl.ch> wrote:
> I don't know. The proposal only deals with method parameters, what about
> results? There are many other places where value classes come into the mix
> and we'd have to be sure we deal with them all. My preference in this case
> would be to have as simple a semantic model as possible.

The boxed bridge method would both unbox parameters and box results.

I see. How about fields? I assume you'd do the same thing for getters and setters, right? Next question: How do $unboxed variants relate to overrriding?
By now I fear getting another exponent of interaction with other bridge methods (overriding bridges, specialized variants). I would not want to do that before we had a careful look at specialization of value classes. And that's many months away, at best. (E.g. if the proposal means that we get another doubling of specialized methods, it won't fly).

Generally, $unboxed methods are very similar to specialized methods. We know that specialized is a rats nest of complexity. What kind of arguments do we have that could convince us that this proposal will not lead to the same problems?
 
> The current model is: Value classes erase to the same type as their
> underlying type.
>
> I think we can keep to it and address the concerns:
>
>  - structural types: Have a restriction that method signatures in structural
> types may not refer to value classes. We already have a restriction that
> they may not refer to enclosing class parameters, so this is nothing new. At
> some point we might want to come back and change the structural type
> dispatch scheme, which could then hopefully get rid of both restrictions in
> one fell sweep.
>
> - reflection: not sure yet, but I guess we can find the right rules. In any
> case, I vote for keeping reflection experimental for the current release so
> we have time to fix it.

Yep, Structural Types and Scala reflection both have enough
information to do the Right Thing, eventually. But I think it's
tempting to install unboxed bridges to avoid touching either of them.

Or declare Value Classes experimental?
  
>  - Java interop: Have the simple rule that value classes are invisible for
> Java. Java APIs have to deal with the underlying type instead. This is good,
> in my opinion. The whole purpose of value classes is to give nice syntactic
> coating for something that does not need boxing. Having to box from Java is
> not an improvement, IMO.  
They also offer type safety. One Java caller might prefer convenience
+ type safety, another might prefer performance. I don't think we can
say one need is greater than the other.

Yes, but offering choices to Java programmers is not one of our priorities. We need something simple that works, that's all. And, it would not really be an even choice anyway. With $unboxed methods we force Java programmers to box, unless they want to write $s in their method calls.
  
But if that's the goal, it really needs to be explicitly stated, and
well tested. The erasure transformation should also be applied to the
generic signatures; I suspect Eclipse will choke on what is currently
emitted (can someone knowledgeable test this please?):

Of course, the generic signatures have to be adapted. That's critical! 
I would classify this as a blocker. Somebody please file a ticket or this. Who could have a look at this? Paul?

  scala> class Val[A](val value: Int) extends AnyVal
  defined class Val

  scala> class C2 { def foo[A](in: Val[A]) = in }
  defined class C2

  scala> classOf[C2].getMethods.head
  res4: java.lang.reflect.Method = public int C2.foo(int)

  scala> res4.getGenericParameterTypes
  res5: Array[java.lang.reflect.Type] = Array(.Val<A>)

With a bit of work, IntelliJ can be made to expose the right
signatures to Java; it doesn't use the generic signatures.

> Regarding time frame: I think the fixes for value classes can be done this
> week or next at the outset. Redoing the design will add at least one or two
> months to the release schedule. I think our time is better spent on
> extending value classes to multiple parameters.

I'd favour the walk-before-we-run approach here.

Yes, except that we are on the countdown to a release. Stopping that countdown should not be taken lightly, and is in direct violation of our policy of going towards time-based releases.

So my vote is: Put in the restrictions now. Keep in mind a more flexible scheme with $unboxed methods that could overcome the restrictions. Wait until we have specialization overhauled and adapted to value classes before deciding on rolling in the flexible scheme. It will be binary incompatible, that's for sure, but it could overcome the restrictions.

I have a couple of other value class problems on my plate right now. It still might be that one of them is a blocker which causes a reassessment. I hopefully will be able to tell by the end of the week (very little time right now to put into it).

Cheers

 - Martin


Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/10/12 2:10 AM


True. And since this is not feasible, our only hope is to reduce the number of interacting features. That's why I don't like higher-kinded and existential types. Both have a multitude of nasty interactions with the rest of the language. Or, to pick a seemingly simple feature: Precise getClass typing. Undoubtedly useful, but has tripped us up many times because it is an irregular case. Last bug was (surprise!) with value classes. Or lazy vals and objects: We should have unified these two a long time ago.

Cheers

 - Martin
Re: SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/10/12 2:30 AM
On Mon, Sep 10, 2012 at 11:08 AM, martin odersky <martin....@epfl.ch> wrote:
>> The boxed bridge method would both unbox parameters and box results.
>>
> I see. How about fields? I assume you'd do the same thing for getters and
> setters, right? Next question: How do $unboxed variants relate to
> overrriding?

Good questions, I don't have answers just yet.

> By now I fear getting another exponent of interaction with other bridge
> methods (overriding bridges, specialized variants). I would not want to do
> that before we had a careful look at specialization of value classes. And
> that's many months away, at best. (E.g. if the proposal means that we get
> another doubling of specialized methods, it won't fly).

Without giving it full thought, I think it would be +1 rather than *2;
the unboxed bridge should invoke the unspecialized bridge.

> Generally, $unboxed methods are very similar to specialized methods. We know
> that specialized is a rats nest of complexity. What kind of arguments do we
> have that could convince us that this proposal will not lead to the same
> problems?

It borrows a nice idea from specialization, but I don't think it  --
we're not creating new subclasses. Conceptually, one could add all
these unboxed bridges as a post-processing step after the current
scheme as run. (assuming you can live with the unboxed and boxes
methods both having the same original name.)

>> They also offer type safety. One Java caller might prefer convenience
>> + type safety, another might prefer performance. I don't think we can
>> say one need is greater than the other.
>>
> Yes, but offering choices to Java programmers is not one of our priorities.
> We need something simple that works, that's all. And, it would not really be
> an even choice anyway. With $unboxed methods we force Java programmers to
> box, unless they want to write $s in their method calls.

Again, we could use the original name on both methods.

> Of course, the generic signatures have to be adapted. That's critical!
> I would classify this as a blocker. Somebody please file a ticket or this.
> Who could have a look at this? Paul?

https://issues.scala-lang.org/browse/SI-6344

> Yes, except that we are on the countdown to a release. Stopping that
> countdown should not be taken lightly, and is in direct violation of our
> policy of going towards time-based releases.
>
> So my vote is: Put in the restrictions now. Keep in mind a more flexible
> scheme with $unboxed methods that could overcome the restrictions. Wait
> until we have specialization overhauled and adapted to value classes before
> deciding on rolling in the flexible scheme. It will be binary incompatible,
> that's for sure, but it could overcome the restrictions.
>
> I have a couple of other value class problems on my plate right now. It
> still might be that one of them is a blocker which causes a reassessment. I
> hopefully will be able to tell by the end of the week (very little time
> right now to put into it).

I can understand that reasoning.

-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/10/12 2:32 AM
On Mon, Sep 10, 2012 at 11:30 AM, Jason Zaugg <jza...@gmail.com> wrote:

> It borrows a nice idea from specialization, but I don't think it  --
> we're not creating new subclasses.

*but I don't think it is anywhere near as complicated.

-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/10/12 3:14 AM


On Mon, Sep 10, 2012 at 11:30 AM, Jason Zaugg <jza...@gmail.com> wrote:
On Mon, Sep 10, 2012 at 11:08 AM, martin odersky <martin....@epfl.ch> wrote:
>> The boxed bridge method would both unbox parameters and box results.
>>
> I see. How about fields? I assume you'd do the same thing for getters and
> setters, right? Next question: How do $unboxed variants relate to
> overrriding?

Good questions, I don't have answers just yet.

> By now I fear getting another exponent of interaction with other bridge
> methods (overriding bridges, specialized variants). I would not want to do
> that before we had a careful look at specialization of value classes. And
> that's many months away, at best. (E.g. if the proposal means that we get
> another doubling of specialized methods, it won't fly).

Without giving it full thought, I think it would be +1 rather than *2;
the unboxed bridge should invoke the unspecialized bridge.

> Generally, $unboxed methods are very similar to specialized methods. We know
> that specialized is a rats nest of complexity. What kind of arguments do we
> have that could convince us that this proposal will not lead to the same
> problems?

It borrows a nice idea from specialization, but I don't think it  --
we're not creating new subclasses. Conceptually, one could add all
these unboxed bridges as a post-processing step after the current
scheme as run. (assuming you can live with the unboxed and boxes
methods both having the same original name.)

I think you are right. The hard parts of specialization seem to be indeed related to creating new subclasses. Method count explosion is still somewhat of a concern, for instance if a method has a value class parameter and another primitive class parameter it seems we now end up with twice as many specializations as before, because both normal and unboxed methods need to be specialized.

Another problem is this:

Say you have

  class V(x: Int) extends AnyVal
  def f(x: V, y: V)

and you end up calling f with an unboxed argument a and a boxed argument b. Do you translate to

  f(box(a), b)

or do you translate to

  f$unboxed(a, unbox(b))

? In either case you need to do the current caller-side box/unbox logic, so it seems you can't simplify anything from the current proposal.

Cheers

 - Martin

Re: SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/10/12 3:20 AM
On Mon, Sep 10, 2012 at 12:13 PM, martin odersky <martin....@epfl.ch> wrote:
>> It borrows a nice idea from specialization, but I don't think it  --
>> we're not creating new subclasses. Conceptually, one could add all
>> these unboxed bridges as a post-processing step after the current
>> scheme as run. (assuming you can live with the unboxed and boxes
>> methods both having the same original name.)
>
> I think you are right. The hard parts of specialization seem to be indeed
> related to creating new subclasses. Method count explosion is still somewhat
> of a concern, for instance if a method has a value class parameter and
> another primitive class parameter it seems we now end up with twice as many
> specializations as before, because both normal and unboxed methods need to
> be specialized.

We've got @nospecialize now; it might be suitable for the boxed bridge.

> Another problem is this:
>
> Say you have
>
>   class V(x: Int) extends AnyVal
>   def f(x: V, y: V)
>
> and you end up calling f with an unboxed argument a and a boxed argument b.
> Do you translate to
>
>   f(box(a), b)
>
> or do you translate to
>
>   f$unboxed(a, unbox(b))
>
> ? In either case you need to do the current caller-side box/unbox logic, so
> it seems you can't simplify anything from the current proposal.

I think the latter, as happens today. Scala would never generate calls
to the boxed version. Perhaps I don't quite understand your point
here.

-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/10/12 3:31 AM


My point was that it would have been nice if we could encapsulate the current call-site boxing and unboxing logic in the bridge method. That would have gained us something in simplicity. But the example shows that we cannot do that. So it's strictly an addition to the current proposal. The boxed methods are there only for interoperability with structural types, reflection and Java.

One way to turn the proposal around would be to have

  def f$boxed(x: V)

I.e.use the mangled name for the boxed version. Then we could call that from reflection or structural types or Java. And that change would be backwards compatible. Is that something to consider?

Cheers

 - Martin






Cheers

 - Martin
 
-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/10/12 3:43 AM
On Mon, Sep 10, 2012 at 12:31 PM, martin odersky <martin....@epfl.ch> wrote:
> My point was that it would have been nice if we could encapsulate the
> current call-site boxing and unboxing logic in the bridge method. That would
> have gained us something in simplicity. But the example shows that we cannot
> do that. So it's strictly an addition to the current proposal. The boxed
> methods are there only for interoperability with structural types,
> reflection and Java.

I see. Hopefully we could at least generated the body of the bridge
with the existing call-site transformations.

> One way to turn the proposal around would be to have
>
>   def f$boxed(x: V)
>
> I.e.use the mangled name for the boxed version. Then we could call that from
> reflection or structural types or Java. And that change would be backwards
> compatible. Is that something to consider?

Definitely a possibility. I'd lean towards just calling if "f".

-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/10/12 3:46 AM


And accept the potential erasure type clashes. Yes, that's definitely a possibility. And it's something we can probably still do post 2.10.

Cheers

 - Martin

Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/10/12 5:41 AM
Just found that now scalac generates half-boxed versions for value class methods: their arguments of this.type are boxed while other value types are not:

class Foo(val value: Int) extends AnyVal {
  def fb_f(f: Foo, b: Bar): Foo = bf_f(b, f)
  def fb_b(f: Foo, b: Bar): Bar = bf_b(b, f)
  def bf_f(b: Bar, f: Foo): Foo = fb_f(f, b)
  def bf_b(b: Bar, f: Foo): Bar = fb_b(f, b)
}
class Bar(val value: Int) extends AnyVal


results in:

=========================================
public final class Foo {
    public static int extension$bf_b(int i, int j, int k) {
        return Foo$.MODULE$.extension$bf_b(i, j, k);
    }
   ...
    public Foo fb_f(Foo f, int b) {
        return new Foo(Foo$.MODULE$.extension$fb_f(value(), f.value(), b));
    }

    public int fb_b(Foo f, int b) {
        return Foo$.MODULE$.extension$fb_b(value(), f.value(), b);
    }

    public Foo bf_f(int b, Foo f) {
        return new Foo(Foo$.MODULE$.extension$bf_f(value(), b, f.value()));
    }

    public int bf_b(int b, Foo f) {
        return Foo$.MODULE$.extension$bf_b(value(), b, f.value());
    }
================================

By the way, why there are all these methods in Foo$.MODULE$ ?
I believe it's better to generate code right in static methods in Foo.class.
This way calling value class methods may be somewhat more efficient.
And yes, we can replace tripling the value class methods count to just doubling it.

Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/10/12 6:53 AM


On Mon, Sep 10, 2012 at 2:08 AM, martin odersky <martin....@epfl.ch> wrote:
Of course, the generic signatures have to be adapted. That's critical! 
I would classify this as a blocker. Somebody please file a ticket or this. Who could have a look at this? Paul?

Given the spectacular amount of time I invested in SI-3452 without entering the end zone, I can't promise much.  But I'll look at it.

For reference only, I don't expect anyone to actually dive into this:


Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/10/12 6:59 AM


On Mon, Sep 10, 2012 at 5:41 AM, Pavel Pavlov <pavel.e...@gmail.com> wrote:
By the way, why there are all these methods in Foo$.MODULE$ ?
I believe it's better to generate code right in static methods in Foo.class.
This way calling value class methods may be somewhat more efficient.

I noted this in 


It isn't done mostly because there is not yet a general facility for bypassing the companion module instance.  Work has begun (by which I mean, there is now @static) but there is nothing robust.

However, I benchmarked extensively at some point and discovered that the performance difference is negligible.  (I went in expecting otherwise.) It's still worth doing.

Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/12/12 1:03 AM
SI-6260 has uncovered a root problem that affects many of the other value class tickets as well. 

It looks like, with the current implementation scheme it is fundamentally impossible to have a value class wrap over one of its type parameters.

I.e.  

     class V[T](x: T) extends AnyVal

is problematic whereas

     class W[T](x: List[T]) extends AnyVal

is not. There are a multitude of tickets open against the former case. I tracked down one which seems to have no straightforward solution, SI-6260:


class Box[X](val x: X) extends AnyVal {
  def map[Y](f: X => Y): Box[Y] =
    ((bx: Box[X]) => new Box(f(bx.x)))

      (this)

}

new Box(42) map (_ + 1)

The symptom was a "duplicate method". The duplicate method was a bridge method for Function1 in the closure of map (line 3). Both the bridge and the original apply method had signature (Object)Object. But the bridge does essential work; it unboxes the incoming Object to an Int value which then needs to be boxed again as an Integer. 

So, no easy solution.

It seems without re-thinking the whole erasure thing (which is a huge undertaking, because erasure is very subtle and worked pretty well so far, so better not upheave it), we simply cannot
implement value classes that wrap a type parameter.

In fact, SIP-15 as discussed and accepted did not allow this case; it was added later. (I believe to make ArrowAssoc work as a value class, are there other cases where we need it in the libraries right now?)

So we have to revert that.

Unfortunately it also means that the likelihood of making Option a value class has become much smaller.

One more question: I noticed that the structural type ticket SI-6336 also relies on on type-parameter wrapping value classes. Can anyone think of a case where the same problem happens with structural types, but with other value classes? If not, we can maybe not introduce the structural type restriction, eliminating the problematic form of value classes is enough.

Cheers

 - Martin

Re: SIP-15 Value class encoding breaks reflection, structural types Jason Zaugg 9/12/12 1:30 AM
On Wed, Sep 12, 2012 at 10:02 AM, martin odersky <martin....@epfl.ch> wrote:
> One more question: I noticed that the structural type ticket SI-6336 also
> relies on on type-parameter wrapping value classes. Can anyone think of a
> case where the same problem happens with structural types, but with other
> value classes? If not, we can maybe not introduce the structural type
> restriction, eliminating the problematic form of value classes is enough.

https://issues.scala-lang.org/browse/SI-6347 (which is merged with SI-6336).

-jason
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/12/12 6:51 AM


On Wed, Sep 12, 2012 at 1:02 AM, martin odersky <martin....@epfl.ch> wrote:
In fact, SIP-15 as discussed and accepted did not allow this case; it was added later. (I believe to make ArrowAssoc work as a value class, are there other cases where we need it in the libraries right now?)

There are other places where it is used (Tuple2Zipped at least) but there isn't anywhere that we need it.  Parameterized value classes will not be truly useful until we can also specialize them; for that reason, and because I am afraid this situation has even more interactions and complications than we yet realize, I am in favor of prohibiting them now and putting all the effort into making the simplest value class scenarios work robustly.

There is no category of bugs I can think of upon which forward progress has been slower than the broad category of "generating the right method signatures and dispatching to the right targets" in the face of overloading, erasure, implementation classes, specialization, structural types, bridges, and accessors.  Throwing parameterized value classes into that mix seems like excitement which can wait.

Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/12/12 7:01 AM


Agreed. Can you take care of removing them? I am afraid I am totally loaded with MOOC recordings for the rest of the week.

Just to make sure: When you talk about parameterized value classes you mean

  class V[T](x: T) extends AnyVal

Right? I.e 

  class V[T](xs: List[T])

should still work, even though it is parameterized. Throwing out the latter would render value classes next to useless.

Cheers

 - Martin


Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/12/12 7:09 AM


On Wed, Sep 12, 2012 at 7:01 AM, martin odersky <martin....@epfl.ch> wrote:
  class V[T](xs: List[T])

should still work, even though it is parameterized. Throwing out the latter would render value classes next to useless.

It would leave our current usage of them essentially untouched to throw them out! But it should be no problem to leave those alone because the erasure is fixed.

Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/12/12 7:09 AM
So yes, barring obstacle I'll pull them today.
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/12/12 7:16 AM


On Wed, Sep 12, 2012 at 4:09 PM, Paul Phillips <pa...@improving.org> wrote:


On Wed, Sep 12, 2012 at 7:01 AM, martin odersky <martin....@epfl.ch> wrote:
  class V[T](xs: List[T])

should still work, even though it is parameterized. Throwing out the latter would render value classes next to useless.

It would leave our current usage of them essentially untouched to throw them out! But it should be no problem to leave those alone because the erasure is fixed.

I was thinking mostly of implicit wrappers, which all tend to be parametric. - Martin 



Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/12/12 7:23 AM


On Wednesday, September 12, 2012, martin odersky wrote:
I was thinking mostly of implicit wrappers, which all tend to be parametric. - Martin 

Right, in everyone else's source bases that is no doubt true (it's good to occasionally be reminded that our source base is not entirely representative) but in our case there's lots of hand-specialization.

class Boolean private extends AnyVal
class Byte private extends AnyVal
class Char private extends AnyVal
class Double private extends AnyVal
class DurationDouble(val d: Double) extends AnyVal
class DurationInt(val n: Int) extends AnyVal
class DurationLong(val n: Long) extends AnyVal
class Float private extends AnyVal
class Int private extends AnyVal
class Long private extends AnyVal
class RichBoolean(val self: Boolean) extends AnyVal
class RichByte(val self: Byte) extends AnyVal
class RichChar(val self: Char) extends AnyVal
class RichDouble(val self: Double) extends AnyVal
class RichFloat(val self: Float) extends AnyVal
class RichInt(val self: Int) extends AnyVal
class RichLong(val self: Long) extends AnyVal
class RichShort(val self: Short) extends AnyVal
class Short private extends AnyVal
class StringAdd(val self: Any) extends AnyVal
class StringFormat(val self: Any) extends AnyVal
class StringOps(override val repr: String) extends AnyVal
class Unit private extends AnyVal
class ofBoolean(override val repr: Array[Boolean]) extends AnyVal
class ofByte(override val repr: Array[Byte]) extends AnyVal
class ofChar(override val repr: Array[Char]) extends AnyVal
class ofDouble(override val repr: Array[Double]) extends AnyVal
class ofFloat(override val repr: Array[Float]) extends AnyVal
class ofInt(override val repr: Array[Int]) extends AnyVal
class ofLong(override val repr: Array[Long]) extends AnyVal
class ofShort(override val repr: Array[Short]) extends AnyVal
class ofUnit(override val repr: Array[Unit]) extends AnyVal
 
class ArrowAssoc[A](val __leftOfArrow: A) extends AnyVal
class Ensuring[A](val __resultOfEnsuring: A) extends AnyVal
class Ops[T1, T2, T3](val x: (T1, T2, T3)) extends AnyVal
class Ops[T1, T2](val x: (T1, T2)) extends AnyVal
class ofRef[T <: AnyRef](override val repr: Array[T]) extends AnyVal

Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/12/12 7:32 AM


On Wednesday, September 12, 2012 9:01:41 PM UTC+7, Martin wrote:


On Wed, Sep 12, 2012 at 3:51 PM, Paul Phillips <pa...@improving.org> wrote:


On Wed, Sep 12, 2012 at 1:02 AM, martin odersky <martin....@epfl.ch> wrote:
In fact, SIP-15 as discussed and accepted did not allow this case; it was added later. (I believe to make ArrowAssoc work as a value class, are there other cases where we need it in the libraries right now?)

There are other places where it is used (Tuple2Zipped at least) but there isn't anywhere that we need it.  Parameterized value classes will not be truly useful until we can also specialize them; for that reason, and because I am afraid this situation has even more interactions and complications than we yet realize, I am in favor of prohibiting them now and putting all the effort into making the simplest value class scenarios work robustly.

There is no category of bugs I can think of upon which forward progress has been slower than the broad category of "generating the right method signatures and dispatching to the right targets" in the face of overloading, erasure, implementation classes, specialization, structural types, bridges, and accessors.  Throwing parameterized value classes into that mix seems like excitement which can wait.

Agreed. Can you take care of removing them? I am afraid I am totally loaded with MOOC recordings for the rest of the week.

Just to make sure: When you talk about parameterized value classes you mean

  class V[T](x: T) extends AnyVal

Right? I.e 

  class V[T](xs: List[T])

should still work, even though it is parameterized. Throwing out the latter would render value classes next to useless.

Unfortunately, this won't help.
`class V(val x: Any) extends AnyVal` suffers from the same problem:
===================================
object Box1 {
  type T = Any //or any other type erased to `Object`: AnyRef, AnyVal, structural type etc.
  def main(args: Array[String]) {
    new Box1(42) map { case x: Int => x + 1 }
  }
}

class Box1(val x: Box1.T) extends AnyVal {
  type T = Box1.T
  def map(f: T => T): Box1 =
    ((bx: Box1) => new Box1(f(bx.x))) (this)
}
===================================

I believe the right way to solve the problem is to mangle method names which have value class arguments/results.
This way we can cure SI-6260, allow structural-value types interaction and add a bit of Java interop, all in one shot.

Regards,
Pavel
Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/12/12 7:43 AM


On Wednesday, September 12, 2012 9:32:39 PM UTC+7, Pavel Pavlov wrote:

I believe the right way to solve the problem is to mangle method names which have value class arguments/results.
This way we can cure SI-6260, allow structural-value types interaction and add a bit of Java interop, all in one shot.

I mean that signature of `def foo(x: VC)` should not be touched, the method should be converted to forwarder
for synthetic `def foo$vc(x: UnderlyingTypeOfVC)`
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/12/12 7:50 AM


On Wed, Sep 12, 2012 at 7:32 AM, Pavel Pavlov <pavel.e...@gmail.com> wrote:
Unfortunately, this won't help.

I think we should/must restrict our way to a solution right now.  I am terrified of adding any new, hastily conceived name-mangling schemes.  We are already so fenced in by our existing mangling.

I would be fine with also prohibiting value classes where the value erases to Object.
Re: SIP-15 Value class encoding breaks reflection, structural types Seth Tisue 9/12/12 8:05 AM
On Wed, Sep 12, 2012 at 10:50 AM, Paul Phillips <pa...@improving.org> wrote:
> I think we should/must restrict our way to a solution right now.

+1. This stuff can always continue to improve in 2.11 and beyond.

Better to ship a more limited feature that actually works, than
something that seems to promise the sky but is full of holes if you
look closely.

--
Seth Tisue | Northwestern University | http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/
Re: SIP-15 Value class encoding breaks reflection, structural types Rex Kerr 9/12/12 8:43 AM


On Wed, Sep 12, 2012 at 11:04 AM, Seth Tisue <se...@tisue.net> wrote:
On Wed, Sep 12, 2012 at 10:50 AM, Paul Phillips <pa...@improving.org> wrote:
> I think we should/must restrict our way to a solution right now.

+1. This stuff can always continue to improve in 2.11 and beyond.

Better to ship a more limited feature that actually works, than
something that seems to promise the sky but is full of holes if you
look closely.

+2 on that sentiment.  (*cough* specialization *cough*--I wasted way too much time trying to work around all the bugs there)

  --Rex

Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/12/12 8:44 AM


On Wednesday, September 12, 2012, Rex Kerr wrote:
+2 on that sentiment.  (*cough* specialization *cough*--I wasted way too much time trying to work around all the bugs there)

Imagine what it has been like trying to FIX them.
 
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/12/12 10:44 AM
Note that we are talking about the bridge for the _apply_ method of Function1. I do not see how that name could be mangled.

Barring value classes over Any or Object will not be enough either, because the bridge conflict might easily arise with other types. I think we need to put in another test against bridges that erase to the same type as an existing definition. In principle that's nothing new, we do have these tests, but they did not do the right thing for value classes. But now I think I know exactly what to do. Thanks for putting me on the right track!

I'll have a look and report back. 

That discussion notwithstanding I still think it might be wise to disallow fully parametric wrappers of form

  class V[T](x: T)

as they seem to be a source of several other problems and we can't claim we understand the interactions fully.

Cheers

 - Martin



Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/12/12 11:18 AM


On Wed, Sep 12, 2012 at 7:43 PM, martin odersky <martin....@epfl.ch> wrote:


On Wed, Sep 12, 2012 at 4:43 PM, Pavel Pavlov <pavel.e...@gmail.com> wrote:


On Wednesday, September 12, 2012 9:32:39 PM UTC+7, Pavel Pavlov wrote:

I believe the right way to solve the problem is to mangle method names which have value class arguments/results.
This way we can cure SI-6260, allow structural-value types interaction and add a bit of Java interop, all in one shot.

I mean that signature of `def foo(x: VC)` should not be touched, the method should be converted to forwarder
for synthetic `def foo$vc(x: UnderlyingTypeOfVC)`

Note that we are talking about the bridge for the _apply_ method of Function1. I do not see how that name could be mangled.

I see now. Leave the bridge as `apply`, but rename the concrete apply method containing the Box types, I guess. That could work. But I agree with Paul that it's too risky to introduce new name mangling for 2.10.

Cheers

 - Martin

Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/12/12 11:40 AM


On Thursday, September 13, 2012 1:18:04 AM UTC+7, Martin wrote:


On Wed, Sep 12, 2012 at 7:43 PM, martin odersky <martin....@epfl.ch> wrote:


On Wed, Sep 12, 2012 at 4:43 PM, Pavel Pavlov <pavel.e...@gmail.com> wrote:


On Wednesday, September 12, 2012 9:32:39 PM UTC+7, Pavel Pavlov wrote:

I believe the right way to solve the problem is to mangle method names which have value class arguments/results.
This way we can cure SI-6260, allow structural-value types interaction and add a bit of Java interop, all in one shot.

I mean that signature of `def foo(x: VC)` should not be touched, the method should be converted to forwarder
for synthetic `def foo$vc(x: UnderlyingTypeOfVC)`

Note that we are talking about the bridge for the _apply_ method of Function1. I do not see how that name could be mangled.

I see now. Leave the bridge as `apply`, but rename the concrete apply method containing the Box types, I guess.
Exactly.
 
That could work. But I agree with Paul that it's too risky to introduce new name mangling for 2.10.
I understand this decision. At this point of time risks are too high.
 

How about half-boxing in signatures of value class methods?
https://groups.google.com/d/msg/scala-sips/KSm99KoiYNA/wuX6j_XgeZEJ


Cheers

 - Martin

Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/12/12 1:09 PM


On Wednesday, September 12, 2012, martin odersky wrote:
That discussion notwithstanding I still think it might be wise to disallow fully parametric wrappers of form

  class V[T](x: T)

as they seem to be a source of several other problems and we can't claim we understand the interactions fully.

I have that part done.  Should we care to take advantage of it, I also wrote a private[scala]

  scala.annotation.unchecked.uncheckedValueClass

which suppresses the check.  These issues do not arise unless one is creating closures or otherwise causing methods to be generated in the value class, right? We can trust ourselves not to do that, and then we don't have to lose ArrowAssoc and friends, which I'm fairly sure have a measurable effect on performance.

I could also place the general prohibition at the point of closure creation rather than disallowing the value class entirely, though that's a little riskier. 

Re: SIP-15 Value class encoding breaks reflection, structural types Mark Harrah 9/13/12 7:10 AM
It looks like SI-6260 is the only bug being discussed here from the bare type parameter and closure comments.  However, if SI-6337 is included in this discussion, that one doesn't require a bare type parameter and I don't think it has anything to do with generating closures or methods in the value class.  The generic signatures bug (SI-6344) doesn't either.  It doesn't even use the type parameter.

(I tagged all of the value class bugs with the valueclass label if you want to look at the open value class bugs.)

-Mark

https://issues.scala-lang.org/browse/SI-6260
https://issues.scala-lang.org/browse/SI-6337
https://issues.scala-lang.org/browse/SI-6344
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/15/12 9:26 AM


On Wed, Sep 12, 2012 at 10:43 AM, martin odersky <martin....@epfl.ch> wrote:
I think we need to put in another test against bridges that erase to the same type as an existing definition. In principle that's nothing new, we do have these tests, but they did not do the right thing for value classes. But now I think I know exactly what to do. Thanks for putting me on the right track!

I don't see that this has gotten us much closer.  As I understand the problem, we have a class like

  class Box[X](val x: X) extends AnyVal

And we are generating a closure along the lines of 

  (x: Int) => x + 1

The closure is faced with two possibilities involving boxing:

 - it may be applied to an Integer
 - it may be applied to a Box[Int]

The (Object)Object implementations for these situations look very different:

 - one looks like the ones in BoxesRunTime, new Integer / intValue()
 - the other looks different for every value class - it is the duplicate method suppressed by https://github.com/scala/scala/pull/1285

  public final java.lang.Object apply(java.lang.Object);
    flags: ACC_PUBLIC, ACC_FINAL, ACC_BRIDGE, ACC_SYNTHETIC
    Code:
      stack=4, locals=2, args_size=2
         0: new           #33                 // class Box
         3: dup           
         4: aload_0       
         5: aload_1       
         6: checkcast     #33                 // class Box
         9: invokevirtual #37                 // Method Box.x:()Ljava/lang/Object;
        12: invokevirtual #38                 // Method apply:(Ljava/lang/Object;)Ljava/lang/Object;
        15: invokespecial #42                 // Method Box."<init>":(Ljava/lang/Object;)V
        18: areturn       

So by suppressing that, we avoid the VerifyError and walk right into a ClassCastException, because we need that method.

If these things are true:

 - there can only be one (Object)Object bridge
 - the same closure must be usable for a function which acts on Ints and one which acts on "value class Ints"

This would seem to imply either the bridge method or the box/unbox methods know how to a) distinguish these cases and b) box/unbox to the right structures.  Even if that is viable, I don't want to know how much it would slow things down.

I could easily be going off the beam somewhere here; if someone has a better understanding, I would be glad to be corrected.

Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/15/12 9:59 AM
Hi Paul,

I think your understanding is correct. The idea in the case of SI-6260 is to reject the program because there is no way to generate a bridge.
Even that is tricky. I am currently trying to find the right criterion. 

Cheers

 - Martin

--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/15/12 11:44 AM
Hi Paul,

Please correct me if I'm wrong below:

Any one-argument closure is subclass of `Function1[A, B] { def apply(x: A): B }`.
As we have no generics in bytecode, we have there `def apply(x: Object): Object` signature for Function1's apply method.
This abstract method is in fact the only entry point for (any) function application for any possible subclass of Function1 (let's leave specialization aside for now).
So, our closure must implement this signature to be callable function.
Stronger guarantees for input/output types (for example for `Function1[Int, Boolean]`) cannot be expressed at bytecode level and thus
cannot be "statically" (i.e. at load-time) checked by verifier, so there we have in fact "dynamically typed" language with consistence checks performed at run-time.
So, your second condition for closures (_it may be applied to a Box[Int]_) is inavoidable for any Function1's subclass.

As regards to first condition (_it may be applied to an Integer_) it is not necessary if we think in general, and not about particular implementation scheme we have now in scalac.
Moreover, this condition is also unsound: imagine for example a function returning value class:

val f: Any => Any = _ => Box(1)

We have no static guaranties for the result of this function (as its type is Any), but we have strong run-time guaranties: this function must always return a `Box` object and nothing else.

For this function "_it may be applied to an Integer_" means that call to `f` may result in returning Integer which definitely is not what we want.
So, as at call site we in no way can distinguish which `apply` version (returning Box or returning Int) we must call, we have no choice in implementing Function1's
`apply(Object)Object` method: it must be method that always returns Box. So, `apply` returning Integer is not only conflicting with another bridge method, but also wrong and pretty useless: we cannot implement abstract Function1's `apply` with it nor we can use it anywhere we don't have enough information about actual f's type parameters.

So I still believe that mangling methods which have unboxed value class arguments/results (at least methods which overrides/implements those of superclasses/traits) is the only viable solution.

Regards,
Pavel
Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/15/12 12:00 PM


On Sunday, September 16, 2012 1:44:07 AM UTC+7, Pavel Pavlov wrote:

So I still believe that mangling methods which have unboxed value class arguments/results (at least methods which overrides/implements those of superclasses/traits) is the only viable solution.

Or we can skip unboxing of method's parameter/return types which have not-value-unboxed corresponding parameter/return type in any supermethod (not sure that I expressed this clearly enough).
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/15/12 12:08 PM


On Sat, Sep 15, 2012 at 11:44 AM, Pavel Pavlov <pavel.e...@gmail.com> wrote:
For this function "_it may be applied to an Integer_" means that call to `f` may result in returning Integer which definitely is not what we want.

Maybe it is not what we want, but it's not apparent to me yet that we can't work with it.  Why can't value classes also deal with primitive boxes directly.  The bridge method in the closure would deal only with boxing and unboxing Int/Integers, just as it did before value classes.  Give every value class box and unbox methods in its companion which translate between the value class's personal box and the generic "Integer"-style box.

At call sites where you know nothing, you route everything through standard boxes; at locations where you know more, you can bypass them.

Is this conception broken?

By the way, keep in mind that we can overload on return type at the jvm level.  So in principle all these methods can coexist if it serves our purposes:

  (Object)Box
  (Object)Integer
  (Object)Object

Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/15/12 12:14 PM


On Sunday, September 16, 2012 2:08:41 AM UTC+7, Paul Phillips wrote:

By the way, keep in mind that we can overload on return type at the jvm level.  So in principle all these methods can coexist if it serves our purposes:
  (Object)Box
  (Object)Integer
  (Object)Object

Of course, they can coexist, but how you will call them?  Via reflection? Or will you generate extra interfaces declaring methods with such signatures?
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/15/12 12:19 PM


On Sat, Sep 15, 2012 at 12:14 PM, Pavel Pavlov <pavel.e...@gmail.com> wrote:
Of course, they can coexist, but how you will call them?  Via reflection? Or will you generate extra interfaces declaring methods with such signatures?

I figure we generate them based on local knowledge.  It may have to have a generic (Object)Object method, but it's not like we don't generally know from surrounding context what sort of boxed types are of particular interest.

Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/15/12 12:30 PM
If you (I mean scalac) resrict youself with creating only one anonymous class (let's call it $anon) for each closure then you have two cases:
1) val f has type `$anon`. In that case any call to `f.apply` may be devirtualized and inlined. I believe scalac inliner is exactly the one who hould care about this.
2) val f has type `Function1[A, B]` (as Function1 is the _immediate_ supertrait of `$anon`). In that case you're restricted to call `apply(Object)Object`, as there are no other methods in `Function1` interface at bytecode level.
This situation can be changed only by injecting additional interfaces into `Function` -> $anon` inheritance chain.
Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/15/12 12:37 PM

BTW, I think now this may be solution to all the problems discussed in this thread (if it hasn't any holes).
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/15/12 12:52 PM


On Sat, Sep 15, 2012 at 12:30 PM, Pavel Pavlov <pavel.e...@gmail.com> wrote:
) val f has type `$anon`.

Right, this is the case I am talking about - it should be the overwhelming majority of cases.

Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/15/12 12:57 PM

And this is _exactly_ the case for totally safe & cheap[1] inlining/closure elimination.

[1] Cheap in terms of required static analysis: this case require no analysis at all, so all transformations can be done right in ASTs.

Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/17/12 9:26 AM
So here is the current status:

I have pull requests for most outstanding value class tickets. We can now handle the infamous bridge methods of SI-6260, even though error messages can be improved. The crash when interacting with lazy vals is gone. Other problems are solved by restrictions:

 - no value classes in structural refinements
 - value classes cannot unbox to other derived value classes.

It strikes me that in all this the previous suspect, fully polymorphic value classes, did not make a difference, contrary to what I had assumed and feared. So, should we still introduce that restriction, or drop it and allow fully polymorphic value classes? I would be now be in favor of dropping the restriction, but could probably be convinced otherwise.

Cheers

 - Martin




--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/17/12 9:56 AM
On Mon, Sep 17, 2012 at 9:25 AM, martin odersky <martin....@epfl.ch> wrote:
> It strikes me that in all this the previous suspect, fully polymorphic value
> classes, did not make a difference, contrary to what I had assumed and
> feared. So, should we still introduce that restriction, or drop it and allow
> fully polymorphic value classes? I would be now be in favor of dropping the
> restriction, but could probably be convinced otherwise.

Test cases would be instrumental in convincing us one way or the
other.  If anyone out there has some time to exercise value classes
(the referenced bugs will be helpful in guiding what sorts of things
to test) it would be very helpful.
Re: SIP-15 Value class encoding breaks reflection, structural types Mark Harrah 9/17/12 1:58 PM
On Mon, 17 Sep 2012 18:25:45 +0200
martin odersky <martin....@epfl.ch> wrote:

> So here is the current status:
>
> I have pull requests for most outstanding value class tickets. We can now
> handle the infamous bridge methods of SI-6260, even though error messages
> can be improved. The crash when interacting with lazy vals is gone.

There is a problem with type parameters getting messed up.  See my last comment in:

 https://issues.scala-lang.org/browse/SI-6358

> Other
> problems are solved by restrictions:
>
>  - no value classes in structural refinements

See my last comment on

 https://issues.scala-lang.org/browse/SI-6336

that  shows the return type needs to exclude value classes as well:

>  - value classes cannot unbox to other derived value classes.

Might need to include universal traits in that exclusion:

 https://issues.scala-lang.org/browse/SI-6337

> It strikes me that in all this the previous suspect, fully polymorphic
> value classes, did not make a difference, contrary to what I had assumed
> and feared. So, should we still introduce that restriction, or drop it and
> allow fully polymorphic value classes? I would be now be in favor of
> dropping the restriction, but could probably be convinced otherwise.

I don't know what the underlying cause is, but see:

 https://issues.scala-lang.org/browse/SI-6385

-Mark

> Cheers
>
>  - Martin
>
>
>
>
>
> On Sat, Sep 15, 2012 at 9:57 PM, Pavel Pavlov <pavel.e...@gmail.com>wrote:
>
> >
> >
> > On Sunday, September 16, 2012 2:52:07 AM UTC+7, Paul Phillips wrote:
> >
> >>
> >>
> >> On Sat, Sep 15, 2012 at 12:30 PM, Pavel Pavlov <pavel.e...@gmail.com>wrote:
> >>
> >>> ) val f has type `$anon`.
> >>
> >>
> >> Right, this is the case I am talking about - it should be the
> >> overwhelming majority of cases.
> >>
> >>
> > And this is _exactly_ the case for totally safe & cheap[1]
> > inlining/closure elimination.
> >
> > [1] Cheap in terms of required static analysis: this case require no
> > analysis at all, so all transformations can be done right in ASTs.
> >
> >
>
>
> --
> Martin Odersky
> Prof., EPFL <http://www.epfl.ch> and Chairman, Typesafe<http://www.typesafe.com>
Re: SIP-15 Value class encoding breaks reflection, structural types Pavel Pavlov 9/18/12 3:37 AM
More specimens from the same can of worms:

== test1.scala ==
trait Foo extends Any {
  def box1(x: Box1): String
  def box2(x: Box2): String
}

class Box1(val value: String) extends AnyVal

class Box2(val value: String) extends AnyVal with Foo {
  def box1(x: Box1) = "box1: ok"
  def box2(x: Box2) = "box2: ok"
}

object test1 {

  def main(args: Array[String]) {
    val b1 = new Box1("")
    val b2 = new Box2("")
    val f: Foo = b2
    println(f.box1(b1))
    println(f.box2(b2))
  }
}
=============
produce
-----
box1: ok
java.lang.ClassCastException: java.lang.String cannot be cast to Box2
    at Box2.box2(test1.scala:8)
-----

== test2a.scala ==
trait Foo[T <: AnyVal] extends Any {
  def foo(x: String): String
  def foo(x: T): String
}

class Box1(val value: String) extends AnyVal with Foo[Box2] {
  def foo(x: String) = "foo(String): ok"
  def foo(x: Box2) = "foo(Box2): ok"
}

class Box2(val value: String) extends AnyVal

object test2a {

  def main(args: Array[String]) {
    val b1 = new Box1(null)
    val b2 = new Box2(null)
    val f: Foo[Box2] = b1
    println(f.foo(""))
    println(f.foo(b2))
  }
}
==============
produce compile error:
-----
test2a.scala:8: error: double definition:
method foo:(x: Box2)String and
method foo:(x: String)String at line 7
have same type after erasure: (x: String)String
  def foo(x: Box2) = "foo(Box2): ok"
      ^
one error found
-----

while almost identical
== test2b.scala ==
trait Foo[T <: AnyVal] extends Any {
  def foo(x: String): String
  def foo(x: T): String
}

/*class Box1(val value: String) extends AnyVal with Foo[Box2] {
  def foo(x: String) = "foo(String): ok"
  def foo(x: Box2) = "foo(Box2): ok"
}*/

class Box2(val value: String) extends AnyVal with Foo[Box2] {
  def foo(x: String) = "foo(String): ok"
  def foo(x: Box2) = "foo(Box2): ok"
}

object test2b {

  def main(args: Array[String]) {
    val b2 = new Box2(null)
    val f: Foo[Box2] = b2
    println(f.foo(""))
    println(f.foo(b2))
  }
}
==============
compiles and runs flawlessly:
-----
foo(String): ok
foo(Box2): ok
-----

All these are variations of the same bug with non-uniform value-class-unboxing in method signatures (I've tried to draw attention to this problem twice in this thread but with no luck).

These bugs looks very similar to SI-6385 for me.
Re: SIP-15 Value class encoding breaks reflection, structural types Josh Suereth 9/18/12 5:19 AM
Actually, can you open a new bug for this?  Also thanks much for the reports.  We've been listening, even if you haven't seen it on the ML.   Value classes are top of my discussion for the Scala meeting today.
Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/18/12 7:33 AM
I'd be fine with disallowing overloading involving value classes entirely.  I think we can disallow a great deal where value classes are involved and still hang onto the great majority of the current uses, which I am loathe to give up.  I have watched erasure bugs like these go unsolved for years so it stretches credulity to think we will be dashing off non-regrettable solutions for 2.10.

Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/18/12 7:38 AM


On Mon, Sep 17, 2012 at 10:58 PM, Mark Harrah <dmha...@gmail.com> wrote:
On Mon, 17 Sep 2012 18:25:45 +0200
martin odersky <martin....@epfl.ch> wrote:

> So here is the current status:
>
> I have pull requests for most outstanding value class tickets. We can now
> handle the infamous bridge methods of SI-6260, even though error messages
> can be improved. The crash when interacting with lazy vals is gone.

There is a problem with type parameters getting messed up.  See my last comment in:

 https://issues.scala-lang.org/browse/SI-6358

Oh yes, I see. It's really not the fault of value classes but of the broken implementation of lazy vals. Let's see what we can do about it.
 
> Other
> problems are solved by restrictions:
>
>  - no value classes in structural refinements

See my last comment on

 https://issues.scala-lang.org/browse/SI-6336

that  shows the return type needs to exclude value classes as well:

I agree, this needs to be added. 

>  - value classes cannot unbox to other derived value classes.

Might need to include universal traits in that exclusion:

 https://issues.scala-lang.org/browse/SI-6337

I have to dig a bit deeper on that one. 



--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/19/12 2:20 AM


On Tue, Sep 18, 2012 at 4:33 PM, Paul Phillips <pa...@improving.org> wrote:
I'd be fine with disallowing overloading involving value classes entirely.  I think we can disallow a great deal where value classes are involved and still hang onto the great majority of the current uses, which I am loathe to give up.  I have watched erasure bugs like these go unsolved for years so it stretches credulity to think we will be dashing off non-regrettable solutions for 2.10.


I now also think that the "partial unboxing" which was done to allow overloading in value classes is more trouble than it's worth.

To remind everyone:

Partial unboxing was put in so we could write code like:

  class Meter(x: Double) extends AnyVal {
    def / (other: Meter): Double = ...
    def / (other: Double): Meter = ...
  }

With complete unboxing both erased signatures of "/" would be (Double)Double, so we'd get a clash. With partial unboxing, "Meter" in the value class itself would not be erased.

However, this irregularity leads to surprising bridge methods which are not working correctly. That was Pavel's example (which I append below).

Now, I would argue that the scheme is too limiting anyway, and that there exist more robust ways to avoid name clashes.
Too limited: 

1) If we abstract out the operations into a universal trait, we do get an erasure clash:

  trait MeterTrait extends Any {
    def / (other: Meter): Double
    def / (other: Double): Meter
  }
  class Meter(x: Double) extends AnyVal with MeterTrait {
    def / (other: Meter): Double = ...
    def / (other: Double): Meter = ...
  }

2) If we mix more than one value class we also get erasure clashes, as in:

   class Meter(x: Double) extends AnyVal {
     def / (other: Second): Speed = ...
     def / (other: Double): Meter = ...
   }
   class Second(x: Double) extends AnyVal ...

So the trick does not buy us much. Furthermore, we know that we can disambiguate more robustly using implicit phantom parameters.  E.g.

   class Meter(x: Double) extends AnyVal {
     def / (other: Meter)(implicit dummy: MeterArg = null) = ...
     def / (other: Double): Meter = ...
   }
   object Meter {
     trait MeterArg
   }

So, I am for removing this special case.

What do people think?

 - Martin


Here's Pavel's problematic example:
Re: SIP-15 Value class encoding breaks reflection, structural types Roland Kuhn 9/19/12 3:25 AM
While I am not in a position to make well thought-through proposals on this matter, I would like to say that

less magic + more regular = profit!

With that I mean more direct error messages (possibly with a clear work-around suggestion) instead of searching for why things break at a completely different place in the code mysteriously. If that means writing a few more characters as a library implementor, that’s a price I consider to be small.

Regards,

Roland

19 sep 2012 kl. 11:20 skrev martin odersky:

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn


Re: SIP-15 Value class encoding breaks reflection, structural types Erik Osheim 9/19/12 7:57 AM
On Wed, Sep 19, 2012 at 11:20:08AM +0200, martin odersky wrote:
> I now also think that the "partial unboxing" which was done to allow
> overloading in value classes is more trouble than it's worth.

...

> With complete unboxing both erased signatures of "/" would be
> (Double)Double, so we'd get a clash. With partial unboxing, "Meter" in the
> value class itself would not be erased.
>
> However, this irregularity leads to surprising bridge methods which are not
> working correctly. That was Pavel's example (which I append below).
>
> Now, I would argue that the scheme is too limiting anyway, and that there
> exist more robust ways to avoid name clashes.

...

> So, I am for removing this special case.
>
> What do people think?

+1

Given that value classes currently exist primarily to avoid boxing I
support removing this feature, especially since it seems to add a lot
of complexity to the implementation. Users of value classes shouldn't
expect them to work like normal types in these cases.

-- Erik
Re: SIP-15 Value class encoding breaks reflection, structural types Martin 9/19/12 10:15 AM
I have a new pull-request 


that eliminates halfboxing. Several reported bugs disappeared: Mark's last variant of 3667, 3685 as well as Pavel's counter-examples on this list. So this looks like a definite win. 

With that pull-request we are currently good again: I see no more outstanding critical bugs against value classes, except that we have to fix the Java generic signatures. That should also be simpler now because erasure got more regular.

More tests are very much appreciated.

Cheers

 - Martin
--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

Re: SIP-15 Value class encoding breaks reflection, structural types Paul Phillips 9/19/12 10:24 AM
On Wed, Sep 19, 2012 at 10:15 AM, martin odersky <martin....@epfl.ch> wrote:
> With that pull-request we are currently good again: I see no more
> outstanding critical bugs against value classes, except that we have to fix
> the Java generic signatures. That should also be simpler now because erasure
> got more regular.

I opened that yesterday, but it probably is not done.

  https://github.com/scala/scala/pull/1345
Re: SIP-15 Value class encoding breaks reflection, structural types Mark Harrah 9/19/12 4:50 PM
On Wed, 19 Sep 2012 19:15:29 +0200
martin odersky <martin....@epfl.ch> wrote:

> I have a new pull-request
>
>   https://github.com/scala/scala/pull/1352
>
> that eliminates halfboxing. Several reported bugs disappeared: Mark's last
> variant of 3667, 3685 as well as Pavel's counter-examples on this list. So
> this looks like a definite win.
>
> With that pull-request we are currently good again: I see no more
> outstanding critical bugs against value classes, except that we have to fix
> the Java generic signatures. That should also be simpler now because
> erasure got more regular.

I think this is correct, except I believe the lazy val problem is still open:

 https://issues.scala-lang.org/browse/SI-6358

> More tests are very much appreciated.

I've posted a draft for the value class guide as an attachment here:

 https://issues.scala-lang.org/browse/SI-6398

I spent a fair amount of space on concrete examples of limitations as well as when allocations occur.  I'd appreciate corrections for the allocations section, additions to the introduction, additional examples of limitations, or just pointing out important omissions to the list.  I considered adding a section explicitly listing features they should interact ok with, like type parameters, case classes, and implicit classes.  Would that be useful?

In the process of writing the guide, I found:

 https://issues.scala-lang.org/browse/SI-6408

I mention that the SIP is the definitive reference for implementation details, but the SIP is actually rather out of date.

Finally, I think there needs to be an annotation @allocationBehavior(silent | warn | error) for value classes and maybe a @allowAllocation to override warnings/errors at a usage site.  It is difficult to know when allocations occur from first principles or just reading the code and javap is really the only reliable way.

-Mark
> Prof., EPFL <http://www.epfl.ch> and Chairman, Typesafe<http://www.typesafe.com>
More topics »