Scala anomalies noticed while writing a unit test for type tags

[Ben Hutchison]

While writing a unit test for the recently discussed type tags change [https://github.com/benhutchison/shapeless/commit/1ce29c9e3af865135155028308141ac12103b7d5] I noticed two independent anomalies. 

Would be interested to hear opinion on whether they're (a) misunderstanding on my part, (b) design limitations of Scala, or (c) bugs.

Case 1: type "A with B" not viewable as B

The goal is to implicitly convert "Int with Tagged[ATag]" to "String".

Code:

implicit def taggedToString(value: Tagged[ATag]): String = message

[error] Test shapeless.TypeOperatorTests.testImplicitScopeForTaggedType failed:

java.lang.Integer cannot be cast to shapeless.TypeOperators$Tagged

[error]     at shapeless.TypeOperatorTests.testImplicitScopeForTaggedType(typeop

erators.scala:29)

Code:

implicit def taggedToString[T](value: T with Tagged[ATag]): String = message

No compiler error.

Anomaly: I'd expect "Int with Tagged[ATag]" to be viewable as type "Tagged[ATag]", but somehow the compiler seems to have lost the "with Tagged[ATag] part of the type since it's trying to convert Integer."

Case 2: Introducing intermediate val resolves compiler error

Code:

val s: String = tag[ATag](1)

[error]  found   : Int(1)

[error]  required: String

[error]     val s: String = tag[ATag](1)

Code:

val x = tag[ATag](1)

val s: String = x

No compiler error.

Anomaly: I would expect pulling sub-expressions into vals to be side-effect free. But that seems not the case here?

-Ben

[Paul Phillips]

On Sat, Feb 16, 2013 at 5:28 PM, Ben Hutchison <brhut...@gmail.com> wrote:

While writing a unit test for the recently discussed type tags change [https://github.com/benhutchison/shapeless/commit/1ce29c9e3af865135155028308141ac12103b7d5] I noticed two independent anomalies. 

Would be interested to hear opinion on whether they're (a) misunderstanding on my part, (b) design limitations of Scala, or (c) bugs.

Sometimes I wonder whether people fully understand what a miracle it is that "tagged types" work at all. Even the parts of scala into which people have put design and implementation effort aren't going to win any robustness prizes at the county fair. In the case of tagged types... are there are any tests for it? Is the behavior specified somewhere? Is there someone for whom it is a priority? These are real questions - maybe there's a little universe going which I don't see. But if there isn't, then it works mostly by accident and could stop working at any time. Contribute some test cases if you want to protect however much does work, because without that, the sword of damocles must already be falling.

For sure it's completely unsurprising that types like "Int with Foo" have poor internal fidelity.

[Ben Hutchison]

Hi Paul,

I'll try to reply as to the usefulness and relevance of tagged type in a separate email (IMO there's a interesting story there), but the more pressing point is simply:  neither of the reported anomalies has anything particularly to do with tagged types.

1. The "A with B" ought to be viewable as B seems to be specified on page 33 of the 2.9.2 Scala Spec:

"A compound type T 1 with ... with T n {R } conforms to each of its component types T i "

2. The nearest to specifying that naming an expression with a "val" should not affect it's type is on page 38:

"A value declaration val x: T introduces x as a name of a value of type T .., the type T may be omitted,

in which case the packed type (§6.1) of expression e is assumed. "

I expect you're more familiar with these rules than I. My main point is that, regardless of whether type tags are important to the wider Scala community, the issues I raised are relevant in broader contexts, and should be.

-Ben

--
You received this message because you are subscribed to the Google Groups "Shapeless Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shapeless-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Paul Phillips]

On Sat, Feb 16, 2013 at 7:44 PM, Ben Hutchison <brhut...@gmail.com> wrote:

1. The "A with B" ought to be viewable as B seems to be specified on page 33 of the 2.9.2 Scala Spec:

I didn't mean to suggest either that they should not work as you envision or that the issue is not of interest. I was attempting to make an observation about the implementation: the aspects of scala which one can describe as "working" without laughing are the parts which have been the specific target of implementation effort, in particular the ironing out of the interactions with other features.

Since I have never witnessed such effort being put into tagged types, and the reality of intersection and abstract types vs. boxing and erasure is a hornet's nest of implementation complexity and bugs, the only outcome which can surprise me is when it works.

[Ben Hutchison]

On Sun, Feb 17, 2013 at 1:22 PM, Paul Phillips <pa...@improving.org> wrote:

Sometimes I wonder whether people fully understand what a miracle it is that "tagged types" work at all. Even the parts of scala into which people have put design and implementation effort aren't going to win any robustness prizes at the county fair. In the case of tagged types... are there are any tests for it? Is the behaviour specified somewhere? Is there someone for whom it is a priority? These are real questions - maybe there's a little universe going which I don't see.

Undoubtedly, tagged types are a "little universe" right now. Perhaps 95% of Scala programmers are unaware of their existence. Here's how they ended up mattering alot to me...

Scala's something of a "broad church"; one partition is to divide it into a majority group for whom the main way to achieving data abstraction is by object-oriented techniques, i.e. dispatching on the run-time class, and a significant minority who see type class-based dispatch using implicits as the main game, and object-orientation as part of the legacy of Scala's origins and context(*).

As a Java guy, I started in the OO mainstream, but somewhere along the journey an impression that type classes are superior became overwhelming, and I stopped believing in OO. So my future, as I see it, lies in learning how to effectively re-purpose Scala as a type-class based language, and learning from other people who have made a similar realisation. 

Type classes rely much more heavily on the type-system, and associating rich types with data is crucial to get dispatch to work right. My recent investigation of tagged types has been motivated by similar goals as the Value Classes SIP; that is, to get richer behaviour and polymorphism over primitive data types. Lets take stock of how the 2 approaches are going:

Tagged Primitive Types + Typeclasses:

- Can be stored unboxed in Arrays [caveat: https://issues.scala-lang.org/browse/SI-7088]

- Can be processed without boxing

- Easy to reuse and compose operations over them without boxing effects

- Took a few people a few months to invent them by composing existing Scala features

- One major limitation is collision with innate operations defined over the native primitive type, which cannot be re-declared -  exactly the problems of OO that type classes are trying to get away from.

Value Classes:

- Cannot be stored unboxed in arrays

- Cannot reuse implementation via traits without causing boxing

- Took a team a year to invent, compiler level changes needed, lots of implementation difficulties along the way, current result a compromise vs what was hoped for.

- Undoubtedly useful for optimising implicit wrappers. Further work seems needed, to be useful beyond that.

-Ben

(*) I'm interested Scala with type-classes, over Haskell, because:

- I want to be on the JVM and Java-compatible

- I want a strict not lazy language

- Good IDE tooling in the Java tradition matters to me

[Paul Phillips]

On Sat, Feb 16, 2013 at 8:51 PM, Ben Hutchison <brhut...@gmail.com> wrote:

Undoubtedly, tagged types are a "little universe" right now.

I meant, is there someone committing compiler code for whom they are a priority. 

Value Classes:

- Cannot be stored unboxed in arrays

- Cannot reuse implementation via traits without causing boxing

- Took a team a year to invent, compiler level changes needed, lots of implementation difficulties along the way, current result a compromise vs what was hoped for.

- Undoubtedly useful for optimising implicit wrappers. Further work seems needed, to be useful beyond that.

I don't remember the team or the year spent inventing them, but I miss a lot of things. I promise you too will need compiler level changes (which will come with implementation difficulties, because they all do) unless you are satisfied with tagged types as they are.

I know they're useful beyond implicit wrappers already, unless these were pointless exercises. I didn't think so.

  https://github.com/scala/scala/blob/master/src/reflect/scala/reflect/internal/Variance.scala

  https://github.com/scala/scala/blob/master/src/reflect/scala/reflect/internal/Mode.scala

Can you pattern match your way back to "Array[Int] with Foo" if given an array in an Any? That's why value classes are as they are. Maybe that's not important to you, but it was to martin.

[Ben Hutchison]

On Sun, Feb 17, 2013 at 5:31 PM, Paul Phillips <pa...@improving.org> wrote:

On Sat, Feb 16, 2013 at 8:51 PM, Ben Hutchison <brhut...@gmail.com> wrote:

Undoubtedly, tagged types are a "little universe" right now.

I meant, is there someone committing compiler code for whom they are a priority. 

As a user of the scala compiler technology, what I can do it argue why they ought to be. And it's for the reasons I mentioned in the last email - combining tagging primitives with type info allows type classes to selectively bind to them. That in turn enables functionality similar value classes, without boxing in arrays, and in way that appears far simpler to me.

 

Value Classes:

- Cannot be stored unboxed in arrays

- Cannot reuse implementation via traits without causing boxing

- Took a team a year to invent, compiler level changes needed, lots of implementation difficulties along the way, current result a compromise vs what was hoped for.

- Undoubtedly useful for optimising implicit wrappers. Further work seems needed, to be useful beyond that.

I don't remember the team or the year spent inventing them, but I miss a lot of things.

Ok that was a glib comment and is probably inaccurate. It was based on observing discussions on the mailing lists through 2012, which certainly gives the impression that alot of time & effort was put into the value classes feature.

 

I promise you too will need compiler level changes (which will come with implementation difficulties, because they all do) unless you are satisfied with tagged types as they are.

I believe the work needed is alot less than making unboxed primitives that resemble objects, with an accurate runtime type. The type class approach just sets the bar lower, and so has an easier time:all the dispatch is compile time, so it doesn't need to deal with runtime JVM constraints.

Specifically, SI-7088 is needed for tagged primitives to get off the ground. That'd enable array ops on tagged types that extend existing primitive types. So typeclasses can attach methods to primitives, so long as they avoid collision with existing operators like '+'.

Nicer yet is some kind of fully unboxed newtype, at compile-time only. A way to tell the compiler - don't route to the usual + method for Int etc, on values ascribed a certain type. I admit I'm not a compiler expert, but stuff that operates purely in compile-time types has surely got to be easier to build than features that have a runtime aspect you can reflect on.

 

I know they're useful beyond implicit wrappers already, unless these were pointless exercises. I didn't think so.

  https://github.com/scala/scala/blob/master/src/reflect/scala/reflect/internal/Variance.scala

  https://github.com/scala/scala/blob/master/src/reflect/scala/reflect/internal/Mode.scala

Agreed, this is useful. My comment was intended to refer to the limits around reuse and abstraction over value classes, when inheriting from traits triggers boxing.

Can you pattern match your way back to "Array[Int] with Foo" if given an array in an Any? That's why value classes are as they are. Maybe that's not important to you, but it was to martin.

That's it, it's a question of what's important. Unfortunately, value classes as designed gives me a feature I don't use (accurate runtime type) at the price of omitting a feature I would love, namely arrays of primitives with rich types and rich, re-useable behaviour.

My perception of why this was important was to ensure value class instances behaved like good object-oriented citizens, eg accurate runtime type (at the top level - we already tolerate erasure, after all). The commitment to Scala being uniformly object-oriented is closing off fruitful design pathways.

-Ben

[Paul Phillips]

On Sun, Feb 17, 2013 at 3:08 AM, Ben Hutchison <brhut...@gmail.com> wrote:

As a user of the scala compiler technology, what I can do it argue why they ought to be. And it's for the reasons I mentioned in the last email - combining tagging primitives with type info allows type classes to selectively bind to them. That in turn enables functionality similar value classes, without boxing in arrays, and in way that appears far simpler to me.

Since I am sure this sequence sends a confusing message, let me recap: my initial question was trying to establish whether there is someone who is trying to make "tagged types" work properly. It seems, as I anticipated, that there is not. This does not suggest that there should not be, nor that only the opinions of compiler committers are relevant. The "little universe I don't see" referred specifically to a little universe where someone is working to improve tagged types within the compiler, not to a much larger little universe where people use tagged types.

I'm making an empirical observation about what can be expected to work, not a value judgment.

Ok that was a glib comment and is probably inaccurate. It was based on observing discussions on the mailing lists through 2012, which certainly gives the impression that alot of time & effort was put into the value classes feature.

In those same discussions you can see the desirability of what you want being acknowledged. You can see martin explicitly contrasting them here:

  https://groups.google.com/forum/?fromgroups=#!topic/scala-sips/agxDXHCEsxA

Including leaving newtype open as a future possibility.

 

Specifically, SI-7088 is needed for tagged primitives to get off the ground. That'd enable array ops on tagged types that extend existing primitive types. So typeclasses can attach methods to primitives, so long as they avoid collision with existing operators like '+'.

I looked at this when it was opened; it appears strangely difficult to fix.

Agreed, this is useful. My comment was intended to refer to the limits around reuse and abstraction over value classes, when inheriting from traits triggers boxing.

We expect to come back to that.

[Miles Sabin]

On Sat, Feb 16, 2013 at 11:31 PM, Paul Phillips <pa...@improving.org> wrote:
> On Sat, Feb 16, 2013 at 8:51 PM, Ben Hutchison <brhut...@gmail.com> wrote:
>>
>> Undoubtedly, tagged types are a "little universe" right now.
>
> I meant, is there someone committing compiler code for whom they are a
> priority.

I'd have thought that Jason was at least vaguely interested in their
well-being, seeing as he sprinkled them liberally over Scalaz 7 ;-)

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
skype: milessabin
gtalk: mi...@milessabin.com
g+: http://www.milessabin.com
http://twitter.com/milessabin