Manifests and macros
Grzegorz Kossakowski
Grzegorz Kossakowski
Eugene Burmako
Grzegorz Kossakowski
Hi Greg,In 2.10 we plan to introduce typetags that will replace manfests. Typetags use reification, so that they can capture everything that can be recorded in pickles (all kinds of types and trees, except for local variables and local classes), as opposed to the current hacky state of things.
It works as follows. Whenever an implicit typetag is required, but is not in scope, implicit search emits a macro that reifies the type and wraps the reification in a typetag. Depending on the flavor of the tag you required, at runtime you will get either a classtag (that carries an erasue) or a typetag (that carries a full-blown scala.reflect.mirror.Type).Since we wouldn't like to have two competing approaches to type reification, we're going not only to deprecate manifests, but also alias them to corresponding typetags (ClassManifest -> ClassTag, Manifest -> GroundTypeTag, NoManifest -> discontinued, aliased to Object).
Bad news:1) <:<, >:> and typeArguments won't work for ClassManifests anymore (will be deprecated and will throw UnsupportedOperationException).
2) All factory methods in Manifest and ClassManifest objects will be removed (they were marked as internal, so they won't go through the deprecation cycle).
3) Manifests (not ClassManifests, only Manifests) are going to be tied to reflection.Good news:1) If you use ClassManifests only to create arrays and not for introspection purposes, you're good (otherwise, you will have replace them with full manifests aka GroundTypeTags, which support <:<, >:>, typeArguments and basically everything that is supported by Types from the compiler).2) If you use Manifests, everything should work as usual, granted your platform supports reflection (also see the discussion below).3) If you want to deconstruct manifests and emit equivalent code for another platform, you won't have to deal with current manifest cruft - you will have reified types that correspond to the ones the compiler uses.
I don't know much about Scala+GWT, but I imagine that that bad news #3 would be a bummer. However, we have a solution for that. Current plan is to introduce a lightweight reflective mirror that would work on the platforms that lack reflection. I suggest we discuss the required feature set.
--
Grzegorz Kossakowski
Eugene Burmako
Grzegorz Kossakowski
I have this crazy idea. Why won't you write a macro that generates manifests if your liking and put it in classpath before scala-library? I did that trick once to test reification, and it worked flawlessly.
Grzegorz Kossakowski
Eugene Burmako
iulian dragos
Well, obviously generating what I want in the first place instead of unrolling what default implementation does is a better solution. Since I'm rewriting the whole library for GWT purposes anyway it would be trivial for me to replace default macro with my own. Sounds like a really good solution!On 5 April 2012 17:25, Eugene Burmako <eugene....@epfl.ch> wrote:I have this crazy idea. Why won't you write a macro that generates manifests if your liking and put it in classpath before scala-library? I did that trick once to test reification, and it worked flawlessly.When it comes to your proposal I think that is fairly hard. I believe that most tools do not give you precise control over order on classpath so you shouldn't rely on this. For example, Scala IDE puts library into bootclasspath and it's fairly hard to put something in front of it.
Other example is sbteclipse plugin which generates Eclipse project for your sbt project. It also generates classpath files. AFAIK you have no control over the order of entries put into that file.In short: it's a clever trick but I doubt you can convert it into a best practice :-)--
Grzegorz Kossakowski
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais
Paul Phillips
> 1) <:<, >:> and typeArguments won't work for ClassManifests anymore (will be
> deprecated and will throw UnsupportedOperationException).
There's no reason to make <:< on ClassManifest throw an exception,
certainly not in 2.10. That is the standard way of checking
conformance within the level of granularity offered. Surely ClassTags
offer some means of doing a subclass check.
If we think those should be separate operations, fine, but then we
have to transition to that; I prefer not to aggressively break
existing code which isn't even using any deprecated or commented as
"don't use this" constructs.
Josh Suereth
Eugene Burmako
1) Previously ClassManifests captured type arguments. Now they do not.
Hence the results of <:< and >:> might be skewed. typeArguments would
be plain wrong in all the interesting cases.
2) AnyValManifest. Is anyone supposed to use it?
3) Manifests for core types are singletons, with identities preserved
across construction and deserialization. Typetags are simple case
classes, so I could create two different typetags for, say, IntTpe.
They would be equal, but not reference equal. Is this something we're
okay with?
4) Can we remove OptManifest and NoManifest? Would be pretty hard to
deprecate them without breaking the semantics.
5) What should the following snippets do (no manifest/typetag context
bounds, that's not a typo):
def foo[T] = Array[T]()
def foo[T <: Int] = Array[T]()
def foo[T] = Array[Array[T]]()
def foo[T] = Array[List[T]]()
On Apr 9, 5:12 pm, Paul Phillips <pa...@improving.org> wrote:
Vlad Ureche
4) Can we remove OptManifest and NoManifest? Would be pretty hard to
deprecate them without breaking the semantics.
Is there a major difficulty in porting OptManifests over to TypeTags? Or why do you plan to deprecate them?
Cheers,
Vlad
Eugene Burmako
Paul Phillips
<grzegorz.k...@gmail.com> wrote:
> Well, let me explain why I needed to mess with Manifests. Since Scala+GWT
> doesn't support any kind of reflection then I didn't bother about supporting
> methods like <:< but Manifests are crucial for generic Array support in
> Scala collections so I absolutely needed to find some solution for this
> particular problem. Internally, Manifests use Java reflection for Array
> allocation (they are factories for arrays).
That sucks. What do you think about this, which I see no reason not
to do. Does it improve the situation? There's no reason you should
have to think about anything but array creation when all the compiler
needs is array creation.
http://github.com/paulp/scala/branches/topic/array-manifest
/** An interface for array creation. Although ClassManifest[T]
* is the usual way to provide ArrayManifest[T], this is as much
* as is necessary to inform the compiler that Arrays of element
* type T can be created.
*
* TODO - further separate this so def newArray can be impelmented
* and taken advantage of without mandating the multidimensional methods.
*/
trait ArrayManifest[T] {
def newArray(len: Int): Array[T]
def newArray2(len: Int): Array[Array[T]]
def newArray3(len: Int): Array[Array[Array[T]]]
def newArray4(len: Int): Array[Array[Array[Array[T]]]]
def newArray5(len: Int): Array[Array[Array[Array[Array[T]]]]]
}
Eugene Burmako
Paul Phillips
> 1) Previously ClassManifests captured type arguments. Now they do not.
> Hence the results of <:< and >:> might be skewed. typeArguments would
> be plain wrong in all the interesting cases.
It was a bug that they ever captured type arguments. They should be
ignored in the <:< performed at the ClassManifest or equivalent level.
It's just a subclass check.
> 2) AnyValManifest. Is anyone supposed to use it?
Not exactly, but if you're wondering why it's not sealed:
commit cebc49b9f37ec8c8a3befcee6142f9c0590c41e2 (refs/pull/104/head)
Author: Grzegorz Kossakowski <grzegorz.k...@gmail.com>
Date: 3 months ago
Make AnyValManifest not a sealed class.
Scala+GWT has a whole new hierarchy of Manifests that
does not use reflection. Every type in new hierarchy
is a subtype of a type from old hierarchy. Sealed modifier
introduced in 2e92de4cd66532404081eec6b9e82c6f85b51434
breaks this scheme. Removing it so Scala+GWT can compile again.
> 3) Manifests for core types are singletons, with identities preserved
> across construction and deserialization. Typetags are simple case
> classes, so I could create two different typetags for, say, IntTpe.
> They would be equal, but not reference equal. Is this something we're
> okay with?
It seems a desirable property that there be only one "Int" type and so
on. If we can preserve that property I think we should.
> 4) Can we remove OptManifest and NoManifest? Would be pretty hard to
> deprecate them without breaking the semantics.
Why is that? Variance? It'd too bad there's no easy type system escape
so you can have an invariant type and a "None"-like error object which
conforms to it. We can adjust the semantics if we must (and I don't
think there was much to rely upon in them) but we need to support the
existing source constructs until they have been deprecated.
> 5) What should the following snippets do (no manifest/typetag context
> bounds, that's not a typo):
>
> def foo[T] = Array[T]()
Fail.
> def foo[T <: Int] = Array[T]()
Fail, I don't see motivation to make this succeed.
> def foo[T] = Array[Array[T]]()
Fail.
> def foo[T] = Array[List[T]]()
Succeed with warning?
Paul Phillips
> What's the supposed difference between ClassManifest and ArrayManifest?
def erasure: jClass[_]
lazy val tpe: mirror.Type = mirror.classToType(erasure)
def symbol: mirror.Symbol = mirror.classToSymbol(erasure)
def <:<(that: ClassManifest[_]): Boolean
def >:>(that: ClassManifest[_]): Boolean
def arrayManifest: ClassManifest[Array[T]]
def newArray(len: Int): Array[T]
def newArray2(len: Int): Array[Array[T]]
def newArray3(len: Int): Array[Array[Array[T]]]
def newArray4(len: Int): Array[Array[Array[Array[T]]]]
def newArray5(len: Int): Array[Array[Array[Array[Array[T]]]]]
def newWrappedArray(len: Int): WrappedArray[T]
def newArrayBuilder(): ArrayBuilder[T]
def typeArguments: List[OptManifest[_]]
That's what's in ClassManifest. What does the compiler actually need?
newArray, and (optionally in some hypothetical future) newArray[2-5].
> If it's about ArrayManifests being solely for creating arrays, then what's the
> purpose of ClassManifest?
Everything else. If you mean "purpose" as contrasted with TypeTags,
then there's at least the purpose of backward compatibility.
> https://github.com/scalamacros/kepler/blob/a2d80afa8bbe58edb6af88a6f8d8dbe70e7349c6/src/library/scala/reflect/TypeTags.scala
>
> Does this fit your idea of a ClassManifest done right?
That's a big improvement, but a "ClassManifest done right" doesn't
remove the need for an ArrayManifest. Arrays are special because the
compiler doesn't know how to create them without help. There is no
reason to bundle a bunch of other concerns and a whole model of the
type system with that. The missing piece is "T => Array[T]" for any
T, and someone in J Random environment looking to use scala should not
have to look beyond the delivery of T => Array[T].
martin odersky
On Thu, Apr 5, 2012 at 8:01 AM, Eugene Burmako <eugene....@epfl.ch> wrote:There's no reason to make <:< on ClassManifest throw an exception,
> 1) <:<, >:> and typeArguments won't work for ClassManifests anymore (will be
> deprecated and will throw UnsupportedOperationException).
certainly not in 2.10. That is the standard way of checking
conformance within the level of granularity offered. Surely ClassTags
offer some means of doing a subclass check.
Eugene Burmako
Eugene Burmako
Eugene Burmako
martin odersky
Thanks for the comments. Then I have some more questions
1) Previously ClassManifests captured type arguments. Now they do not.
Hence the results of <:< and >:> might be skewed. typeArguments would
be plain wrong in all the interesting cases.
he break
2) AnyValManifest. Is anyone supposed to use it?
3) Manifests for core types are singletons, with identities preserved
across construction and deserialization. Typetags are simple case
classes, so I could create two different typetags for, say, IntTpe.
They would be equal, but not reference equal. Is this something we're
okay with?
4) Can we remove OptManifest and NoManifest? Would be pretty hard to
deprecate them without breaking the semantics.
5) What should the following snippets do (no manifest/typetag context
bounds, that's not a typo):
def foo[T] = Array[T]()
def foo[T <: Int] = Array[T]()
def foo[T] = Array[Array[T]]()
def foo[T] = Array[List[T]]()
On Apr 9, 5:12 pm, Paul Phillips <pa...@improving.org> wrote:
> On Thu, Apr 5, 2012 at 8:01 AM, Eugene Burmako <eugene.burm...@epfl.ch> wrote:
> > 1) <:<, >:> and typeArguments won't work for ClassManifests anymore (will be
> > deprecated and will throw UnsupportedOperationException).
>
> There's no reason to make <:< on ClassManifest throw an exception,
> certainly not in 2.10. That is the standard way of checking
> conformance within the level of granularity offered. Surely ClassTags
> offer some means of doing a subclass check.
>
> If we think those should be separate operations, fine, but then we
> have to transition to that; I prefer not to aggressively break
> existing code which isn't even using any deprecated or commented as
> "don't use this" constructs.
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967
martin odersky
martin odersky
On Mon, Apr 9, 2012 at 8:23 AM, Eugene Burmako <eugene....@epfl.ch> wrote:
Thanks for the comments. Then I have some more questionshe break
1) Previously ClassManifests captured type arguments. Now they do not.
Hence the results of <:< and >:> might be skewed. typeArguments would
be plain wrong in all the interesting cases.I believe typeArguments were mostly ignored, so I think we can live with the breakage. If not, we could (a) deprecate <:< and add a warning that its behavior has changed (b) introduce a new method isSubClass on ClassTag. That might be the safer way to proceed, so I lean towards that.2) AnyValManifest. Is anyone supposed to use it?
It's easy to generate right? AnyVal.class exists, so just make a ClassManifest from it. We can deprecate it as an explicit member of Manifest, precisely because it is so easy to generate.3) Manifests for core types are singletons, with identities preserved
across construction and deserialization. Typetags are simple case
classes, so I could create two different typetags for, say, IntTpe.
They would be equal, but not reference equal. Is this something we're
okay with?I think it's wise to maintain this. Both for backwards compatibility and for efficiency. It can be achieved by makingClassTag an abstract case class (that way constructor is not created automatically), complemented by a memoizing constructor in the companion object. All reflect.internal.Type subclasses are constructed the same way, so you can check there how it's done.
Eugene Burmako
Paul Phillips
> Well, I would hate to multiply entities beyond necessity. I already feel bad
> about having class tags and type tags, it would be a shame to introduce the
> third flavor of tags.
>
> How about we leave only newArrayXXX and arrayManifest in ClassTag itself,
> and stuff everything else into a helper that is implicitly convertible from
> ClassTag?
This still bundles array creation and a specific model of the type
system far beyond what I think is appropriate.
Failing to properly separate concerns of this kind is how one ends up
with overlapping mechanisms, neither of which subsumes the other and
which don't interoperate, for instance the interaction of
specialization, arrays, and manifests.
I'm not so interested in how the tags are modeled (though a third one
doesn't seem to matter to me, given that ClassTag would extend it and
nobody need use it.) The key is that the compiler contains special
code which translates "new Array" into something else. One should be
able to implement "something else" without taking on unrelated
concerns. Everything but "T => Array[T]" is an unrelated concern.
Paul Phillips
> Succeed. No warning needed because the array can be constructed.
I threw in 'with warning?' because the array can be created but it is
not a faithful representation of the type which was stated. This
difference, I mean. I'm not in favor of warning about it though, I
was just question marking out loud.
scala> val a1 = new Array[String](0)
a1: Array[String] = Array()
scala> val a2 = new Array[List[String]](0)
a2: Array[List[String]] = Array()
scala> a1.isInstanceOf[Array[Int]]
res0: Boolean = false
scala> a2.isInstanceOf[Array[List[Int]]]
warning: there were 1 unchecked warnings; re-run with -unchecked for details
res1: Boolean = true
Grzegorz Kossakowski
concerns. Everything but "T => Array[T]" is an unrelated concern.
Grzegorz Kossakowski
Paul Phillips
<grzegorz.k...@gmail.com> wrote:
> //probably ArrayFactory is a better name
Yeah, at first I called it ArrayCreator but I didn't want to
gratuitously deviate from the existing names. I do not like
"factories" though; would prefer a term without so much negative
baggage. (At least in my eyes, factories too often come from people
whose next offering may be an AbstractFactoryFactoryFactory.)
Vlad Ureche
Currently we have: 1) ClassTags (== ClassManifests), 2) TypeTags (can reify any type), 3) GroundTypeTags (same as TypeTags, but compile error if the type involves abstract type symbols).So currently type tags don't have a notion of optionality. You either require a GroundTypeTag and prepared to face compiler errors, or you ask for a TypeTag and it gets generated in any circumstances, then you can do whatever you wish with that TypeTag (e.g. validate it at runtime and what not).My impression was that that OptManifests exist only because manifests in general are broken and cannot represent certain types. Since the problem is gone, I thought that we no longer need OptManifests. Right?
I'm not sure why OptManifests were created in the first place. What I'm ultimately looking for is an Option[ClassTag[_]], without the Option boilerplate.
Also, could you, please, elaborate on the specialization idea?
Okay, here goes: the specialization phase works by creating monomorphic versions of a polymorphic class for value types (like special casing List[T] for Ints to get rid of boxing). It also takes care of redirecting the `New` tree nodes to the correct monomorphic version of the class, provided the type parameters are known at compile-time. If the type parameters are not known at compile time, the `New` node creates the generic (polymorphic) class (like List[T]).
Now, this does not cover a very useful case: if you instantiate the specialized class from a generic class, you'll always get the polymorphic version that boxes values internally. Even if you have a manifest/classTag in scope. And we can improve it: we can statically determine if classTags for the type parameters exist in scope and instantiate the correct monomorphic version (by dispatching on the type in the classTag).
The opposite is even more interesting: if we have an instance of a specialized class in scope, we should be able to extract the ClassTags for its type parameters. But since the truly polymorphic class may also appear (if there were no manifests in scope at the allocation site), there's a need to model the lack of a ClassTag. I'm not sure what the most elegant implementation is, but OptManifest seemed much prettier than Option[ClassTag[_]] for this. Any suggestions?
Vlad
martin odersky
On Mon, Apr 9, 2012 at 6:16 PM, Eugene Burmako <eugene....@epfl.ch> wrote:Currently we have: 1) ClassTags (== ClassManifests), 2) TypeTags (can reify any type), 3) GroundTypeTags (same as TypeTags, but compile error if the type involves abstract type symbols).So currently type tags don't have a notion of optionality. You either require a GroundTypeTag and prepared to face compiler errors, or you ask for a TypeTag and it gets generated in any circumstances, then you can do whatever you wish with that TypeTag (e.g. validate it at runtime and what not).My impression was that that OptManifests exist only because manifests in general are broken and cannot represent certain types. Since the problem is gone, I thought that we no longer need OptManifests. Right?
I'm not sure why OptManifests were created in the first place. What I'm ultimately looking for is an Option[ClassTag[_]], without the Option boilerplate.
Also, could you, please, elaborate on the specialization idea?
Okay, here goes: the specialization phase works by creating monomorphic versions of a polymorphic class for value types (like special casing List[T] for Ints to get rid of boxing). It also takes care of redirecting the `New` tree nodes to the correct monomorphic version of the class, provided the type parameters are known at compile-time. If the type parameters are not known at compile time, the `New` node creates the generic (polymorphic) class (like List[T]).
Now, this does not cover a very useful case: if you instantiate the specialized class from a generic class, you'll always get the polymorphic version that boxes values internally. Even if you have a manifest/classTag in scope. And we can improve it: we can statically determine if classTags for the type parameters exist in scope and instantiate the correct monomorphic version (by dispatching on the type in the classTag).
The opposite is even more interesting: if we have an instance of a specialized class in scope, we should be able to extract the ClassTags for its type parameters. But since the truly polymorphic class may also appear (if there were no manifests in scope at the allocation site), there's a need to model the lack of a ClassTag. I'm not sure what the most elegant implementation is, but OptManifest seemed much prettier than Option[ClassTag[_]] for this. Any suggestions?
Vlad
Eugene Burmako
Eugene Burmako
Eugene Burmako
address it with implicit macros a bit later.
On Apr 10, 8:56 pm, Eugene Burmako <eugene.burm...@epfl.ch> wrote:
> Thanks for a nice idea, guys! Here you go:
>
>
> https://github.com/scalamacros/kepler/blob/0504f718c9f940f29a5825c83d...
> On 9 April 2012 20:45, Grzegorz Kossakowski
Vlad Ureche
Aha, now I see. Is this an urgent matter, or we can discuss this during/after Scala Days?
Vlad
Paul Phillips
> Currently we have: 1) ClassTags (== ClassManifests), 2) TypeTags (can reify
> any type), 3) GroundTypeTags (same as TypeTags, but compile error if the
> type involves abstract type symbols).
If GroundTypeTags are something we anticipate being exposed to people
other than compiler hackers, can we please give them a name people
which will mean something to regular people? Is "ConcreteTypeTag"
inaccurate in some important way? ResolvedTypeTag? FullyKnownTypeTag?
Really anything is better than "Ground" - I understand that's the term
in the literature, but this is exactly the place where we should apply
our expertise at making things accessible as adeptly as we apply our
other expertises.
Grzegorz Kossakowski
Thanks for a nice idea, guys! Here you go:
--
Grzegorz Kossakowski
martin odersky
Eugene Burmako
Paul Phillips
> by the way, what about the pull request itself?
I've had a long and failure-plagued trip but I'm now in london and
managed 14 hours of sleep so I'll look at it and maybe even understand
it.
Daniel Sobral
> omw. eta ~= 1h (need to rename a lot of stuff and to bootstrap). after I'm
> done, I'll incorporate changes into the pull request.
>
> by the way, what about the pull request itself?
This is one of the most trivial refactoring possible, and it's one of
the few available right now as well. Can't Eclipse's (or ENSIME's)
refactoring do this? If not, feedback on why not would be valuable.
This is really the kind of thing that people coming from Java take for
granted.
>
> On 12 April 2012 06:54, martin odersky <martin....@epfl.ch> wrote:
>>
>>
>>
>> On Tue, Apr 10, 2012 at 2:08 PM, Paul Phillips <pa...@improving.org>
>> wrote:
>>>
>>> On Mon, Apr 9, 2012 at 9:16 AM, Eugene Burmako <eugene....@epfl.ch>
>>> wrote:
>>> > Currently we have: 1) ClassTags (== ClassManifests), 2) TypeTags (can
>>> > reify
>>> > any type), 3) GroundTypeTags (same as TypeTags, but compile error if
>>> > the
>>> > type involves abstract type symbols).
>>>
>>> If GroundTypeTags are something we anticipate being exposed to people
>>> other than compiler hackers, can we please give them a name people
>>> which will mean something to regular people? Is "ConcreteTypeTag"
>>> inaccurate in some important way? ResolvedTypeTag? FullyKnownTypeTag?
>>> Really anything is better than "Ground" - I understand that's the term
>>> in the literature, but this is exactly the place where we should apply
>>> our expertise at making things accessible as adeptly as we apply our
>>> other expertises.
>>
>>
>> OK, ConcreteTypeTag should work. Thanks for the suggestion! Eugene, can
>> you make that change please?
>>
>> Thanks
>>
>> - Martin
>>
>>
>
--
Daniel C. Sobral
I travel to the future all the time.
Mirko Stocker
> > omw. eta ~= 1h (need to rename a lot of stuff and to bootstrap). after I'm
> > done, I'll incorporate changes into the pull request.
> >
> > by the way, what about the pull request itself?
>
> This is one of the most trivial refactoring possible, and it's one of
> the few available right now as well. Can't Eclipse's (or ENSIME's)
> refactoring do this? If not, feedback on why not would be valuable.
Yes, please let me know what's lacking in their refactoring support (Eclipse
and Ensime both use scala-refactoring.org).
Cheers,
Mirko
--
Mirko Stocker | m...@misto.ch
Work: http://ifs.hsr.ch | http://infoq.com
Personal: http://misto.ch | http://twitter.com/m_st