Reified Generics on the JVM

[Stephane Epardaud]

[As opposed to for JavaScript which was already done]

I just merged the longest-standing branch in history and we have reified generics support for Ceylon on the JVM :) Yay! Now let's talk about what happened:

- All the checks in the typechecker that bitched about non-reified generics got commented out

- Every constructor/method that has type parameters now accepts the same number of extra method parameters that represent the generics at runtime, of the form: @Ignore TypeDescriptor $reifiedT

- Every call to a type-parameterised constructor/method now passes reified generics information (unless the constructor/method in question does not support reified generics, such as java methods), most of the time by passing a TypeDescriptor instance that we already have: "void foo<T>(){ bar<T>(); }", or by using constant TypeDescriptors: "bar<Integer>();" (one such constant is created per non-type-parameterised Ceylon type, plus one for the Nothing type), or by pretty cheap/fast construction at runtime: "bar<Integer&T>();" or "bar<List<T>>();"

- I intend to create constants for constant TypeDescriptors such as "List<Integer|Character>", as an additional optimisation

- These TypeDescriptor objects are created by static factory methods in TypeDescriptor and are cheap/lightweight and might be cacheable in the future.

- I intend to add a runtime option to disable reified generics, if only for timing its costs by returning null to all TypeDescriptor factory methods.

- When we encounter an "is" operator that requires reified generics "is T" or "is List<Integer>", we replace it with a "Util.isReified(val, TypeDescriptor)" call that is expensive ATM: it transforms the TypeDescriptor into a ProducedType using the typechecker and reflection model loader, and delegates the complex operation to "ProducedType.isSubtypeOf(ProducedType)"

- The previous point means that we are now embarking the typechecker, java compiler, common and CMR at runtime, in order to build a typechecker model of the Ceylon modules at runtime

- This kicks ass because it means we have most of the pieces ready to implement the metadata

- This is great because it means we have robust well-tested technology to implement the metadata at runtime, rather than reimplement it

- This also means we have metadata at runtime for Java modules

- In the future we might (or not) be able to speed up the metadata for Ceylon modules by generating static code to instantiate it, but that will be less tested and non-lazy so potentially slower than what we have ATM.

- Because we now require Ceylon modules to have a model at runtime, we need the systems invoking the Ceylon code to tell us which Ceylon modules are installed and running and where their car/jar files are, as well as their dependencies in the case of Maven modules (we can't figure that out from within the language module, or rather that'd be a duplicate of the work done by the system invoking the Ceylon code so it'd be pretty stupid)

- The systems invoking the Ceylon code now need to call "Util.loadModule" for each Ceylon module required at runtime, which is done automatically by the ceylon-runtime system, semi-automatically by the compiler runtime tests, and automatically via a crass temporary hack by the Ceylon IDE (that will go away as soon as it uses the ceylon-runtime system)

- The runtime module system is cheap to initialise, as it only loads the package/module descriptors on init, the rest being only loaded when we use reified generics with "is"

- The runtime module system currently logs a few things I intend to remove ASAP but it also helps in noticing buggy behaviour: I've fixed quite a number of bugs by noticing nonsensical logs in the past days, so bear with them and report nonsence ;)

- There are various open issues remaining that I'll log in github.

--
Stéphane Épardaud

[Stephane Epardaud]

Oh, and the ceylon.language build now uses the ceylon-run ant task to run the tests.

--
Stéphane Épardaud

[Stephane Epardaud]

If you're curious about the Ceylon IDE hack, that's because I can't call Util.loadModule from the code that executes the Ceylon code in the Eclipse launcher because it's running in a different JVM. So I generate a "module descriptor" temporary file and invoke a Runner in the langauge module which loads this file, calls all the Util.loadModule based on this info, and then invokes the Ceylon program using reflection. And removes the temp file.

--
Stéphane Épardaud

[Gavin King]

Congratulations!

> --
> You received this message because you are subscribed to the Google Groups
> "ceylon-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ceylon-dev+...@googlegroups.com.
> To post to this group, send email to ceylo...@googlegroups.com.
> Visit this group at http://groups.google.com/group/ceylon-dev?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Gavin King
gavin...@gmail.com
http://profiles.google.com/gavin.king
http://ceylon-lang.org
http://hibernate.org
http://seamframework.org

[Tako Schotanus]

Yeah, great job Stef!

-Tako

[Stephane Epardaud]

Update: "is Foo<Anything>" is now optimised to a simple raw "instanceof".

Stéphane Épardaud

[Gavin King]

I hope you only do that when Foo is covariant, right?

On Fri, Feb 22, 2013 at 5:17 PM, Stephane Epardaud

[Stephane Epardaud]

Pffffffffff… I guess I will, you spoiler ;)

[Ross Tate]

There are a lot more optimizations we can do. The Scala study found that most casts can be resolved using just instanceof without reification.

[Stephane Epardaud]

What Scala study?

[Vlad Ureche]

Hi everyone,

I came across this announcement because someone pointed it out on the scala-internals mailing list.

First of all congratulations, that's a pretty radical change!

Second thing, I'm really curious about the performance impact. I imagine such a change couldn't go without any impact, right?

Thanks,
Vlad

[Stephane Epardaud]

There is an impact, we haven't measured it yet, but we've done good work to keep it minimal, and we've plenty of ideas to make it even faster later on. At some point you have to accept that there's no magic road that will give you reified generics for free. Not even if the JVM implemented it. In the end we're still talking about extra parameters to pass around. We can optimise that a lot, to a point where it's an acceptable compropose for a feature we want. Just like closures: closures are horribly expensive, but if you want it you pay the cost and try to drive that cost to a minimum. Everything has a price, and reified generics have value, just like closures, so we're willing to pay that price.

--

You received this message because you are subscribed to the Google Groups "ceylon-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceylon-dev+...@googlegroups.com.
To post to this group, send email to ceylo...@googlegroups.com.
Visit this group at http://groups.google.com/group/ceylon-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Stéphane Épardaud

[Ross Tate]

Here's some of our earlier discussion on this topic (and the study/thesis): https://groups.google.com/forum/#!topic/ceylon-dev/eQ0wQKmDsCE/discussion

Unfortunately, they only mention this "statistic" anecdotally in 6.2.3: "almost all of these membership tests could actually be optimised and performed [with] only the erased types". Nonetheless, even with that optimization the overhead of reification seemed to be rather high for them.

[Stephane Epardaud]

This thesis seems to date from 2005 and the benchs were done with Java 1.4, so I wonder if they are at all relevant with today's 1.7 JVM. Well, only one way to find out ;)

[Gavin King]

Also their implementation may have just sucked ;-)

I agree, we need to actually measure things for ourselves to know.

On Fri, Feb 22, 2013 at 9:13 PM, Stephane Epardaud

[Ross Tate]

Nonetheless, the anecdote about casts rarely needing reification is implementation independent (or even rarer).

>>> Visit this group at http://groups.google.com/group/ceylon-dev?hl=en--

Gavin King
gavin...@gmail.com
http://profiles.google.com/gavin.king
http://ceylon-lang.org
http://hibernate.org
http://seamframework.org