Scala's pluggable typing architecture

[Miguel Garcia]

Hi,

Now that the support for annotations in X10 was mentioned, I was
reminded of Scala's pluggable typing architecture (maybe because I
contribute to its compiler [2])

There are already several success stories leveraging the
infrastructure for type annotations of Scala (and the accompanying
extensible plugin architecture [3]). Some examples:

(a) Compiling generics through user-directed type specialization,
http://infoscience.epfl.ch/record/150134

(b) Capabilities for Uniqueness and Borrowing,
http://infoscience.epfl.ch/record/148219

(c) Implementing First-Class Polymorphic Delimited Continuations by
a Type-Directed Selective CPS-Transform,
http://infoscience.epfl.ch/record/149136

These are success stories already released by the Scala team. I know
there are other efforts around non-nullness, immutability (some
inspired by JSR-308 work!) with which I'm not so familiar, and thus
can't mention details about them. But I'm sure their authors would be
willing to join this discussion in case questions show up :-)


Miguel

--
Miguel Garcia
Swiss Federal Institute of Technology
EPFL - IC - LAMP1 - INR 328 - Station 14
CH-1015 Lausanne - Switzerland
http://lamp.epfl.ch/~magarcia/


[1] Internals of Scala annotations, SID 5 at http://www.scala-lang.org/sids

[2] Scala Compiler Corner, lamp.epfl.ch/~magarcia/
ScalaCompilerCornerReloaded/

[3] Scala Compiler Phase and Plug-In Initialization for Scala 2.8, SID
2 at http://www.scala-lang.org/sids

[Miguel Garcia]

Hi,

I'm no expert about the Checker Framework (although I would like to)
but I've read about the customizable analyses it provides (flow-
sensitive type refinement [1] as well as an infrastructure for
typestate analysis).

Out of the box, the Scala compiler provides functionality on which to
build infrastructure like that above, as an AnnotationChecker:

"An additional checker for annotations on types. Typically these
are registered by compiler plugins with the addAnnotationChecker
method."

Full source comments can be found at:
http://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/src/compiler/scala/tools/nsc/symtab/AnnotationCheckers.scala

Additionally, the Scala compiler provides an iterative dataflow
analysis [2] although in late phase where the program has been reduced
to ICode, a stack-based OO language that is the last shared
representation before one of the backends GenJVM or GenMSIL takes
over. ICode is described in Phd thesis by Iulian Dragos [3]

To have JSR-308 style functionality in Scala, what is needed then is a
Scala compiler plugin [4] that implements an AnnotationChecker, and
works at the AST level earlier in the compilation pipeline. For
reference, that pipeline comprises:

C:\scala28\bin>scalac -Xshow-phases
parser
namer
packageobjects
typer
superaccessors
pickler
refchecks
liftcode
uncurry
tailcalls
specialize
explicitouter
erasure
lazyvals
lambdalift
constructors
flatten
mixin
cleanup
icode
inliner
closelim
dce
jvm
terminal

Resources to get familiar with those phases are linked from the "Scala
Compiler Corner", which used to focus on the JVM backend [5] but now
focuses on the .NET backend [6].

It would be great if the missing pieces were contributed to the Scala
ecosystem to match the ease of use that the JSR-308 Checker Framework
offers.

Miguel


--
Miguel Garcia
Swiss Federal Institute of Technology
EPFL - IC - LAMP1 - INR 328 - Station 14
CH-1015 Lausanne - Switzerland
http://lamp.epfl.ch/~magarcia/

[1] http://types.cs.washington.edu/checker-framework/current/checkers-manual.html#type-refinement

[2] Sec. 2 "Dataflow analyses on ICode" at
http://www.sts.tu-harburg.de/people/mi.garcia/ScalaCompilerCorner/StacklessICode.pdf

[3] http://infoscience.epfl.ch/record/150270

[4] http://www.scala-lang.org/node/140

[5] http://www.sts.tu-harburg.de/people/mi.garcia/ScalaCompilerCorner/

[6] http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

> CH-1015 Lausanne - Switzerlandhttp://lamp.epfl.ch/~magarcia/
>
> [1] Internals of Scala annotations, SID 5 athttp://www.scala-lang.org/sids

[Werner Keil]

Miguel,

The above link to Scala Track is broken.
I'm very interested in this both for Java and Scala. Based on Doug Lea pointing me to that, I forked a 3rd party contribution to a checker for Units of Measurement using JSR 308.

As the contributor stated, he won't be able to do anything further on that, our goal is to explore, how Unit-API (www.unitsofmeasurement.org) could work with 308 and maybe other JVM languages like Scala. The precursor to Unit-API, JSR-275 has already been used successfully by the makers of Groovy ans Grails to create a specialized DSL for Diabetes Research and Treatment.

[Werner Dietl]

Hi Werner,

> The above link to Scala Track is broken.

> I'm very interested in this both for Java and Scala. Based on Doug Lea
> pointing me to that, I forked a 3rd party contribution to a checker for Units of
> Measurement using JSR 308.

Which 3rd party contribution do you mean? Maybe Alex Spoon's?

http://www.lexspoon.org/expannots/

Please do let me know if there is another one.

> As the contributor stated, he won't be able to do anything further on that,
> our goal is to explore, how Unit-API (www.unitsofmeasurement.org) could work
> with 308 and maybe other JVM languages like Scala. The precursor to Unit-API,
> JSR-275 has already been used successfully by the makers of Groovy ans Grails to
> create a specialized DSL for Diabetes Research and Treatment.

I haven't looked at JSR 275 or the Unit-API, thank's for pointing those out.
I've recently written a Unit checker that comes with the Checker Framework:

http://types.cs.washington.edu/checker-framework/current/checkers-manual.html#units-checker

It would be an interesting project to look at how this could be integrated with JSR 275/Unit-API.
Are you interested in using such a checker or in developing one?

As this discussion is about type checkers and not JSR 308 itself,
let us move the discussion either to personal mails or the
Checker Framework discussion list at:

http://groups.google.com/group/checker-framework-discuss

Best regards,
cu, WMD.

[Miguel Garcia]

Werner, 

There are two plugins written by Lex Spoon on Units of Measure: 

  (a) a JSR308 plugin, http://www.lexspoon.org/expannots/

  

  (b) a compiler plugin for Scala. A how-to for it can be found at: 

      http://www.michaelnygard.com/blog/2009/05/units_of_measure_in_scala.html

That compiler plugin is already a few years old, and if you're going to compile it against Scala's trunk, chances are small modifications will be needed. BTW, the best forum for questions about that is: 

  http://groups.google.com/forum/#!forum/scala-internals

For example one of the tweaks that might be needed has to do with the new location of AnnotationCheckers, 

  http://lampsvn.epfl.ch/trac/scala/browser/scala/trunk/src/compiler/scala/reflect/internal/AnnotationCheckers.scala

But in fact the "annotation processing" model in Scala hasn't changed. 

There differences with JSR 308 due to the more granular nature of the Scala compiler as compared to javac (e.g., dataflow can be analyzed on high-level ASTs, on a stackless three-address-like IR, or on a stack based bytecode-like IR). A website covering some compiler phases is: 

  http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/

However, one of the topic missing there is annotation processors. For that, the scala-internals group is your best bet. 

regards, 

[Werner Keil]

Hi Werner,

On Mon, Sep 5, 2011 at 1:01 AM, Werner Dietl <wdi...@yahoo.com> wrote:

Hi Werner,

> The above link to Scala Track is broken.

> I'm very interested in this both for Java and Scala. Based on Doug Lea
> pointing me to that, I forked a 3rd party contribution to a checker for Units of
> Measurement using JSR 308.


Which 3rd party contribution do you mean? Maybe Alex Spoon's?

http://www.lexspoon.org/expannots/

Yes, that's correct. Compared to Unit-API it is missing a few paradigms, especially the notion of "Quantity" vs. "Dimension" which Lex' code has merged into one. It is not so uncommon, e.g. even Kennedy's F# implementation currently seems to follow earlier approaches calling "Length" a "Dimension", but many scientific use cases for multi-dimensional calculations require this term to be dealt with correctly.
 

Please do let me know if there is another one.


> As the contributor stated, he won't be able to do anything further on that,
> our goal is to explore, how Unit-API (www.unitsofmeasurement.org) could work
> with 308 and maybe other JVM languages like Scala. The precursor to Unit-API,
> JSR-275 has already been used successfully by the makers of Groovy ans Grails to
> create a specialized DSL for Diabetes Research and Treatment.


I haven't looked at JSR 275 or the Unit-API, thank's for pointing those out.
I've recently written a Unit checker that comes with the Checker Framework:

http://types.cs.washington.edu/checker-framework/current/checkers-manual.html#units-checker


It would be an interesting project to look at how this could be integrated with JSR 275/Unit-API.
Are you interested in using such a checker or in developing one?

Yes, it looks interesting primarily for a simpler notation similar to your example. Compile time type-safety can already be acomplished via the current Unit-API, but combining it e.g. with other annotation-based frameworks like BeanValidation sound very attractive.
I am actively looking at BeanValidation to handle special units like "Storm category" that are defined by a range rather than a fixed formula. Here adding a constraint sounds like a good idea.
 

As this discussion is about type checkers and not JSR 308 itself,
let us move the discussion either to personal mails or the
Checker Framework discussion list at:

http://groups.google.com/group/checker-framework-discuss


Best regards,
cu, WMD.

Happy to do so. Please also note the following Unit-API discussion lists http://groups.google.com/group/units-dev (probably best for  discussions like these) and http://groups.google.com/group/units-users
(the end user forum, targeted at solutions and developers using it rather than contributing to)

Kind Regards,
--

Werner Keil | UOMo Lead | Eclipse.org

Twitter @wernerkeil | Skype: werner.keil | www.eclipse.org/uomo | #EclipseUOMo

* JavaOne: October 2-6 2011, San Francisco, USA. Werner Keil, Agile Coach, UOMo Lead will co-present "JSR 321: Trusted Java API"

[Werner Keil]

Miguel,

Thanks for the information. I saw that old Scala plugin, and also mentioned it to a couple of Scala committers at an earlier W-JAX in Munich.
It never had any mention of an author, so it wasn't obvious, he also did that one. While the one for Java tries to stick very closely to Andrew Kennedy's own work and papers, including what he did for the F# language, the libraries underneath the Scala one seemed like a very small rip-off from JSR-275 ;-)

I went into some of the archives and looked at the classes in more detail, and they seemed quite familiar. The usability and representation, e.g. that "*" notation is probably the only way Scala allowed it at the time, but poor and unusable for real world requirements. I don't recall the project name or URL by heart, but recently saw a more promising new approach on GitHub. Should you be interested to use that, please try it there.

With regards to JSR-308, Unit-API does provide compile time type-safety using Java Generics for nearly every operation. Some times the type of a calculation result has no predefined quantity (something Lex and Andrew Kennedy call Dimension, it goes back to Newton and isn't wrong, but as Unit-API, JScience and other similar libraries support multi-dimensional calculations, mixing the terms adds confusion in some context) but that is pretty much the only case, where you may need generics like Unit<?> instead of a concrete quantity.

I try to combine it with JSR-308, first to allow usage of a more comprehensive unit framework with 308 than that draft (he confirmed it as such and has no plans to continue working on it) and second because there is interest from other JCP Members to have that kind of functionality.
However, there are some JCP Members who prefer Generics over Annotations and told me using the latter in too many places they found makes code less readable and elegant in their opinion.
Annotations already are used a lot by some other JSRs, especially BeanValidation. I have some work in progress to define certain units that are not bound by a single value or calculation, but rather a range (e.g. wind speed) with the help of its constraints, or even define specialized constraints for Unit-API. Doing the same with 308 makes sense, especially once it becomes clearer, which version of Java it may join, at least as optional "module".

Regards,
Werner

[Miguel Garcia]

Not sure if this has been mentioned before, but video + slides of a talk on ScalaU by Adrian Fritsch are now online at http://days2011.scala-lang.org/node/138/325 

For further details on that library, please contact the author, I'm just a compiler hacker! ;-) 

Miguel

http://lamp.epfl.ch/~magarcia/ScalaCompilerCornerReloaded/