Google Groups

Scala - a Roadmap

martin Mar 20, 2012 4:29 AM
Posted in group: scala-language
There is much controversy surrounding SIP 18. I believe at least part
of it is because people don't have a clear idea where Scala is going.
So people are making up crazy reasons like that the primary purpose of
SIP 18 is to pander to naysayers.

Let me first say something about that:

First, the criticism of Scala that's most vexing for me is when people
say it's like C++, or it's a huge language that includes everything
including the kitchen sink. I'm hurt by the criticism because I
believe it describes Scala as the exact opposite of what I panned it
to be. I have always tried to make Scala a very powerful but at the
same beautifully simple language, by trying to find unifications of
formerly disparate concepts. Class hierarchies = algebraic types,
functions = objects, components = objects, and so on.

So I believe the criticisms are overall very unfair. But they do
contain a grain of truth, and that makes them even more vexing. While
Scala has a simple and consistent core, some of its more specialized
features are not yet as unified with the rest as they could be. My
ambition for the next 2-4 years is that we can find further
simplifications and unifications and arrive at a stage where Scala is
so obviously compact in its design that any accusations of it being a
complex language would be met with incredulity. That will be the best
counter-argument to the naysayers. But much more importantly, it will
be a big help for the people writing advanced software systems in
Scala. Their job will be easier because they will work with fewer but
more powerful concepts.

If we arrive to do the simplifications, that would be a good basis for
a Scala 3. Right now, all of this is very tentative. Scala 3 does not
have an arrival date, and it's not even sure that it will ever arrive.
But to give you an idea on what I will be working on, here are some
potential simplifications.

- The easiest win is probably XML literals. Seemed a great idea at the
time, now it sticks out like a sore thumb. I believe with the new
string interpolation scheme we will be able to put all of XML
processing in the libraries, which should be a big win. It also means
we could provide swappable alternatives to the current XML system such
as Anti XML or others.

- The type system. Ideally, Scala's types will be built from just
traits, mixin composition, refinements, and paths, and nothing else.
That's the true core of Scala as is captured in our dependent object
types formalism. We'll throw in classes for Java compatibility. We
still have to make this into a practical programming language
compatible with what Scala currently is. The potential breakthrough
idea here is to unify type parameters and abstract type members. The
idea would be to treat the following two types as equivalent

  trait Seq[type Elem]     and   trait Seq { type Elem }

The definition

  trait Seq[Elem]

could still be kept and be interpreted as a type that has an
inaccessible member Elem, similar to the distinction between

  class C(x: T)     and    class C(val x: T)

that we have now. The big simplifications are then:

(1) Any type can be written without its parameters.
(2) Any type that has abstract type members can be retroactively parameterized.
(3) Type parameters can be unified by name.

To illustrate (3), right now to create a synchronized hashmap, one has to write:

  new HashMap[String, List[Int]] with SynchronizedMap[String, List[Int]]

That's one of the thankfully few aspects where current Scala violates
the DRY principle. In the future you'd be able to write

  new HashMap[String, List[Int]] with SynchronizedMap

because SynchronizedMap could refer to the type Key and Value
parameters in Map. Or you could write

  new HashMap[Key = String, Value = List[Int]] with SynchronizedMap

to make it even clearer. The two formulations would be equivalent.

Now if we do that then we have suddenly gained the essential
functionality of higher-kinded types and existential types for free! A
higher-kinded type is simply a type where some parameters are left
uninstantiated. And an existential type is the same! Now clearly,
something must get lost in a scheme that unifies higher-kinded and
existential types by eliminating both. The main thing that does get
lost is early checking of kind-correctness. Nobody will complain that
you have left out type parameters of a type, because the result will
be legal. At the latest, you will get an error when you try to
instantiate a value of the problematic type. So type-checking will be
delayed. Everything will still be done at compile-time. But some of
the checks that used to raise errors at the declaration site will now
raise errors at the use site. In my mind that could be a price worth
paying for the great overall simplification and gain in expressive
power. The other thing that gets lost are the more complicated forms
of existential types that cannot be expressed as a (composition of)
types with uninstantiated type members.

One particularly amusing twist is that this could in one fell sweep
eliminate what I consider the worst part of the Scala compiler. It
turns out that the internal representation of higher-kinded types in
the Scala compiler is the same as the internal representation of raw
types in Java (there are good reasons for both representation
choices). But raw types should map to existentials, not to
higher-kinded types. We therefore need to map Java raw types to Scala
existential types. The code that does this is probably the most
fragile and intricate part of the Scala compiler. There's basically no
good way to do it without either forgetting some transformations or
accidentally tripping off cyclic reference errors. But with the
projected simplifications we would get

  raw types = existentials = higher-kinded types = types with
uninstantiated parameters

so the whole issue would vanish in a puff of smoke.

To summarize: If we do this simplification,

 - we could eliminate two classes of types in Scala, so that only a
simple core remains,
 - we could gain expressive power through unification of concepts,
 - we could avoid unnecessary repetition of parameters in mixin
compositions and extends clauses
 - we could strengthen the analogies between type parameters and value

Other more narrowly scoped ideas for the type system are to introduce
least upper bounds and greatest lower bounds of types as type
constructors. This would avoid the explosion of computed lub types
that we sometimes see in codebases today. And there are some ideas to
make type inference more powerful by making it constraint based.

One big unknown right now is how to ensure a high degree of backwards
compatibility or alternatively provide migration strategies. It's
clear that we have to do this if we want this to fly. It will require
an implementation and lots of experimentations. Therefore, I don't
expect any of these things to materialize before a timeframe of 2-4

So, what does this have to do with SIP 18? Two things:

First, while we might be able to remove complexities in the definition
of the Scala language, it's not so clear that we can remove
complexities in the code that people write. The curse of a very
powerful and regular language is that it provides no barriers against
over-abstraction. And this is a big problem for people working in
teams where not everyone is an expert Scala programmer. Hence the idea
to put in an import concept that does not prevent anything but forces
people to be explicit about some of the more powerful tools that they
use. I am certain there is no way we can let macros and dynamic types
into the language without such a provision.

Second, the discussion here shows that complex existentials might
actually be something we want to remove from a Scala 3. And
higher-kinded types might undergo some (hopefully smallish) changes to
syntax and typing rules. So I think it is prudent to make people flag
these two constructs now with explicit imports, because, unlike for
the rest of the language we do not want to project that these two
concepts will be maintained as they are forever. If you are willing to
keep your code up to date, no reason to shy away from them. But if you
want a codebase that will run unchanged 5 years from now, maybe you
should think before using complex existentials or higher kinded types.
Of course the docs for these two feature flags will contain a
discussion of these aspects, so people can make an informed choice for

I know that despite these explanations SIP 18 will still be
contentious. But let's keep the discussion of SIP 18 on scala-sips. Of
course I'd be happy to see responses to all other parts of this mail
in this thread.


 - Martin