scala-reflect.jar

[Paul Phillips]

If in principle scala.reflect should have no dependencies on
scala.tools.nsc (and that is what I'm attempting over here) any
theories on what I'm supposed to do with the appended code? Or maybe
someone should clarify for me exactly what division we think is
realistic.

Here's how I understand things:

-- library jar alone, interfaces + minimalist setup (what works?)
some things are stubs
-- reflection jar, enables something more (but what, and what not?)
-- compiler jar, enables everything

But the details at the boundary are fuzzy, at least in terms of what's expected.

The immediate sense problem child:

// Toolboxes. Another major offender is FrontEnds / Reporters.
package scala.reflect
package runtime

import scala.tools.nsc.reporters._
import scala.tools.nsc.ReflectGlobal
import scala.tools.nsc.CompilerCommand
import scala.tools.nsc.Global
import scala.tools.nsc.typechecker.Modes
import scala.tools.nsc.io.VirtualDirectory
import scala.tools.nsc.interpreter.AbstractFileClassLoader
import scala.tools.nsc.util.FreshNameCreator
import scala.reflect.internal.Flags
import scala.tools.nsc.util.{NoSourceFile, NoFile}
import java.lang.{Class => jClass}
import scala.compat.Platform.EOL

trait ToolBoxes extends { self: Universe =>

[Eugene Burmako]

I remember this idea coming from Martin last autumn, but we didnt
discuss the details.

What I can definitely say is that it is impossible to decouple
ToolBoxes and their Reporters from scala-compiler. It uses
tools.nsc.Global by design.

[Paul Phillips]

On Thu, Apr 26, 2012 at 10:58 AM, Eugene Burmako <eugene....@epfl.ch> wrote:
> What I can definitely say is that it is impossible to decouple
> ToolBoxes and their Reporters from scala-compiler. It uses
> tools.nsc.Global by design.

Okay, but if Universe mixes in Toolboxes, which it does, then there is
no separation between scala.reflect and the compiler whatsoever.

[Eugene Burmako]

Then, I guess, we will have to move mkToolBox away from Universe, e.g.
into scala.reflect package object, similarly to mkMirror.

[Paul Phillips]

On Thu, Apr 26, 2012 at 11:03 AM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Then, I guess, we will have to move mkToolBox away from Universe, e.g.
> into scala.reflect package object, similarly to mkMirror.

In case anyone was thinking I'd just dash this one off, I can now say
this will take me quite a long time. It was kind of a rookie mistake
not to have boundaries being enforced during the development of these
things. Now the degree of intertwingle is very high. And the last
thing I want to do is just start pushing more compiler into the
library. It will be an API tragedy of the highest order.

I call for a moratorium on adding anything to scala.reflect.

[Eugene Burmako]

Toolboxes are supposed to be an integral part of the new reflection API, so we have to add them somewhere. What would be the most appropriate location?

[Paul Phillips]

On Thu, Apr 26, 2012 at 11:56 AM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Toolboxes are supposed to be an integral part of the new reflection API, so
> we have to add them somewhere. What would be the most appropriate location?

There's no way for me to answer that until I've pulled this thread
longer. If they're an "integral" part of the API then we are in
trouble, because they apparently require the compiler. Fancy
applications like reification and macros cannot lead us to impose a 15
MB unnecessary dependency on the great majority of people who just
want to be able to tell what the signature of that method is.

Just to review, the biggest reason a reflection library is important
is that you cannot retrieve accurate types via java generic signatures
because of restrictions (implicit and explicit) on their form. This
is felt especially keenly because until 2.8.x we provided accurate
signatures for primitive types, but due to the intolerance of the rest
of the world to nonconformant signatures we stopped doing that in 2.9,
after people had come to rely on it.

The direct remedy for that situation involves a few interfaces,
analogous to java.lang.reflect.Method, java.lang.Class, etc. which
read the scala signature and provide accurate information. Everything
else - all of it - should be secondary to providing this. And the
part I described requires only a tiny bit of new code.

The ambitious project - reification, expression trees, macros,
whatever else - is nice, but it cannot be allowed to impede the simple
delivery of accurate type information at runtime, or we have failed.

I almost want to branch off and do that little bit from scratch and
without dependencies rather than attempting to untangle things any
further.

[Eugene Burmako]

Thanks for the historical excursion. Now the background behind reflection makes much more sense.

[Eugene Burmako]

On the topic of untangling. What other problems except toolboxes and reporters do you have?

[Paul Phillips]

On Thu, Apr 26, 2012 at 1:30 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> On the topic of untangling. What other problems except toolboxes and
> reporters do you have?

I don't know how deep those go; it's hard to tell how much requires
changes when it all depends on other changes which have to be made to
get it working that way. Some of the interfaces are not going to make
very good API as they are now. So one problem is how much of it I
feel like I have to modify. And then my next biggest problem is how
hard it already is to change anything; this is what I'm talking about
when it comes to DummyMirror. It slows me down like you can't
believe. I have to either drag it around with me everywhere I go, or
just stop letting stuff compile. When I do the latter I have a habit
of painting myself into a corner. It's not a choice I want to have to
make.

Then there is the problem that if it is incumbent upon me to try to
draw some lines so everyone doesn't have to have the compiler for
everything, I have to analyze all kinds of stuff with which I have
only vague familiarity.

Misc notes:

- Can "Context" be renamed to something more specific? There are
already too many Contexts in the compiler.
- default arguments don't belong in API
- as mentioned before, boolean arguments to methods are bad; but
boolean arguments with defaults are worse

[Eugene Burmako]

Lets remove the dummy for now. If this costs you too much, I think, we can sacrifice a few broken nightlies, right?

[martin odersky]

On Thu, Apr 26, 2012 at 9:29 PM, Paul Phillips <pa...@improving.org> wrote:

On Thu, Apr 26, 2012 at 11:56 AM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Toolboxes are supposed to be an integral part of the new reflection API, so
> we have to add them somewhere. What would be the most appropriate location?

There's no way for me to answer that until I've pulled this thread
longer.  If they're an "integral" part of the API then we are in
trouble, because they apparently require the compiler. Fancy
applications like reification and macros cannot lead us to impose a 15
MB unnecessary dependency on the great majority of people who just
want to be able to tell what the signature of that method is.

Indeed. Toolboxes need to be in the compiler. It seems I made a mistake in my grepping that I have not seen this before. So we need to move toolboxes out from Universe. I believe it's best if Eugene has a go at that. I do not think it's very difficult to do it.

There's the added problem that toolboxes are currently required if we want to have typed trees at runtime. I think we should consider optionally reifing trees with full type info. That's how LINQ does it. In that case toolboxes would not be required for standard situations.

Cheers

 - Martin

Just to review, the biggest reason a reflection library is important
is that you cannot retrieve accurate types via java generic signatures
because of restrictions (implicit and explicit) on their form.  This
is felt especially keenly because until 2.8.x we provided accurate
signatures for primitive types, but due to the intolerance of the rest
of the world to nonconformant signatures we stopped doing that in 2.9,
after people had come to rely on it.

The direct remedy for that situation involves a few interfaces,
analogous to java.lang.reflect.Method, java.lang.Class, etc. which
read the scala signature and provide accurate information.  Everything
else - all of it - should be secondary to providing this.  And the
part I described requires only a tiny bit of new code.

The ambitious project - reification, expression trees, macros,
whatever else - is nice, but it cannot be allowed to impede the simple
delivery of accurate type information at runtime, or we have failed.

I almost want to branch off and do that little bit from scratch and
without dependencies rather than attempting to untangle things any
further.

--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

[Eugene Burmako]

Speaking of toolboxes, when saying that they are a core part of the
reflection API, I meant that they are important to have (e.g.
DynamicReflect uses them to resolve some stuff that requires
typechecking), not that they are used a lot. In fact, most likely, the
only usage of toolboxes in scala/scala is eval in Expr (can be
disabled for now). Oh, and DynamicReflect.

So i think we will have to expose them in some way, but they should be
easily separable.

About the unfamiliar stuff, I am ready to answer any number of
questions. I am going to bed now, but in the morning I can explain
whatever you would like to know about reflection.

[Eugene Burmako]

Can we have mkToolBox along with mkMirror? Or do I put mkToolBox in scala.reflect.runtime?

[Eugene Burmako]

Reifying types would be hard, because that would require the reifier to generate symbol creation code for all the symbols in the reifee. I tried that before and failed. Is there a way to do that easily?

Also, DynamicProxy https://github.com/scala/scala/blob/master/src/library/scala/reflect/DynamicProxy.scala currently uses toolboxes to resolve default parameters and infer type arguments. I believe that would need to be changed as well.

[martin odersky]

On Thu, Apr 26, 2012 at 11:09 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Can we have mkToolBox along with mkMirror? Or do I put mkToolBox in scala.reflect.runtime?

How about putting it into a new package, scala.tools.reflect, which goes into scala-compiler.jar?

Cheers

 - Martin

[martin odersky]

On Thu, Apr 26, 2012 at 11:13 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Reifying types would be hard, because that would require the reifier to generate symbol creation code for all the symbols in the reifee. I tried that before and failed. Is there a way to do that easily?

What failed? I thought the previous code was basically working. Were there fundamental reasons to throw it out?

 

Also, DynamicProxy https://github.com/scala/scala/blob/master/src/library/scala/reflect/DynamicProxy.scala currently uses toolboxes to resolve default parameters and infer type arguments. I believe that would need to be changed as well.

I do not think DynamicProxy is essential at the current point. In any case, something that uses toolbox compilation is likely to be so slow that it has only limited usage anyway.

Cheers

 - Martin

 

[Eugene Burmako]

Yep, that's what I meant when mentioning scala.reflect.runtime, which is a package in scala-compiler. We definitely can move toolboxes into compiler.

Lets go through the problems we have to solve first (mentioned in the previous email).

[Eugene Burmako]

Didnt throw anything out, just switched off a few lines, so it can be restored as necessary.

The main problem is with reification of types that refer to locally defined stuff.

For example. How would you reify

class C

object C {

  def foo: C = ???

}

To assign types to all the nodes of the reification, you need to create symbols that will stand for C and its companion, and reference them in the reification result. 

In October I tried to do that, failed to make it work because of scope problems, and you talked me out of it, saying that it will be very challenging. Did I misunderstand something?

[martin odersky]

On Thu, Apr 26, 2012 at 11:17 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Yep, that's what I meant when mentioning scala.reflect.runtime, which is a package in scala-compiler. We definitely can move toolboxes into compiler.

No, scala.reflect.runtime needs to go into scala-reflect.jar, otherwise we won't be able to have a usable mirror.

Remember, there need to be scala-library.jar, with the dummy mirror, scala-reflect.jar, with the runtime mirror, and scala-compiler, with the toolboxes. The sooner we perform that split the less confusion.

Regarding the dummy mirror: I also think it can simply be omitted until all things about reflection stabilize. 

Cheers

 - Martin

[Paul Phillips]

On Thu, Apr 26, 2012 at 2:22 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> The main problem is with reification of types that refer to locally defined
> stuff.
>
> For example. How would you reify
>
> class C
> object C {
>   def foo: C = ???
> }

Oh yeah, I meant to email you about this earlier because I saw that
question in the code.

You've discovered "the avoidance problem", or at least a variation of
it. It's similar to the question of what type you infer for this:

def foo = { class C ; new C }

The answer must depend on what it means to be reified and what if any
contextual requirements are attached to reified types. In the type
inference example you pick the least supertype of C which isn't C.
Depending on what you want out of the reified type, you could do
something like that, or if you try to preserve it you have to reify
more stuff.

Note: I only dabble in theory.

[Eugene Burmako]

Right, my mistake

[Eugene Burmako]

Personally I'd expect reification to preserve everything meaningful
about the code, s.t. reification result could be typechecked again and
produce the same result. In the same vein, if I declare a class in a
reifee, I'd like to get all the info about its type.

However I am not sure about Martin's use case with typed reifications.
Maybe it would be enough to reify packed types. Maybe we dont need to
reify everything. Martin, could you, please, elaborate?

[martin odersky]

On Thu, Apr 26, 2012 at 11:37 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Personally I'd expect reification to preserve everything meaningful
about the code, s.t. reification result could be typechecked again and
produce the same result. In the same vein, if I declare a class in a
reifee, I'd like to get all the info about its type.

However I am not sure about Martin's use case with typed reifications.
Maybe it would be enough to reify packed types. Maybe we dont need to
reify everything. Martin, could you, please, elaborate?

I am not sure I understand yet. We'd need to reify simply exactly the types and symbols that are in the trees. 

Just so there's no misunderstanding, I do not propose that for 2.10. We should tackle it afterwards. Macros, and reification are still experimental, so we have leeway to add these things later.

Cheers

 - Martin

 

On 27 Apr 2012, at 00:31, Paul Phillips <pa...@improving.org> wrote:

> On Thu, Apr 26, 2012 at 2:22 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
>> The main problem is with reification of types that refer to locally defined
>> stuff.
>>
>> For example. How would you reify
>>
>> class C
>> object C {
>>   def foo: C = ???
>> }
>
> Oh yeah, I meant to email you about this earlier because I saw that
> question in the code.
>
> You've discovered "the avoidance problem", or at least a variation of
> it.  It's similar to the question of what type you infer for this:
>
>  def foo = { class C ; new C }
>
> The answer must depend on what it means to be reified and what if any
> contextual requirements are attached to reified types.  In the type
> inference example you pick the least supertype of C which isn't C.
> Depending on what you want out of the reified type, you could do
> something like that, or if you try to preserve it you have to reify
> more stuff.
>
> Note: I only dabble in theory.

[Eugene Burmako]

If its not urgent, I can explain in more detail offline, on Monday. And so far we will move toolboxes away from the library, and have those, who want typechecked reifications, explicitly reference scala-compiler. Okay?

Paul, how would you prefer to do it? I will have time tomorrow to perform the move, but I'd hate to have us produce conflicting changesets. Could you push your results before going to sleep?

[Paul Phillips]

I'm kind of stuck; you can have free reign.

[Eugene Burmako]

We have two usages of toolboxes in library:
1) DynamicProxy,
2) Expr.eval (when used outside of reify).

The former needs toolboxes to resolve default parameters and type
arguments. The latter I've seen a lot in macros, when people eval
stuff in macro expansions (e.g. see Heiko's example here:
https://issues.scala-lang.org/browse/SI-5713).

I think both uses are quite important, so I suggest that we do leave
toolboxes in mirrors, and instantiate them in the same way we
instantiate mirrors in library: try { <reflective construction> }
catch { throw new UnsupportedOperationException(...) }. So, scala-
library will reflectively call into scala-reflect, when it needs a
mirror, and scala-reflect will reflectively call into scala-internals,
when it needs a toolbox.

As we've seen, these reflection links are brittle, but at least we
don't add conceptually new brittleness with this approach.

[martin odersky]

On Fri, Apr 27, 2012 at 12:51 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

We have two usages of toolboxes in library:
1) DynamicProxy,
2) Expr.eval (when used outside of reify).

The former needs toolboxes to resolve default parameters and type
arguments. The latter I've seen a lot in macros, when people eval
stuff in macro expansions (e.g. see Heiko's example here:
https://issues.scala-lang.org/browse/SI-5713).

I think both uses are quite important, so I suggest that we do leave
toolboxes in mirrors, and instantiate them in the same way we
instantiate mirrors in library: try { <reflective construction> }
catch { throw new UnsupportedOperationException(...) }. So, scala-
library will reflectively call into scala-reflect, when it needs a
mirror, and scala-reflect will reflectively call into scala-internals,
when it needs a toolbox.

As we've seen, these reflection links are brittle, but at least we
don't add conceptually new brittleness with this approach.

OK. I also would hate too lose eval on Expr. - Martin 

On Apr 27, 12:58 am, Paul Phillips <pa...@improving.org> wrote:
> I'm kind of stuck; you can have free reign.
>
>
>

> On Thu, Apr 26, 2012 at 2:52 PM, Eugene Burmako <eugene.burm...@epfl.ch> wrote:
> > Paul, how would you prefer to do it? I will have time tomorrow to perform
> > the move, but I'd hate to have us produce conflicting changesets. Could you
> > push your results before going to sleep?

[Paul Phillips]

Having reflected on the matter (ha ha) I see only one avenue which has
much chance of making everyone happy. I can further break down the
exposed API into information which I can obtain in direct fashion from
reading scala signatures, and all the rest of it. We have a simple
read-only reflection interface similar to java's which is all in the
library: no dependency at all. The code would be essentially the same
as what's in scalap. And then I don't care so much how code breaks
down between scala-reflect.jar and scala-compiler.jar.

Importantly for my cause of not making things harder or heavierweight
in 2.10 than they were in 2.9, I would need some kind of "Tag" or
whatever we're calling them these days which can be synthesized in the
same no-dependency fashion, from the same mechanism. Would anyone
like to speculate on that point.

[Eugene Burmako]

What would you like to have in this tag? Do you need an erasure only, or you need something more?

[Paul Phillips]

Just erasure. Array creation and subclass tests, that's it.

[Eugene Burmako]

To do that you can use either:

* ErasureTag (provides erasure for arbitrary types, including untagged type parameters and abstract types)

* ClassTag (provides erasure and guarantees that the type is concrete)

Both have just erasure and don't have <:<, however, you can roll your own subtype test with asSubClass: http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Class.html#asSubclass(java.lang.Class). I didn't want to add <:< for the sake of minimalism, but we can discuss that.

[Jason Zaugg]

On Fri, Apr 27, 2012 at 4:09 PM, Paul Phillips <pa...@improving.org> wrote:
> Having reflected on the matter (ha ha) I see only one avenue which has
> much chance of making everyone happy.  I can further break down the
> exposed API into information which I can obtain in direct fashion from
> reading scala signatures, and all the rest of it.  We have a simple
> read-only reflection interface similar to java's which is all in the
> library: no dependency at all.  The code would be essentially the same
> as what's in scalap.  And then I don't care so much how code breaks
> down between scala-reflect.jar and scala-compiler.jar.

I question the utility of a purely read-only API for reflection.
Usually, one needs to need to invoke the methods and access the fields
that one sees in the mirror.

-jason

[martin odersky]

This means you create types, symbols, and trees, but do not put any functionality on them? Do I get that right? 

For instance, if you want to find the members of a Scala type and their types, that would not be supported?

Cheers

 - Martin

[Paul Phillips]

I mean, plus or minus, a simple and direct interface to exactly what
java reflection can tell you, but accurately.

class Foo { def f: List[Int] = Nil }

The best java can tell you about f is this:

()Lscala/collection/immutable/List<Ljava/lang/Object;>;;

Which is wrong in two different ways.

So: no Trees. Symbols primarily. If scalap can print it, it's
available, otherwise it isn't. That's all I'm after. I believe the
audience for this dwarfs the audience for anything else.

> For instance, if you want to find the members of a Scala type and their
> types, that would not be supported?

That would be supported at the same level scalap can provide it.
Here's an example.

package foo {
class B {
class A[+T >: Int with String]
def f = new A[Int]
def g = new A[String]

type Bippy
class DingDong
protected val zz: List[Array[Int]] = Nil
}
}

// scalap says:
class B extends java.lang.Object {
def this() = { /* compiled code */ }
class A[+T >: scala.Int with scala.Predef.String] extends java.lang.Object {
def this() = { /* compiled code */ }
}
def f : foo.B.A[scala.Int] = { /* compiled code */ }
def g : foo.B.A[scala.Predef.String] = { /* compiled code */ }
type Bippy
class DingDong extends java.lang.Object {
def this() = { /* compiled code */ }
}
protected val zz : scala.List[scala.Array[scala.Int]] = { /*
compiled code */ }
}

[martin odersky]

So, no inherited members?

 - Martin

[Paul Phillips]

We can make inherited members available, because it is easily
available in the same way it is in java, by loading the parent classes
and looking at their scala signatures. I didn't mean that I want to
literally bind myself to scalap's output; I meant that I am looking
only to expose the information which is already stored with each
classfile, but in a vastly more usable way, and with as little new
code as possible, and no new dependencies.

On Fri, Apr 27, 2012 at 9:56 AM, martin odersky <martin....@epfl.ch> wrote:
> So, no inherited members?

[Paul Phillips]

If the issue is "asSeenFrom" and such, we can limit the granularity of
members as much as necessary. I am the good, the perfect is not my
enemy.

[martin odersky]

On Fri, Apr 27, 2012 at 7:03 PM, Paul Phillips <pa...@improving.org> wrote:

If the issue is "asSeenFrom" and such, we can limit the granularity of
members as much as necessary.  I am the good, the perfect is not my
enemy.

Seriously, I think that would just perpetuate the piecework we did so far in scalap, manifests and so on. We should do it right or not do it. 

I also have not yet quite understood what you propose:

There's scala-library.jar  with some minimal mirror (original was essentially a dummy, now we consider adding scalap like functionality?)

There's scala-reflect.jar which gives you everything you need to do proper reflection with asSeenFrom, getField, setField, invoke, reify and so on. 

There's scala-compiler.jar which contains everything you need for runtime compilation.

What changes do you propose? Flesh out the mirror in scala-library? Re-merge scala-reflect and scala-compiler?

Cheers

 - Martin

[Paul Phillips]

On Fri, Apr 27, 2012 at 10:14 AM, martin odersky <martin....@epfl.ch> wrote:
> Seriously, I think that would just perpetuate the piecework we did so far in
> scalap, manifests and so on. We should do it right or not do it.

I'm all for that. However my definition of "doing it right" is "doing
it usefully", not "doing it with maximum possibility fidelity
regardless of what tradeoffs must be made to get there." I can say
without any doubt that there is a very useful improvement to be made
on what one can learn from
x.getClass.getMethods.map(_.toGenericString) which does not require
dragging many additional megabytes to obtain. And it looks to me like
2.10 is shaping up to exclude that improvement.

> I also have not yet quite understood what you propose:

I'm attempting to communicate it sufficiently abstractly that the
trees don't blur out the forest. What I propose is that we don't have
to destroy the village to save it. I propose to provide a superior
version of java reflection which takes into account the information in
scala signatures, without new dependencies, and to make this part of
the library. Failing that, I intend to ship something like that
personally. If I'd ever thought we would fail to provide that in 2.10
I'd have done it long ago.

[Eugene Burmako]

Do we know how much exactly will scala-reflect weigh? If it is 1-2mb,
maybe it is a decent price to pay for fidelity?

[Paul Phillips]

On Fri, Apr 27, 2012 at 11:38 AM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Do we know how much exactly will scala-reflect weigh? If it is 1-2mb,
> maybe it is a decent price to pay for fidelity?

It isn't. It's like groundhog day around here sometimes, because we
do this quite a lot: throw ourselves into solving problems most people
didn't even know they had, while studiously ignoring the problems they
can be observed to have. However I have to say that I don't think
this bears much discussion, because I don't know how I'm going to make
anyone see this who doesn't see it already: I'd rather spend my time
implementing it (even if it won't be in the standard library.)

[Eugene Burmako]

Not intending to force anything onto you, just to provide some info.

scala-reflect (copied from scala.reflect.runtime and scala.reflect.internal, with some compiler-specific cruft removed) is 2059k packed. Of those ~250k is runtime, ~200k is symbols, ~100k is trees, ~700k is types, ~300k is definitions, ~100k is importers. If you're interested I can provide more detailed breakdown later.

Maybe it'll be feasible to add some stuff to scala-library (~500k) and have basic reflection services working without imposing scala-reflect at the user. Would this be interesting to you Paul? Would it be a satisfying price to pay?

[Adriaan Moors]

On Fri, Apr 27, 2012 at 8:54 PM, Paul Phillips <pa...@improving.org> wrote:

 It's like groundhog day around here sometimes, because we
do this quite a lot: throw ourselves into solving problems most people
didn't even know they had, while studiously ignoring the problems they
can be observed to have.

+1, FWIW

I think both have their place (I know you're not arguing against that)

reflection+macros as new, researchy, even experimental features -- let's call those the pure breed shiny race poneys

I see the old reflection as the robust workhorse that pulls the plough, even though all it does is trudge back and forth through the mud

blind to all the tractors racing past it, but people still depend on it for their potato harvest

more technically, it would be a regression to impose more requirements on usages of reflection (classloader issues, size increases, slow downs,...)

adriaan

ps: I never said 'unicorn'

[martin odersky]

On Fri, Apr 27, 2012 at 9:05 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Not intending to force anything onto you, just to provide some info.

scala-reflect (copied from scala.reflect.runtime and scala.reflect.internal, with some compiler-specific cruft removed) is 2059k packed. Of those ~250k is runtime, ~200k is symbols, ~100k is trees, ~700k is types, ~300k is definitions, ~100k is importers. If you're interested I can provide more detailed breakdown later.

Maybe it'll be feasible to add some stuff to scala-library (~500k) and have basic reflection services working without imposing scala-reflect at the user. Would this be interesting to you Paul? Would it be a satisfying price to pay?

I thought we all said we'd want to modularize more and not pack everything into the standard library. Now is a good time to start. 2M seems reasonable for its own jar. And, just to be clear, 2M of code pales in comparison to the mental overload of multiple reflection interfaces.

 - Martin

[martin odersky]

On Fri, Apr 27, 2012 at 9:10 PM, Adriaan Moors <adriaa...@epfl.ch> wrote:

On Fri, Apr 27, 2012 at 8:54 PM, Paul Phillips <pa...@improving.org> wrote:

 It's like groundhog day around here sometimes, because we
do this quite a lot: throw ourselves into solving problems most people
didn't even know they had, while studiously ignoring the problems they
can be observed to have.

+1, FWIW

I think both have their place (I know you're not arguing against that)

reflection+macros as new, researchy, even experimental features -- let's call those the pure breed shiny race poneys

Macros are experimental; but reflection will be  completely standard. I'm just insisting on doing it right. 

 

I see the old reflection as the robust workhorse that pulls the plough, even though all it does is trudge back and forth through the mud

blind to all the tractors racing past it, but people still depend on it for their potato harvest

The old reflection was a pile of hacks, never officially supported, that has crumbled since the changes to signatures in 2.9. There's nothing to be salvaged here.

 

more technically, it would be a regression to impose more requirements on usages of reflection (classloader issues, size increases, slow downs,...)

  

Classloader issues need to be addressed, and I was hoping that we'd be doing that instead of going back to square one with the basic requirements.

 - Martin

[martin odersky]

On Fri, Apr 27, 2012 at 8:30 PM, Paul Phillips <pa...@improving.org> wrote:

On Fri, Apr 27, 2012 at 10:14 AM, martin odersky <martin....@epfl.ch> wrote:
> Seriously, I think that would just perpetuate the piecework we did so far in
> scalap, manifests and so on. We should do it right or not do it.

I'm all for that.  However my definition of "doing it right" is "doing
it usefully", not "doing it with maximum possibility fidelity
regardless of what tradeoffs must be made to get there." I can say
without any doubt that there is a very useful improvement to be made
on what one can learn from
x.getClass.getMethods.map(_.toGenericString) which does not require
dragging many additional megabytes to obtain.  And it looks to me like
2.10 is shaping up to exclude that improvement.

That's your opinion. I have been observing others try this same approach for many years and see it fail. I refuse to say let's do it wrong simply to save the code size to do it right. Code size can be overcome by modularization. It's not an issue in itself.

 

> I also have not yet quite understood what you propose:

I'm attempting to communicate it sufficiently abstractly that the
trees don't blur out the forest.  What I propose is that we don't have
to destroy the village to save it.  I propose to provide a superior
version of java reflection which takes into account the information in
scala signatures, without new dependencies, and to make this part of
the library.  Failing that, I intend to ship something like that
personally.  If I'd ever thought we would fail to provide that in 2.10
I'd have done it long ago.

Reflection has been unchanged since last September/October. It's a bit late to reflect now on what it fails or does not fail to do. What we need to do now (and urgently!) is 3 things:

 - do the modularization breaking out scala-reflect.jar

 - resolve the classloading issues

 - simplify the API

Any help is appreciated. But if no help is forthcoming, I will do it alone.

Cheers

 - Martin

[Adriaan Moors]

but reflection includes Typeable, which depends on reify (last time I checked) , which depends on macro's

I agree fully that the new approach to reflection is 100% theoretically sound, and thus the right thing.

I see the old approach as backwards compatibility baggage that we must either replace fully faithfully,

including the more low-brow requirements people may have, or that we must maintain just a bit longer

[martin odersky]

On Fri, Apr 27, 2012 at 9:27 PM, Adriaan Moors <adriaa...@epfl.ch> wrote:

but reflection includes Typeable, which depends on reify (last time I checked) , which depends on macro's

No, it's just generation of Typeable values from inside the compiler that depends on reify. But then we already have all dependencies anyway. Otherwise Typeable is a trivial wrapper around scala.reflect.api.Type. And that's what it should be. The previous approach to manifests makes me cringe on how partial and wrong it was.

Cheers

 - Martin

[Paul Phillips]

On Fri, Apr 27, 2012 at 12:14 PM, martin odersky <martin....@epfl.ch> wrote:
> I thought we all said we'd want to modularize more and not pack everything
> into the standard library.

I hope it didn't sound like we wanted the worst of both worlds.

> Classloader issues need to be addressed, and I was hoping that we'd be doing that instead of going back to square one with the basic requirements.

I guess this is what comes of not having a SIP. I will continue to
look at the classloader situation, but FYI my basic requirements
haven't changed an iota. It's not like the development process has
been enormously transparent as to which parts were going to wind up
where, as is easily seen by the fact that we still don't know which
parts are going where. But it never occurred to me we would fail to
provide what I consider the most fundamental use case.

[Josh Suereth]

I think that's the distinction we need to make.  The new reflection started encroaching on the old reflection and imposing new constraints that are just infeasible for us to enforce.

If we had 3 years of people not using Manifests *or* people were always including the compiler on the classpath when running scala, maybe we'd have been fine.   

The reality is, that's not the case and it's very breaking for most people.

[Adriaan Moors]

On Fri, Apr 27, 2012 at 9:32 PM, martin odersky <martin....@epfl.ch> wrote:

The previous approach to manifests makes me cringe on how partial and wrong it was.

me too, I'll never forgot the day I looked at its implementation of isSubArgs 

[Josh Suereth]

I think the big mistake with those was releasing anything besides ClassManifest, which is the 99% use case in a lot of my code.

[martin odersky]

On Fri, Apr 27, 2012 at 9:34 PM, Josh Suereth <joshua....@gmail.com> wrote:

I think that's the distinction we need to make.  The new reflection started encroaching on the old reflection and imposing new constraints that are just infeasible for us to enforce.

If we had 3 years of people not using Manifests *or* people were always including the compiler on the classpath when running scala, maybe we'd have been fine.   

The reality is, that's not the case and it's very breaking for most people.

Manifests are kept as is. Reflection is about to be broken out of the compiler. So I hope these gripes are from a temporary snapshot that's no longer true.

It's a big thing, agreed. But we need to tackle it now.

 - Martin

[martin odersky]

On Fri, Apr 27, 2012 at 9:37 PM, Josh Suereth <joshua....@gmail.com> wrote:

I think the big mistake with those was releasing anything besides ClassManifest, which is the 99% use case in a lot of my code.

Yes. Manifests were the experimental thing because we did not think them through to the end and were content with patchwork. If it's one thing I have realized in my adventure into reflection since is that it's totally unforgiving. You have to have things right down to the last iota or you will suffer. That's why I defend the current reflection design like a Tigress defends her kids.

Cheers

 -- Martin

[Josh Suereth]

I think you're right for rich reflection.  However, when most of the time I need just a ClassManifest and I've been writing Manifest, maybe we just deprecate the advanced uses in Manifest, and point people at TypeTags for that functionality.   Was ClassManifest mis-designed?

- Josh

[martin odersky]

On Fri, Apr 27, 2012 at 10:11 PM, Josh Suereth <joshua....@gmail.com> wrote:

I think you're right for rich reflection.  However, when most of the time I need just a ClassManifest and I've been writing Manifest, maybe we just deprecate the advanced uses in Manifest, and point people at TypeTags for that functionality.   Was ClassManifest mis-designed?

Yes, in that ClassManifest uselessly carried around type arguments that needed code and time to generate. But ClassManifest is in any case retained. It's a subtype of Classable which has the minimal interface.

Cheers

 - Martin

[Alex Cruise]

FWIW, at the risk of stating the blindingly obvious....  

I'm happy to see scala-reflect in a separate jar, as long as the distinction reflects a compile-time, not merely runtime, dependency.  That is, if I try to use reflection in my code, but forget to update my dependencies, my project shouldn't compile.  

In more concrete terms, I just want to make sure the reflection API gets evicted at the same time as his noisy neighbour, the reflection implementation.

-0xe1a, $0.02 poorer

[martin odersky]

That's an interesting thought, and a good principle. Let me think what it would take to get there.

Cheers

 - Martin

[Eugene Burmako]

It's perfectly possible to use macros for expressing compile-time
requirements on the environment (remember Paul's idea from a few weeks
ago?). A more interesting question is how we enforce correct runtime
environments.

On Apr 27, 11:45 pm, martin odersky <martin.oder...@epfl.ch> wrote:

[Eugene Burmako]

Maybe it'd be possible to add several hundred kilobytes to the stdlib
and get limited reflection capabilities (say, no importers or
definitions). As I see it, dummy might actually be not a complete
dummy, but provide services of its own. Would it be useful to pursue
this direction or even +300-500k to stdlib is completely unacceptable?

On Apr 27, 10:23 pm, martin odersky <martin.oder...@epfl.ch> wrote:
> On Fri, Apr 27, 2012 at 8:30 PM, Paul Phillips <pa...@improving.org> wrote:

> > On Fri, Apr 27, 2012 at 10:14 AM, martin odersky <martin.oder...@epfl.ch>

[martin odersky]

On Fri, Apr 27, 2012 at 11:04 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Maybe it'd be possible to add several hundred kilobytes to the stdlib
and get limited reflection capabilities (say, no importers or
definitions). As I see it, dummy might actually be not a complete
dummy, but provide services of its own. Would it be useful to pursue
this direction or even +300-500k to stdlib is completely unacceptable?

I think it's definitely worth analyzing. - Martin

 

[Eugene Burmako]

what: scala/reflect/runtime and scala/reflect/internal from scala-compiler.jar

total: 5.1M, compressed 2.3M (2.2x compression ratio)
(all sizes below are in kilobytes and are uncompressed)

essential stuff:
260 symboltable and universe
1500 types + basetypeseqs
117 trees
474 symbols
100 annotationinfos
180 pickling
60 jvmclassinfo
157 javatoscala
30 scalatojava
37 symbolloaders
39 flags
99 treegen + treebuildtil
72 treeprinters
80 custom collections
====
3205  non-compressed

non-essential stuff:
300 definitions
162 importers
424 synchronized
140 toolboxes
66 kinds
175 stdnames
138 treeinfo
108 statistics
24 annotationcheckers
97 erasure, refchecks, uncurry
32 typedebugging
====
1666 non-compressed

recapping essential stuff:

450 symbol loading

1500 types

550 symbols

100 trees

100 annotationinfos

250 abstract classes that inherit from traits

100 tree builders

100 tree printers

===

3.2m compressed ~= 1.5m compressed

it seems to me that if we shave off stuff from types and symbols, AND exclude synchronization, definitions, names, importers, toolboxes, we can reach 700k. 

[Eugene Burmako]

yep, i took the classes from the locker build. quite possible that
optimizer might skew the numbers either positively or negatively

> On 28 April 2012 00:13, martin odersky <martin.oder...@epfl.ch> wrote:

[martin odersky]

 - default arguments don't belong in API

Why? It conveys useful information. Why ban it from the API?

 

 - as mentioned before, boolean arguments to methods are bad; but
boolean arguments with defaults are worse

I disagree. If an argument is a boolean it is a boolean. Just remember to pass a named parameter, that's all. 

  process(x)

  process(x, extendedChecks = true)

I don't think there's a clearer way to express things.

 - Martin

[Jan Vanek]

On 28.04.2012 10:43, martin odersky wrote:

 - as mentioned before, boolean arguments to methods are bad; but
boolean arguments with defaults are worse

I disagree. If an argument is a boolean it is a boolean. Just remember to pass a named parameter, that's all. 

  process(x)

  process(x, extendedChecks = true)

I don't think there's a clearer way to express things.

Each time I refactored a code which used boolean arguments to control the behavior of methods into code without those boolean arguments I felt better about the code. Paul's comment just rung the internal bell that I have the same experience. That's all I can say, unfortunately, I can't express why is it so.

Regards,
Jan

[martin odersky]

Did the code use named arguments for the booleans? If not, I share your feeling. With named arguments I do not see what the issue is. My proposal: 

  process(x)

  process(x, extendedChecks = true)

The most probable alternative:

            process(x)

            processWithExtendedChecks(x)

It's not a dramatic difference, but I still think I prefer the named parameter style.

Cheers

 - Martin

[martin odersky]

I am exploring a tweaked design which would have this property and also avoids creating dummy mirrors with hundreds of unimplemented operations. The gist of it is in 

https://github.com/odersky/scala/blob/topic/reflection/test/files/run/mirrors.scala

The idea is to have in scala-library jar a minimal mirror which just allows to name, construct, and deconstruct the types of discourse (i.e. Types, Symbols, Trees, AnnotationInfos, Names, Positions) but which offers no further operations (or, if we wish, offers a small set of operations such as those found in scalap). 

The mirror's type, scala.reflect.Universe, is abstract but can be implemented by a FallbackUniverse. 

Then in scala-reflect.jar we'd have the reflect.internal package as usual. This one contains

fullMirror, a full implementation of the mirror, including all operations. It's essentially unchanged to what we have now. Furthermore in scala-reflect.jar there is an object scala.reflect.ops which contains implicit wrappers for all types in the minimal universe. Each wrapper adds all operations that are now in the AbsXXX classes in reflect.api and implements these operations by simply casting the type to the corresponding fullMirror type.

Finally, the actual value of the scala.reflect.mirror is either fullMirror or the FallbackMirror, depending on whether fullMirror is found.

The correctness of the design rests on the fact that if one imports at compile time reflect.ops from scala-reflect.jar, and executes one of these ops at run-time, then fullMirror must have been found, so scala.reflect.mirror will refer to a scala.reflect.Universe, so the cast will succeed.

I'd like to give the reflection libraries a go with that redesign. Paul, Eugene, can you commit any in-flight changes so that we don't duplicate work?

Thanks

 - Martin

[Eugene Burmako]

Dont have any ongoing reflection work worth committing. Tomorrow I will work on macros, and on Monday we'll resync, okay?

[Eugene Burmako]

How will the ops scale to arbitrary mirrors? Can we make tops and uops dependent methods? Can we avoid explicit "import ops._" in macro code?

On 28 Apr 2012, at 13:34, martin odersky <martin....@epfl.ch> wrote:

[Eugene Burmako]

Also, how do we generate reifieds and tags? They must not include "import ops._", so we will have to have TreeBuildUtil api to scala-library, right?

On 28 Apr 2012, at 13:34, martin odersky <martin....@epfl.ch> wrote:

[Ivan Todoroski]

On 27.04.2012 20:54, Paul Phillips wrote:
> I don't know how I'm going to make anyone see this who doesn't see it
> already

For what it's worth, I think I can see exactly what Paul is talking about.

What is really needed by every day Scala programmers (not compiler gurus
or advanced library designers) is a simple reflection library that has
more or less the capabilities of the java.lang.reflect package, but
extended to provide the richer information found in Scala's type system
in a sensible way, instead of the mangled names and signatures you get
if you try to use java.lang.reflect directly on Scala classes/objects.

We should have this without having to include scala-compiler.jar or any
other JARs in our build dependencies.

Maybe it will help if we try to separate these capabilities into
different levels, although the boundaries will certainly be fuzzy.



Level 1: Basic read-only access to runtime type information for Scala
classes, their methods, fields, companion objects, traits, nested
objects, type parameters, etc. We should be able to get this info either
from an existing object that is an instance of a Scala class, or by
loading a Scala class by name.

Level 2: Reflectively calling methods on Scala objects or modifying
their public fields/properties. Given a method reference obtained from
Level 1, we should be able to call that method on an object of the
appropriate class and give it the appropriate parameters. Same goes for
reflectively setting and getting values of fields/properties (which
should call the appropriate setters/getters if present).

Level 3: Reflectively creating instances of Scala classes. We should be
able to create instances of classes obtained by Level 1, and we should
also be able to create instances of classes that are specified by a
method's type parameters, even if the concrete classes might not be
known at the method definition site. As a special case, we should also
be able to create arrays with that element type.

Level 4: Full on reification, type checking of generated ASTs, macros,
and other assorted wizardry and alien technology.



Levels 1 to 3 must be part of the standard library (or at least as much
of them as feasible). We shouldn't have to go around our build
toolchains adding scala-compiler.jar (or even scala-reflect.jar)
everywhere just to use this basic reflection capability.

I realize that you want to modularize the standard lib, but I consider
basic reflection capabilities a core part of the language, even more so
than e.g. the collection library. Just like you wouldn't spin off the
collection library into a separate JAR, you shouldn't spin off the basic
reflection library either (Level 1 to 3).

Of course, Level 4 can require scala-compiler.jar, scala-reflect.jar, or
whatever other special considerations it needs. It is firmly in the
advanced library designer territory.

It could be problematic if *using* libraries created using Level 4
features required you to add scala-compiler.jar and so forth to the
runtime classpath, but I guess that's a separate problem...

[Seth Tisue]

On Sat, Apr 28, 2012 at 7:28 AM, Ivan Todoroski <grnch...@gmx.net> wrote:
> On 27.04.2012 20:54, Paul Phillips wrote:
>> I don't know how I'm going to make anyone see this who doesn't see it already
>

> I think I can see exactly what Paul is talking about.

I can too, and your unpacking of it is spot-on.

P.S. I love baby tigers and would never harm one.

--
Seth Tisue | Northwestern University | http://tisue.net
lead developer, NetLogo: http://ccl.northwestern.edu/netlogo/

[Jan Vanek]

Dear Martin,

I'd prefer the mirror in reflect.jar strictly extending the mirror in library.jar, not as 2 implementations of an abstract Universe. Close to virtual classes.

// in library
package reflect
abstract class Universe {
  type T <: TImpl
  trait TImpl {
  }
  def T(): T
}

/*internal*/ class UniverseImpl extends Universe {
  // knot tied
}

// in reflect
package reflect.internal
abstract class Universe extends reflect.Universe {
   type T <: TImpl
   trait TImpl extends super.TImpl {
   }
}

/*internal*/ class UniverseImpl extends Universe {
  // knot tied
}

I think quite a lot of code should be the same and thus ideally not repeated in reflect.jar. This is of course an implementation detail. More important is that I like to see it as essentially one mirror, which is simply extended by the functionality in reflect.jar.

Regarding the boundaries (just repeating other people actually):
library.jar - reflection can everything what java can but in a scala way (scala can see itself as scala, up to what java can say about java)
reflect.jar - reflecting about the code (as Paul wrote), "advanced" type operations
compiler.jar - runtime compilation/macros

Best regards,
Jan

P.S. I avoided the word FallbackMirror. Too pejorative. The mirror in library.jar already does what I need at the moment :-)

[martin odersky]

On Sat, Apr 28, 2012 at 3:09 PM, Jan Vanek <j3v...@googlemail.com> wrote:

Dear Martin,

I'd prefer the mirror in reflect.jar strictly extending the mirror in library.jar, not as 2 implementations of an abstract Universe. Close to virtual classes.

// in library
package reflect
abstract class Universe {
  type T <: TImpl
  trait TImpl {
  }
  def T(): T
}

/*internal*/ class UniverseImpl extends Universe {
  // knot tied
}

// in reflect
package reflect.internal
abstract class Universe extends reflect.Universe {
   type T <: TImpl
   trait TImpl extends super.TImpl {
   }
}

/*internal*/ class UniverseImpl extends Universe {
  // knot tied
}

I think quite a lot of code should be the same and thus ideally not repeated in reflect.jar. This is of course an implementation detail. More important is that I like to see it as essentially one mirror, which is simply extended by the functionality in reflect.jar.

Regarding the boundaries (just repeating other people actually):
library.jar - reflection can everything what java can but in a scala way (scala can see itself as scala, up to what java can say about java) 

reflect.jar - reflecting about the code (as Paul wrote), "advanced" type operations

Unfortunately it does not work that way. To do anything of consequence you need "advanced" type operations. They are not optional. As a good example, take invoke. In the case of overloading, to select the right method for a list of arguments, you need about all the mechanisms the Scala type system provides.

Also, reflect is not available on all platforms (eg direct compilation to JavaScript or via GWT miss it). So there's a clear case for breaking reflection out. I believe it's quite likely that Java 8 will do the same when Jigsaw is here.

 

compiler.jar - runtime compilation/macros

Agreed.

Cheers

 - Martin

[Ivan Todoroski]

On 28.04.2012 15:41, martin odersky wrote:
> Regarding the boundaries (just repeating other people actually):
> library.jar - reflection can everything what java can but in a scala
> way (scala can see itself as scala, up to what java can say about java)
>
> reflect.jar - reflecting about the code (as Paul wrote), "advanced"
> type operations
>
>
> Unfortunately it does not work that way. To do anything of consequence
> you need "advanced" type operations. They are not optional. As a good
> example, take invoke. In the case of overloading, to select the right
> method for a list of arguments, you need about all the mechanisms the
> Scala type system provides.

Hi Martin,

I'm wondering, what if I didn't need the reflection library to
automatically resolve overloads for me? Would I still have to mess with
additional JAR dependencies just because of this advanced feature that I
might not ever use?

I would be perfectly happy for the Scala equivalent of
Class.getMethods() to return separate Method references for each of the
overloaded variants, just like Java does. Then my application logic can
worry about which variant to call (if it needs to worry about it at all).

I guess what I'm asking is: why can't the basic Scala reflection go in
the standard lib (minus automatic overload resolution etc.), and have
these advanced compiler-like features like automatic overload resolution
go in a separate scala-reflect.jar for people that actually need them?

I think I can derive *a lot* of value out of a Scala reflection library
in the standard lib that just did for Scala more or less what
java.lang.reflect does for Java, before reaching a point where I would
need automatic overload resolution or other more advanced features that
would justify additional JARs and worrying about classloaders (if I ever
reach such a point at all).

I am approaching this from the standpoint of a pragmatic day-to-day
programmer, who is trying to use Scala wherever he can and must
integrate it with legacy Java code with as little friction as possible,
and who knows next to nothing about Scala compiler/library internals
(it's the only standpoint available to me).

The prospect of dealing with additional JAR dependencies and classloader
issues just because I wanted to use a sane Scala reflection interface
here and there scares me, and would probably drive me away from using
reflection at all.

Especially considering that I still have to worry about integrating my
Scala code with legacy Java code running in large J2EE application
servers, or OSGI containers, or DI frameworks, which already have enough
classloader complexities of their own.

> Also, reflect is not available on all platforms (eg direct compilation
> to JavaScript or via GWT miss it). So there's a clear case for breaking
> reflection out. I believe it's quite likely that Java 8 will do the same
> when Jigsaw is here.

But GWT doesn't support many other features of the Scala/Java platform
either, like threads, sockets, file I/O and many other things. Should
they be split out as well on the same grounds?

[martin odersky]

On Sat, Apr 28, 2012 at 4:42 PM, Ivan Todoroski <grnch...@gmx.net> wrote:

On 28.04.2012 15:41, martin odersky wrote:

   Regarding the boundaries (just repeating other people actually):
   library.jar - reflection can everything what java can but in a scala
   way (scala can see itself as scala, up to what java can say about java)
   reflect.jar - reflecting about the code (as Paul wrote), "advanced"
   type operations


Unfortunately it does not work that way. To do anything of consequence you need "advanced" type operations. They are not optional. As a good example, take invoke. In the case of overloading, to select the right method for a list of arguments, you need about all the mechanisms the Scala type system provides.

Hi Martin,

I'm wondering, what if I didn't need the reflection library to automatically resolve overloads for me? Would I still have to mess with additional JAR dependencies just because of this advanced feature that I might not ever use?

I would be perfectly happy for the Scala equivalent of Class.getMethods() to return separate Method references for each of the overloaded variants, just like Java does. Then my application logic can worry about which variant to call (if it needs to worry about it at all).

I don't know. I think invoking with Scala types is pretty basic. At least Java allows you to select a method with the erased Java types; it does not force you to iterate through getMethods. It would be logical to allow selection with Scala types in the Scala library. 

But there's much more lurking under the covers. Even figuring out which Scala methods correspond to which Java methods is far from trivial. Consider how Scala types are mangled into Java types. You have to redo the full mangling faithfully for something as simple as "give me the Scala method corresponding to this Java method". 

I once thought reflection was simple. After having observed projects to do it that went nowhere and after having jumped into and struggled a lot it I am now convinced that it's anything but simple. Unfortunately, this fact alone is very hard to explain, because most everybody else still thinks it's simple... (Java reflection, btw, is also a multi-MB thing).

In any case, Scala lived without reflection for 8 years. I am a bit surprised that people now find the burden of an additional jar so high. If people could do without reflection for so long, that's good evidence for me that the majority will be able to continue without reflection, so it can and should be in its own module. It's not at all like collections. I still have to see the Scala program that does not use any Scala collection.

I'll stop on this thread now to get going with the implementation.

Cheers

 - Martin

[Jan Vanek]

Just found my keys after 2 hours, crazy! The implication of such refactoring might be more than just "surface treatment", it might require deeper changes like new/changed classes. E.g. in this example you might have:

class Processor {
  def process() = ...
}

class FullCheckingProcessor extends Processor {
  override def process() = ...
}

I emphasize Paul's "there is *almost always* a better way". So it is not a dogma, in this particular case the boolean might really be the best. I just agree with this "almost always".

Yes, I didn't refactor such code with named parameters. However, making the parameter named makes it *almost always* :-) only a bit better.

Best regards,
Jan

[Ivan Todoroski]

On 28.04.2012 17:02, martin odersky wrote:
> (Java reflection, btw, is also a multi-MB thing).
>
> In any case, Scala lived without reflection for 8 years. I am a bit
> surprised that people now find the burden of an additional jar so high.

Oh, sorry I didn't make myself clear. I *do not care* about the
additional MB cost of additional JARs (well, within reason).

What scared me was:

a) The class loader issues when using reflection, which could turn out
to be especially nasty in a classloader-heavy environments (J2EE, OSGI,
etc).

b) The need to thread scala-compiler.jar throughout the build/deployment
cycle, which would essentially make scala-compiler.jar part of the Scala
standard library as far as the application is concerned.

This could lead to version skew issues when deploying Scala libraries
from multiple sources, since scala-compiler.jar is pretty much internal
API and I am not sure compatibility of its internal APIs is guaranteed
even within the same major Scala version.

My wish to keep basic reflection in the standard library stems from
these two concerns. If Scala reflection is robust enough to avoid any
hair-pulling issues with the above, the additional cost in MB is a
non-issue for me (again, within reason).

> If people could do without reflection for so long, that's good evidence
> for me that the majority will be able to continue without reflection, so
> it can and should be in its own module.

I can't speak for that 8 year period as I started using Scala fairly
recently, but at least for me the lack of proper Scala reflection is
very keenly felt.

[Paul Phillips]

On Saturday, April 28, 2012, martin odersky wrote:

I'd like to give the reflection libraries a go with that redesign. Paul, Eugene, can you commit any in-flight changes so that we don't duplicate work?

I'm pretty much offline until tomorrow; I have nothing committable in the hopper.

[martin odersky]

On Sat, Apr 28, 2012 at 12:51 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

How will the ops scale to arbitrary mirrors?

I don't think they need to. I pushed a refined version which corresponds better to what goes on in reflection. 

Essentially, reflect.api.Universe is split in two parts. The part in scala-library is now called reflect.core.Universe. It still contains the same abstract types and creators/extractors as api.Universe and also the traits that form the upper bounds of the abstract types and that contain the operations. But there is no link from the type to the operations, so nothing needs to be implemented in the minimal mirror.

reflect.api has moved to scala-reflect.jar. Its universe inherits from reflect.core.Universe and simply reestablishes the upper bounds. So, for instance where in core we had 

  type Symbol

  trait AbsSymbol { ... }

in api we now have:

  type Symbol <: AbsSymbol

Macros and everybody doing serious reflection work will use reflect.api and that will be guaranteed to come with a backing fully functional mirror mecause both reflect.api and reflect.internal are packaged in scala-reflect.jar. 

But Manifests and Typables will use only reflect.core.Universe. Furthermore, reflect.ops simply decorates reflect.mirror (of type reflect.core.Universe) so that it gets all operations. 

In the end it's not such a big step from where we are now, but I believe it meets all expectations.

We can also consider hanging some minimal reflection operations into reflect.core. That's OK as long as we do not duplicate functionality in reflect.api. But t I am sceptical we can get very far because arguably it's better not even to rely on Java reflection in reflect.core.  

Cheers

 - Martin

[martin odersky]

OK, thanks! I think I have validated the core design that will satisfy all our requirements (see last mail in this thread). I'll try to apply this to the actual reflection library tomorrow. 

Cheers

 - Martin

[Daniel Sobral]

On Sat, Apr 28, 2012 at 12:06, Jan Vanek <j3v...@googlemail.com> wrote:

> Did the code use named arguments for the booleans? If not, I share your
> feeling. With named arguments I do not see what the issue is. My proposal:
>
>   process(x)
>   process(x, extendedChecks = true)
>
> The most probable alternative:
>
>             process(x)
>             processWithExtendedChecks(x)
>
> It's not a dramatic difference, but I still think I prefer the named
> parameter style.

If only if that could be mandatory. Unfortunately, it is very likely
that lines like this will show up in code:

process(x, true)
process(x, false)


--
Daniel C. Sobral

I travel to the future all the time.

[martin odersky]

I agree, and that's what code review needs to guard against. For the same reason I would be very careful with booleans in public APIs. But in an internal project it is perfectly defensible to have booleans that are passed using named arguments.

Cheers

 - Martin

 

[Ismael Juma]

On Sun, Apr 29, 2012 at 4:32 PM, martin odersky <martin....@epfl.ch> wrote:

I agree, and that's what code review needs to guard against. For the same reason I would be very careful with booleans in public APIs. But in an internal project it is perfectly defensible to have booleans that are passed using named arguments.

Another reason why boolean parameters are bad for public APIs is that they are difficult to extend. If suddenly one needs a third state, it can't be added easily. If an enum is used, on the other hand, this is no longer an issue. In a language with good enum support (like Java, ironically), enums are probably a better option in almost every case than a boolean parameter.

Best,

Ismael

[Ismael Juma]

On Sat, Apr 28, 2012 at 4:02 PM, martin odersky <martin....@epfl.ch> wrote:

In any case, Scala lived without reflection for 8 years. I am a bit surprised that people now find the burden of an additional jar so high. If people could do without reflection for so long, that's good evidence for me that the majority will be able to continue without reflection, so it can and should be in its own module.

The burden of an additional jar is not so high, in my opinion. I'd like to add, however, that a _lot_ of projects carry the scala-compiler.jar these days as it's a dependency of popular libraries like lift-json (when using Scala 2.9.x). So, I believe more people are relying on some form of reflection than it may seem.

Best,

Ismael

[Paul Phillips]

On Sun, Apr 29, 2012 at 8:32 AM, martin odersky <martin....@epfl.ch> wrote:
> I agree, and that's what code review needs to guard against.

Why do I want another thing to guard against? Why does the rest of the
world? (We are talking about boolean arguments *in API* which implies
inflicting this guarding requirement on everyone.)

Although it is tempting to burn off some time writing the complete
case against Boolean parameters, I can't bear to use my time that way
right now. Suffice to say that it would be a lengthy screed. Using
booleans to partition behavior among a small set of discrete
alternatives means you forego using one of the best features of
objects - polymorphic dispatch - in order to drizzle your code with
meaningless one-bit carriers. Here's a sample type signature, what
does this tell us?

(Type, Boolean, Boolean) => Type

When you use polymorphism, you don't have to start tacking those
Booleans on as meaningless appendages when you realize you need access
to b in method g. Nor do you find yourself with "invalid states" when
you have three states to cover, and you need the booleans to arrive
true/true, true/false, and false/true, but false/false means nothing.
When you use polymorphism, you already encapsulated data and behavior.
You can extend.

I do recall a before/after where I expunged boolean parameters.

https://github.com/scala/scala/commit/ec233d3db

Samples of the "before" code:

- case IDENTIFIER | BACKQUOTED_IDENT | THIS | SUPER =>
- path(true, false)
...
- t = (t /: annotations(false, false)) (makeAnnotated)
...
- params += typeParam(NoMods.withAnnotations(annotations(true, false)))

Now we can say they should have used named parameters on those trues
and falses until we're blue in the face (even if they didn't exist
yet, doesn't matter) and it won't make the names appear. But even if
we could count on everyone doing that, it's still worse. Booleans
ring of the arbitrary. There's always a better type signature than
(X, Boolean, Boolean) => Y to communicate what is taking place. Have
some pity on the guy who has to read the code.

[Simon Ochsenreither]

 In a language with good enum support (like Java, ironically), enums are probably a better option in almost every case than a boolean parameter.

I agree. Hopefully there will be a good solution for enums in a future Scala release. Maybe macros will help in Scala? (:-D)

[martin odersky]

On Sun, Apr 29, 2012 at 9:47 PM, Simon Ochsenreither <simon.och...@googlemail.com> wrote:

 In a language with good enum support (like Java, ironically), enums are probably a better option in almost every case than a boolean parameter.

I agree. Hopefully there will be a good solution for enums in a future Scala release. Maybe macros will help in Scala? (:-D)

If a domain is a boolean, it's a boolean. Using an enum would be "pompous" (in the words of Niklaus Wirth, with whom I did my PhD). Use the most specific and precise thing that describes your intentions, that's all.

 - Martin

[David Hall]

In a language like Java without named parameters, I think it serves a
useful documentary purpose; who knows what the last parameter in
optimize(init, iterations, ..., true) means?

Scala and other enlightened languages don't need to worry about that, of course.

-- David

[Ismael Juma]

On Mon, Apr 30, 2012 at 1:40 AM, David Hall <dl...@cs.berkeley.edu> wrote:

Scala and other enlightened languages don't need to worry about that, of course.

I think this would have been true if named parameters were required (or could be made required in some situations). I thought we prefer for the compiler to help us when it's easy to do so. :)

Best,

Ismael

[Lukas Rytz]

another use case for the dearly missed style checker plugin.

[Mirco Dotta]

another use case for the dearly missed style checker plugin.

Why dearly missed? check out https://github.com/scalastyle/scalastyle 

There is also an Eclipse plugin for it and it was presented at ScalaDays this year (unfortunately the video is not yet available)

http://skillsmatter.com/podcast/scala/coding-with-style-the-scalastyle-style-checker

On Mon, Apr 30, 2012 at 02:50, Ismael Juma <ism...@juma.me.uk> wrote:

On Mon, Apr 30, 2012 at 1:40 AM, David Hall <dl...@cs.berkeley.edu> wrote:

Scala and other enlightened languages don't need to worry about that, of course.

I think this would have been true if named parameters were required (or could be made required in some situations). I thought we prefer for the compiler to help us when it's easy to do so. :)

Best,

Ismael

---------------

Mirco Dotta

Typesafe - The software stack for applications that scale

PSE-D, 1015 Lausanne, Switzerland

Work: +41 (0)21 691 49 65

Twitter: @mircodotta

[Adriaan Moors]

scalastyle is great because it is fast, but I don't think it can do this kind of analysis accurately.

(it only does syntactic checks; it does not run the type checker)

[Mirco Dotta]

Right, it wouldn't be able to do it in general.  But it would still be a useful rule to never let a boolean value (i.e., true or false) to be passed as argument.

And there is also a Linter Compiler Plugin 

https://github.com/jorgeortiz85/linter

But it would be much slower...

[Mirko Stocker]

On Monday 30 April 2012 14:47:39 Mirco Dotta wrote:
> Right, it wouldn't be able to do it in general. But it would still be a
> useful rule to never let a boolean value (i.e., true or false) to be passed
> as argument.

FYI, I'm planning to add this to Scala IDE's upcoming code-analysis. Right
now, the biggest hurdle is reliably detecting named-arg calls after they've
been desugared (I'm doing this in scala-refactoring, but there I have position
information, which makes it a bit easier).

Cheers,

Mirko

--
Mirko Stocker | m...@misto.ch
Work: http://ifs.hsr.ch | http://infoq.com
Personal: http://misto.ch | http://twitter.com/m_st

[Eugene Burmako]

On a partially related note, will code analysis be usable as a standalone library?

[Mirko Stocker]

On Monday 30 April 2012 14:57:13 Eugene Burmako wrote:
> On a partially related note, will code analysis be usable as a standalone
> library?

Yes it will (it's a compiler plug-in), but it comes with IDE integration and
even quickfixes (that was my main motivation to start the project). At the
moment, my focus is on getting it to work inside the Scala IDE.

At the moment, it detects class/filename mismatches (directory/package-
structure mismatches are coming soon), non-local returns (inspired by Miguel's
Scala Days talk), unused imports and pattern-matches that could be turned into
map/flatMap, etc. calls, see [1] for more information.

Next will be named-boolean args, and whatever I can come up with :-)
(something with shadowing of names would be interesting as well).

Cheers,

Mirko

[1] http://misto.ch/eliminating-pattern-matching/

[Mirko Stocker]

On Mon, Apr 30, 2012 at 2:55 PM, Mirko Stocker <m...@misto.ch> wrote:
> FYI, I'm planning to add this to Scala IDE's upcoming code-analysis. Right

> now, the biggest hurdle blablabla..

It wasn't so hard after all, and it even comes with a quickfix:

http://misto.ch/detecting-and-naming-boolean-parameters/