ToolBox(Factory).inferImplicitView or alternatives?

[Miles Sabin]

Hi folks,

I'm attempting to use reflection to resolve implicit values relative
to reflected Types at runtime. Something like the following seems like
it ought to work,

import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox

implicit val li = List(13, 23)

val toolbox = currentMirror.mkToolBox()

val iv = toolbox.inferImplicitValue(
appliedType(definitions.ListClass.asType,
List(definitions.IntClass.asType)))
showRaw(iv)

However this fails at runtime with a scala.NotImplementedError thanks
to this promissory note,

https://github.com/scala/scala/blob/master/src/compiler/scala/tools/reflect/ToolBoxFactory.scala#L332

Is there any ETA for an implementation of this arriving or,
alternatively, is there any way of achieving the same effect?

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
g+: http://www.milessabin.com
http://twitter.com/milessabin
http://www.chuusai.com

[Eugene Burmako]

Hi Miles,

First of all, your call to inferImplicitValue can be simplified to just `inferImplicitValue(typeOf[List[Int]])`. This is a shortcut I added in M4, and I like it a lot :)

Secondly, your approach won't work, unfortunately. When typechecking/running stuff, toolbox creates a new scope, disconnected from the original scope of the call site. Free terms are an exception, but in the grand scheme of things, this is just a hack. To do it right, we'd need reification of scopes, which is, well, I digress. 

Bottom like is that even if `toolbox.inferImplicitValue` were implemented, it wouldn't find `li`.

There's good news though. Quite possibly, you'll be fine with either a macro (which does have access to scopes and has access to the implemented and tested c.inferImplicitValue).

If you do need to dispatch at runtime, the situation is grim. If you direly need this functionality, though, you could write a macro that extracts necessary info about declared implicit values from scopes, reifies it, and then you could use the reified stuff at runtime. This isn't going to be easy, but it's possible.

Maybe you could explain your use case in more detail, so that we could think of a solution? Also, if, having read all that, you still need me to implement `toolbox.inferImplicitValue` asap, please, let me know and I'll give it a shot.

Cheers,

Eugene

[Miles Sabin]

Hi Eugene,

> Secondly, your approach won't work, unfortunately. When typechecking/running
> stuff, toolbox creates a new scope, disconnected from the original scope of
> the call site. Free terms are an exception, but in the grand scheme of
> things, this is just a hack. To do it right, we'd need reification of
> scopes, which is, well, I digress.

I'd been looking at migrating the code from macros to reflection and
had noticed that there didn't appear to be any immediately obvious way
to connect the scopes.

> Bottom like is that even if `toolbox.inferImplicitValue` were implemented,
> it wouldn't find `li`.

:-(

> There's good news though. Quite possibly, you'll be fine with either a macro
> (which does have access to scopes and has access to the implemented and
> tested c.inferImplicitValue).
>
> If you do need to dispatch at runtime, the situation is grim.

I really do need to dispatch at runtime.

> If you direly
> need this functionality, though, you could write a macro that extracts
> necessary info about declared implicit values from scopes, reifies it, and
> then you could use the reified stuff at runtime. This isn't going to be
> easy, but it's possible.

This is a shame. On the face of it, I could generate source at
runtime, successfully compile it using scalac and load the resulting
bytecode to get the same effect ... what makes it so much harder to do
the equivalent via reflection?

Or is the problem just that the implicits in my sample are locally
scoped? What if they were available via a stable path?

> Maybe you could explain your use case in more detail, so that we could think
> of a solution? Also, if, having read all that, you still need me to
> implement `toolbox.inferImplicitValue` asap, please, let me know and I'll
> give it a shot.

The application is basically lightweight staging. I have code that
depends on static resolution of implicits which I would like to be
able to use by resolving Type's at runtime, and parametrizing the
implicit-using code via reflection's runtime compilation ... does that
make sense? I think that this is a very powerful and potentially quite
widely applicable technique which blends static and dynamic typing in
a novel (at least in mainstream languages) and interesting way.

I'd already made a start at moving the implementation of
inferImplicitValue over to reflection from macros, and I'll carry on
with that. If you do get a chance to look at it as well that'd be
incredibly helpful, but I don't want you to interrupt your holiday :-)

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin
g+: http://www.milessabin.com
http://twitter.com/milessabin

http://underscoreconsulting.com
http://www.chuusai.com

[Miles Sabin]

On Fri, Jun 22, 2012 at 11:01 AM, Miles Sabin <mi...@milessabin.com> wrote:
>> If you direly
>> need this functionality, though, you could write a macro that extracts
>> necessary info about declared implicit values from scopes, reifies it, and
>> then you could use the reified stuff at runtime. This isn't going to be
>> easy, but it's possible.
>
> This is a shame. On the face of it, I could generate source at
> runtime, successfully compile it using scalac and load the resulting
> bytecode to get the same effect ... what makes it so much harder to do
> the equivalent via reflection?
>
> Or is the problem just that the implicits in my sample are locally
> scoped? What if they were available via a stable path?

Answering my own question, this does appear to work if the implicits
in question are visible via a stable path. With the compiler patched
to implement ToolBox.inferImplicitValue the following,

package foo {
trait Foo[T, U]

object Foo {
implicit def mkFoo[T, U] = new Foo[T, U] {}
}
}

package bar {
object Test extends App {

import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox

val toolbox = currentMirror.mkToolBox()

val iv = toolbox.inferImplicitValue(
appliedType(

typeOf[foo.Foo[_, _]],
List(
definitions.IntClass.asType,
definitions.StringClass.asType)))

println(showRaw(iv))
}
}

compiles and produces,

TypeApply(Select(Select(This(newTypeName("foo")), foo.Foo),
newTermName("mkFoo")), List(TypeTree(), TypeTree()))

which looks like it's getting pretty much to where I want it to be :-)

[Eugene Burmako]

Yeah, if the implicits are locatable (official term for this in reflection internals) then there won't be a problem.

By the way if you have a patch for `inferImplicitValue`, you can submit it to scalamacros/kepler as a pull request :)

[Miles Sabin]

On Fri, Jun 22, 2012 at 3:05 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Yeah, if the implicits are locatable (official term for this in reflection
> internals) then there won't be a problem.

Yay! :-)

> By the way if you have a patch for `inferImplicitValue`, you can submit it
> to scalamacros/kepler as a pull request :)

Sure ... I'm just doing the same for inferImplicitView and will send
you a pull request ASAP.

[Eugene Burmako]

Your technique sounds very interesting. Can I take a look at the sources?

Speaking of philosophical differences between macros and runtime compilation there are a few. First of all, non-locatable symbols don't get pickled, hence, some of the information will be lost between stages.

Secondly, a lot of things (e.g. implicits) depend on the hierarchy of scopes that gets built internally during the typer phase and then gets discarded (i.e. doesn't get pickled). This is a problem even inside the compiler itself, because during later phases (e.g. in patmat) it's impossible to perform implicit resolution since the scopes are gone.

Bottom line is that runtime compilation can achieve the majority of things that regular compilation can do, but there are some fundamental problems that prevent things to be 100% smooth.

[Miles Sabin]

I wrote,

> compiles and produces,
>
> TypeApply(Select(Select(This(newTypeName("foo")), foo.Foo),
>  newTermName("mkFoo")), List(TypeTree(), TypeTree()))
>
> which looks like it's getting pretty much to where I want it to be :-)

Unfortunately if I then try and run this tree,

val ivb = toolbox.resetAllAttrs(iv)
val v = toolbox.runExpr(ivb)
println(v)

I get the following error at runtime,

scala.tools.reflect.ToolBoxError: reflective toolbox has failed:
cannot operate on trees that are already typed

Is there some other incantation necessary to compile a tree that's
been inferred in this way?

Alternatively, I'd actually prefer to be able to do this in a single
step rather than infer the implicit and then evaluate it. I've tried
things along the lines of,

package foo {
trait Foo[T, U]

object Foo {

implicit val is = new Foo[Int, String] {
override def toString() = "Foo[Int, String]"

}
}
}

package bar {
object Test extends App {
import scala.reflect.runtime.universe._
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox

val toolbox = currentMirror.mkToolBox()

val tree =
Apply(
TypeApply(
Select(
Select(
Ident(newTermName("scala")),
newTermName("Predef")),
newTermName("implicitly")),
List(
AppliedTypeTree(
Select(
Ident(newTermName("foo")),
newTypeName("Foo")),
List(
Select(
Ident(newTermName("scala")),
newTypeName("Int")),
Select(
Select(
Ident(newTermName("java")),
newTermName("lang")),
newTypeName("String")))))), Nil)

val rv = toolbox.runExpr(tree)
println(rv)
}
}

Here I get a runtime error,

scala.tools.reflect.ToolBoxError: reflective compilation has failed:

not enough arguments for method implicitly: (implicit e:
foo.Foo[Int,String])foo.Foo[Int,String].
Unspecified value parameter e.

So it looks as though in the inferImplicitValue case typechecking goes
too far to enable subsequent compilation and execution, and in the
second case it doesn't do enough work to resolve the implicit
arguments. Any ideas?

[Eugene Burmako]

Could you, please, submit a patch that adds inferImplicitValue to
toolboxes, so that I can take a look at the problem #1?

As to problem #2, it was very simple. Just remove the outermost apply.
Invocations of nullary methods don't need Apply nodes. Scala was
trying to treat Apply(<implicitly>, Nil) as a call to a method with an
empty parameter list, hence the error.

[Miles Sabin]

On Sat, Jun 23, 2012 at 4:26 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Could you, please, submit a patch that adds inferImplicitValue to
> toolboxes, so that I can take a look at the problem #1?

The commit is here,

https://github.com/milessabin/scala/commit/5eee47bd9945013ac0e92721432a628852227da1

It's relative to scala/scala rather than scalamacros/kepler so your
best bet is probably to cherry-pick it.

> As to problem #2, it was very simple. Just remove the outermost apply.
> Invocations of nullary methods don't need Apply nodes. Scala was
> trying to treat Apply(<implicitly>, Nil) as a call to a method with an
> empty parameter list, hence the error.

Doh! ...

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
gtalk: mi...@milessabin.com
skype: milessabin

[Eugene Burmako]

Aha, I see. The problem is that `inferImplicitView` needs to allow for free symbols, and both infers share some code with `typeCheck`, which I would like to eliminate. If that's not urgent for you, I'd take the time to come up with a revised implementation.

[Miles Sabin]

On Thu, Jun 28, 2012 at 3:06 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> Aha, I see. The problem is that `inferImplicitView` needs to allow for free
> symbols, and both infers share some code with `typeCheck`, which I would
> like to eliminate. If that's not urgent for you, I'd take the time to come
> up with a revised implementation.

There's no rush on this ... take your time :-)

[Eugene Burmako]

https://github.com/scala/scala/commit/ddcba109843d4f665a010f3dbbd28a6b99e6185a

[Miles Sabin]

On Thu, Aug 2, 2012 at 5:10 PM, Eugene Burmako <eugene....@epfl.ch> wrote:
> https://github.com/scala/scala/commit/ddcba109843d4f665a010f3dbbd28a6b99e6185a

Fantastic ... thanks :-)

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528

skype: milessabin
gtalk: mi...@milessabin.com