Idea: Dynamic implicits and dyanmic scopes

[Roman Levenstein]

Hi,

I'm wondering if the following idea makes any sense for you. The idea is still at its early stage and not very detailed yet, but hopefully it provides enough information to understand what I have in mind.

To my understanding, Scala's implicit arguments are currently resolved at compile-time using the static scope surrounding the place of invocation. This is very convenient in many cases. But it still requires you to explicitly define those implicits (Sounds really funny :-)

Now, what if you don't know in advance which implicits may be required by a method that is to be invoked? E.g. you only get at run-time the information which classes/methods are to be used and invoked. Or even if the decision is taken statically but has a lot of alternatives, you may be don't know for sure which implicits will be required for real, when a concrete invocation is done.

Since you don't know which implicits may be required, you have two alternatives:
 (1) You declare a lot implicits for all possible cases (if you statically know all potential methods that may be invoked) and then Scala compiler would pick the right ones.
       This works only for static cases but may be very annoying as you have to foresee all possible implicits.

 (2) A new feature called "Dynamic implicits" could be introduced.
      Dynamic implicits are resolved at run-time. The implementation could use e.g. reflection mechanisms. When a method needs to be invoked and this method has implicit parameters, then it would be possible to provide it with dynamic scope, which should be used to resolve implicits. Each such dynamic scope may be a set of triples (variable name, class/type information, value). So, when there is a reflection-based (similar to java.lang.reflection.Method) Method m and a dynamic scope s, then m.invoke(s, obj, args) API could be used to perform the invocation. When this form is used, "invoke" will use the scope s to resolve implicits, obtain their values and pass them as corresponding arguments.

Another application of this idea could be somewhat different. Imagine that you have a system where you have many different implementations of the semantically same method, i.e. a lot of classes which implement a method with the same name, but different sets of explicit and implicit parameters. An example could be, let's say an "authenticate" method. At the high-level the system needs to authenticate a user - how it is done is a secondary question. There could be different implementations for different authentication systems. One such implementation may require username, password as parameters. Another one may require public keys. Third one may required Kerberos credentials/tokens and so on. Now, it may happen that you don't know in advance, which of those implementations has to be used, because it is based on the user of the system or mode of operation which is decided only at run-time. More over, it may happen that some of the users have only username/password credentials, whereas others have only public keys and Kerberos. How do you implement such a thing? One idea could be to use dynamic scope and dynamic implicits again. Using them, one could do roughly the following:

 - get a user name at run-time

 - read from a DB or a config file authentication information available for a given user. It could be username/password credentials, public keys, Kerberos info, etc. Corresponding typed objects are then created from this information.

- such authentication-related objects built using the user information are put into a dynamic scope s

- There is a new API ( let's call it resolveAndInvoke(scope: DynamicScope, candidates: Set[Method]) ) that takes the dynamic scope s and a set of methods/classes that are potential candidates to be used for authentication (e.g. the set would consist of authenticateKerberos(implicit kerberos:KerberosAuthenticationInformation),  authUserNamePassword(username: String, password: String), publicKeyAuthentication(publicKey: PublicKeyAuthenticationInformation)). The API tries to match candidates against the dynamic scope s to see if all required explicit and implicit parameters can be provided by the scope s (and may be current static scope?). If this is the case, then this candidate is taken and invoked. (In the future, one could also think about the case, where multiple candidates match. In such a case, an additional policy or mechanism could be introduced to pick one of them for invocation) This way, the selection of the method to be invoked happened at run-time and was decided based on the dynamic information available through a dynamic scope.

I hope I managed to provide some motivation for the proposed feature. The overall wish was to make implicits more usable in dynamic contexts and to introduce some sort of a very dynamic late binding for method selection and invocation.

Questions:
- Are described use-cases and the proposed solution interesting enough to be discussed? Or do you say that described scenarios are so seldom that it is not worth it to introduce any new mechanisms to solve them? Or may be they can be solved in a very elegant way using already existing solutions and mechanisms?

- Do you see further use-cases where the proposed features could be beneficial?

- Would it be technically possible to implement something like this for Scala or are there any principal show-stoppers (e.g. limitation due to JVM internals, etc)?

Any feedback is very appreciated!

Thanks,
   Roman

[Eugene Burmako]

The notion you're discussing (namely, resolveAndInvoke) can be
realized in 2.10 with runtime compilation via toolboxes. Even
2.10.0-M3 is powerful enough to implement it.

The huge downside would be speed. Currently runtime compilation costs
a lot, and I can't immediately see how to trivialize this overhead.

Would be great to see some progress in the area of runtime
compilation, though. It's tantalizing to have a reflection framework
unified across stages and not being able to use it in unified manner
due to implementation details.

[Daniel Sobral]

> I hope I managed to provide some motivation for the proposed feature.

Actually, that's the thing I failed to see in the whole text: a use
case. You talked about selecting an authentication method, which has
absolutely no need for the mechanism you mentioned, since the static
implicits are enough:

def auth(user: String)(implicit authenticator: Authenticator): Boolean = ???

abstract class Authenticator { def authenticate(user: String): Boolean }
object Authenticator {
implicit def getAuthenticator: Authenticator = /* check available
mechanisms and return one */ ???
}

In fact, I fail to see how dynamic implicits could help you here: if
more than one authentication is found, it would simply fail. You'd
need some code to check which of the declared ones is actually
available, then chose or compose them. None of that is part of
implicit resolution.

Now, I'm not just dismissing the whole notion, but I think it's a
solution looking for a problem. I'd rather see more and better
examples.

--
Daniel C. Sobral

I travel to the future all the time.

[Luke Vilnis]

My immediate thought of a use-case for this is dependency injection - DI containers are often used to simulate a dynamic environment, and the fundamental problem with using implicits for DI is that implicits are static. An example is having 2 different containers passed into a constructor at the same static call site, chosen depending on some runtime state. 

[Roman Levenstein]

On Wednesday, May 30, 2012 7:15:27 PM UTC+2, Luke Vilnis wrote:

My immediate thought of a use-case for this is dependency injection - DI containers are often used to simulate a dynamic environment, and the fundamental problem with using implicits for DI is that implicits are static. An example is having 2 different containers passed into a constructor at the same static call site, chosen depending on some runtime state. 

Yes, dependency injection is a good example.

 

On Wed, May 30, 2012 at 1:03 PM, Daniel Sobral  wrote:

> I hope I managed to provide some motivation for the proposed feature.

Actually, that's the thing I failed to see in the whole text: a use
case. You talked about selecting an authentication method, which has
absolutely no need for the mechanism you mentioned, since the static
implicits are enough:

I agree that for this example my solution would be eventually an overkill and there are other easier solutions. But this was just an example, to explain the idea.

 

def auth(user: String)(implicit authenticator: Authenticator): Boolean = ???

abstract class Authenticator { def authenticate(user: String): Boolean }
object Authenticator {
 implicit def getAuthenticator: Authenticator = /* check available
mechanisms and return one */ ???
}

In fact, I fail to see how dynamic implicits could help you here: if
more than one authentication is found, it would simply fail. You'd
need some code to check which of the declared ones is actually
available, then chose or compose them. None of that is part of
implicit resolution.

Well, I'd like that this "some code to check ..., then chose or compose them" is not required to be written by hand and limited to static information. I want it to be more dynamic and more adaptive to run-time.

 

Now, I'm not just dismissing the whole notion, but I think it's a
solution looking for a problem. I'd rather see more and better
examples.

Let's try.

Another example is like this: Imagine you build a system out of existing components/sub-systems/plugins, where each such component/subsystem or plugin is probably developed by a different party and not all of interfaces are fixed statically and known in advance. Some of them can be even introduced later in time, after the original system was created. Eventually, you even need to do dynamic discovery of components (e.g. using some constraints like types, dependency on other components, implicits etc)  in some sort of runtime repository (e.g. something like UDDI or OSGI service discovery) to figure out which ones are available at the moment and can be satisfied using dynamic scopes and other available components. You want to be able to compose a more complex system out of these parts using minimal manual interventon. If possible, all this should happen completely automatically. If this would be possible, one can imagine systems that are much easier to maintain and that are able to assmeble themselves dynamically out of currently available parts or even reconfigure themselves dynamically reacting to changes in runtime.

- Roman

[Roman Levenstein]

On Wednesday, May 30, 2012 5:16:00 PM UTC+2, Eugene Burmako wrote:

The notion you're discussing (namely, resolveAndInvoke) can be
realized in 2.10 with runtime compilation via toolboxes. Even
2.10.0-M3 is powerful enough to implement it.

Interesting. I'll look into it.

 

The huge downside would be speed. Currently runtime compilation costs
a lot, and I can't immediately see how to trivialize this overhead.

I agree that proposed features may result in quite some overhead.

On the other hand, I'd say that it would be very optimistic to expect a comparable performance from such complex features. 

Also, for many typical use-cases (dependency injection, dynamically assembling systems from discovered components at run-time) you probably do not use it as often as regular invocations. Therefore, up to a certain limit this overhead is probably acceptable.

And, may be some caching could be used? E.g. once you peformed expensive resolution step, you could remember the decisions and next time simply reuse it, assuming/checked that used implicit dependencies are still available in the dynamic scope.

 

Would be great to see some progress in the area of runtime
compilation, though. It's tantalizing to have a reflection framework
unified across stages and not being able to use it in unified manner
due to implementation details.

Absolutely! 

[Eugene Burmako]

You could keep the compiler instance in memory eliminating startup overhead. As to user-guided caching of implicit resolution, I'm afraid this is impossible right now.

[Niels Hoogeveen]

"Would be great to see some progress in the area of runtime
compilation, though. It's tantalizing to have a reflection framework
unified across stages and not being able to use it in unified manner
due to implementation details."

It would be a huge step forward if Scala were not only the most
powerful staticially compiled language on the JVM, but also a really
powerful dynamic language.

If only we could do mix-in composition on runtime.

[Eugene Burmako]

Some dynamic mixinning can already be done with macros:
http://stackoverflow.com/questions/10373318/mixing-in-a-trait-dynamically/10387200#10387200.

Sure it doesn't cover all use cases (those can be implemented with
toolboxes, yet with bad performance), but that's a start.

[richard emberson]

I've been doing something like this for about a year.
In my case, a base object has 7 mixins, Each
mixin has from 3 to 8 variants making a total
of 108864 possible mixins combinations for a base object.
Not only does it make no sense to actually generate
code for each possible combination, having 108864 such
classes, I imagine is a significant overhead - with or
without macro support.
So, after the most general base object is created, which
variants of each mixin is determine to best optimize
performance/memory/etc. and, using the BIC (Build-In-Compiler),
I generate that particular version and copy the general
object's data into the particularized version.
(Actually, I create a builder for that version which is
stored in a cache so it can be reused if that combination
of mixin variants is required again - which is very
often the case.
Also, Note that a base object lives as long as the application
is running and the application is an enterprise level app
and expected to run for weeks, months or more.)

Richard

--
Quis custodiet ipsos custodes

[Niels Hoogeveen]

Thanks Eugene,

I skimmed the code you link to. While this is an interesting concept,
it looks to me, the mixin composition still takes place at run time.

How would we dynamically mixin two traits where we only know the class
names at runtime?

On May 30, 11:41 pm, Eugene Burmako <xeno...@gmail.com> wrote:
> Some dynamic mixinning can already be done with macros:http://stackoverflow.com/questions/10373318/mixing-in-a-trait-dynamic....

[Niels Hoogeveen]

While this is an interesting concept, it looks to me, the mixin
composition still takes place at run time.

Should read:

While this is an interesting concept, it looks to me, the mixin

composition still takes place at compile time.

[Roman Levenstein]

Hi Richard,

On Thursday, May 31, 2012 12:24:01 AM UTC+2, Richard Emberson wrote:

I've been doing something like this for about a year.
In my case, a base object has 7 mixins, Each
mixin has from 3 to 8 variants making a total
of 108864 possible mixins combinations for a base object.
Not only does it make no sense to actually generate
code for each possible combination, having 108864 such
classes, I imagine is a significant overhead - with or
without macro support.

Very good point! I forgot to mention potential explosion of code size in many cases, where static approach covering all potential cases would be used.

 

So, after the most general base object is created, which
variants of each mixin is determine to best optimize
performance/memory/etc. and, using the BIC (Build-In-Compiler),
I generate that particular version and copy the general
object's data into the particularized version.
(Actually, I create a builder for that version which is
stored in a cache so it can be reused if that combination
of mixin variants is required again - which is very
often the case.
Also, Note that a base object lives as long as the application
is running and the application is an enterprise level app
and expected to run for weeks, months or more.)

Richard

Can you make the code or at least the relevant snippets available to others to see how you do it?

-Roman

[Eugene Burmako]

Sure it happens at compile-time, but it allows stuff similar to "new value with T", which is kinda dynamic.

Of course, as I mentioned, this doesn't cover all the use cases of dynamic mixinning, including the one you've mentioned in the previous message.

[Roman Levenstein]

Hi Eugene,

On Wednesday, May 30, 2012 10:05:27 PM UTC+2, Roman Levenstein wrote:

On Wednesday, May 30, 2012 5:16:00 PM UTC+2, Eugene Burmako wrote:

The notion you're discussing (namely, resolveAndInvoke) can be
realized in 2.10 with runtime compilation via toolboxes. Even
2.10.0-M3 is powerful enough to implement it.

Interesting. I'll look into it.

I looked into the runtime compilation via toolboxes, as you suggested. While it is possible to use it for realizing the proposed features, I'm wondering if it is an overkill for this task. After all, for the features that I originally proposed (i.e. not including dynamic mix-ins) we don't need to compile any free-form code with methods, etc. We also don't need to create mixins dynamically. We essentially only need matching of (implicit) parameters provided by dynamic scopes against a set of method declarations obtained e.g. via reflection.

Remark: In a more complex case one could of course start thinking about implicit conversions when doing a dynamic method selection by means of matching. Then things will get more complicated and may indeed require a full-blown compiler. But then it is a question, if such a power is required in most cases.

-Roman

[nafg]

You may be insterested in https://github.com/marcusatbang/Hooks