[SIP-21] Suggestions for improving SIP-21 - Spores
Jim Powers
I think this SIP (SIP-21) is a good start, but the guarantees and safety can be improved. I would like to highlight the following as considerations for improving the specification for spores. The areas I will address are:
- Compile-time guarantees
- Function lifting
- Run-time sanity checks
Compile-time guarantees
As things stand, the current proposal provides very weak guarantees about correct behavior of spores at run-time. I think this situation can and should be improved. For instance, the current specification allows the following construct with no error:
class Bar(file:File) {
private val stream:FileInputStream = // ...
val takeN = spore {
val localStream = stream
(n:Int) => {
val r = new Array[Byte](n)
localStream.read(r)
r
}
}
}This can be improved.
What I would propose is that values captured by spores be required to have a specific typeclass associated with their type (this idea is stolen from Cloud Haskell) that constrains what can be captured sanely. For the purpose of this discussion I'll use a typeclass called Sporable. In discussions with Philipp Haller he suggested going directly to Pickleable (based on the new Scala pickling system) which may be fine, I don't want to get too bogged down in serialization details at this point. The issue is that the typeclass needs to be more than a mere marker - you should be required to provide useful functionality necessary for the correct operation of a spore. In particular, in a non-local (distributed) environment. A meaningful serialization/deserialization implementation would be a good start.
So, the implementation of the spore macro would examine all of the types of the captured values and search through implicit scope for an implementation of an appropriate typeclass instance of Sporable for the target type. If no such instance can be found then fail in the usual ways.
Sporable typeclass instances can be provided for all scala.collection.immutable types. Users can provide their own instances of Sporable for their own types. Yes, this means that if someone wants to pretend that they can correctly serialize a FileInputStream or an object behind an arbitrary trait/interface then more power to them.
Function lifting
Consider the following code:
def foo[I,O](f:I => O) = spore {
val localF = f
(i:I) => f(i)
}While one may be quick to point out that for safety higher-order-functions should transitively require Spore types I think this is too restricting. I propose something lighter-weight that, for instance, does not require changing existing interface types in order to work with spores. In particular, I propose a @sporable attribute that can be used on methods, functions. For example:
@sporable
def doubleString(i:String):String = i+iWould invoke a compile-time check to indicate that the method doubleString is, in fact, a "sporable" function. In the context of a method/function this would mean that the function body abides by more-or-less the same rules as a spore body: the only values captured are arguments, all other values are introduced in the body of the function/method, arguments to the function must have a Sporable typeclass instance in scope, and function arguments must be tagged @sporable or be of a Spore type. The type of the method would still be Function1[String,String].
The original HOF can now be written:
def foo[I,O](f:I => O @sporable) = spore {
val localF = f
(i:I) => f(i)
}Which can be used at call sites to ensure that the only functions passed to foo are of type Spore[?,?] or tagged as @sporable essentially enabling safe lifting into the spore context.
I suspect that implementing this will require macro annotations.
Run-Time Sanity checks
Since what a spore generates during serialization can be a binary blob of JVM bytecode and data it may be useful to indicate limits on exactly how big these blobs can get. I would like to propose a way to produce warnings and errors when the run-time properties of a serialized spore exceed thresholds. For example:
def foo[I,O](f:I => O @sporable) = spore(warn = 1 kilobyte) {
val localF = f
(i:I) => f(i)
}
def foo[I,O](f:I => O @sporable) = spore(error = 2 kilobytes) {
val localF = f
(i:I) => f(i)
}
def foo[I,O](f:I => O @sporable) = spore(warn = 1 kilobyte, error = 2 kilobytes) {
val localF = f
(i:I) => f(i)
}This would cause the implementation of spore to produce useful information when user-defined thresholds are met. A further refinement would be that a specific logging object can be supplied for where to log these messages.
def foo[I,O](f:I => O @sporable) = spore(warn = 1 kilobyte, log = myLogger) {
val localF = f
(i:I) => f(i)
}Another useful feature would be to just log generated sizes:
def foo[I,O](f:I => O @sporable) = spore(logSize = true, log = myLogger) {
val localF = f
(i:I) => f(i)
}
Eugene Burmako
--
You received this message because you are subscribed to the Google Groups "scala-sips" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-sips+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Jim Powers
Simon Ochsenreither
I think it is especially important to figure out the relationship to Function more clearly, because I'd like to avoid ending up i a state were everyone suggests to ignore plain lambdas/Functions and to always pick spores.
(The naming is another issue ... I think Pickles & Spores is cute, but users shouldn't need to be aware of the inner workings to understand the pun.)
The basic question I have is: why are we picking such a seemingly disruptive way with the spores block + val-defs ... isn't there something more C++-like where you can just specify what and how you want things to be captured?
Johannes Rudolph
I think it is especially important to figure out the relationship to Function more clearly, because I'd like to avoid ending up i a state were everyone suggests to ignore plain lambdas/Functions and to always pick spores.
I think this is a valid and realistic concern. Actually, the SIP says too little about where you would use the Spore type in practice. Would you have libraries where some functions would require Spores? Why would you do that? Especially, when you can't pass any kind of regular functions into that library method any more.
I don't say the spore construct itself is useless. You can use it to make sure not to accidentally capture things you don't want to but make all captures explicit. But that's it.
This seems like the lesser problem of the ones you usually have with capturing, the biggest one that you are capturing a reference to something which isn't allowed to leave the scope. So, IMO closing over something accidentally is much less interesting than preventing references to some values from escaping the scope. Speaking in your examples:
Actors: The closure isn't the problem, the problem is that the `this` reference to a mutable instance is allowed to leave the scope. Spores don't help you to decide which part of the expression you actually want to capture and which not (just `this` or `this.sender`).
Serialization: Again you are closing over `this`. For serialization purposes you don't want a closure to reference a Main instance because it isn't serializable. However, again, spores don't help you to decide which value you actually want to capture. Is it `this`? Why not? Or `this.helper`? Or `this.helper.toString()` or an even bigger expression?
Johannes
-----------------------------------------------
Johannes Rudolph
http://virtual-void.net
Johannes Rudolph
def foo[I,O](f:I => O @sporable) = spore { val localF = f (i:I) => f(i) }
Jim Powers
On Wed, Jun 19, 2013 at 5:22 PM, Jim Powers <j...@casapowers.com> wrote:
If spores are about serializability why would you be required to write "val localF = f" if you have already annotated the parameter f that it is @sporable? Maybe you could even remove the requirement to write out the captures explicitly but instead require some static quality that says that this value (or values of a certain kind) are safe to be captured in this context.def foo[I,O](f:I => O @sporable) = spore { val localF = f (i:I) => f(i) }
Simon Ochsenreither
Heather Miller
--
You received this message because you are subscribed to the Google Groups "scala-sips" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-sips+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
Heather Miller
Doctoral Assistant
EPFL, IC, LAMP
http://people.epfl.ch/heather.miller
-- Please excuse my brevity, sent from mobile device
martin odersky
--
Johannes
-----------------------------------------------
Johannes Rudolph
http://virtual-void.net
--
You received this message because you are subscribed to the Google Groups "scala-sips" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-sips+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967
Chris Marshall
Eugene Burmako
Let me elaborate. Imagine I'm a library author exposing a function Foo with the signature `def foo[T, R](f: Function1[T, R]): R = ...`. Now I want to make use of spores to make sure that `f` satisfies certain properties. So I change the signature to `def foo[T, R](f: Spore1[T, R]): R = ...` as suggested on Twitter and then start thinking.
ode...@gmail.com
Sent from my iPhone
I guess my problem about this is: the example illustrates using spores to make sure you've not captured anything. But you have to explicitly use them! If you are explicitly using them, it's because you know that you need to be careful and are paying attention to detail. If you know you need to be careful, it seems unlikely that you captured something by accident.
Hanns Holger Rutz
On 21 Jun 2013, at 17:17, Johannes Rudolph wrote:
> Part of my confusion seems to come from the problem that it's not clear at all what spores are supposed to be. The SIP says
>
> > The main idea behind spores is to provide an alternative way to create closure-like objects, in a way where the environment is controlled.
>
"Si hay reelección, Capriles será reelecto"
Lukas Rytz
Chris Marshall
Michael Pilquist
Roland Kuhn
martin odersky
Simon Ochsenreither
I think, yes, libraries might demand Spore in their APIs. But I argue that even if they don't, Spore as a concept is still useful. I have heard quite a few stories from people at Foursquare, Twitter, in the Spark project and others where people are well aware that care is required yet still get surprised from time to time by what's captured in a closure. Projects like this might well have a coding standard in the future where certain classes of closures should always be spores, even if the library does not (yet) enforce that.
I agree that the concept is pretty important, but isn't there a more principled way to achieve it?
The use-site syntax feels a lot like a hack and I don't want to imagine the mass-migration of people from Functions to Spores as soon as people realize that non-capturing-by-default might be the thing they prefer.
Rex Kerr
}
}
--
Heather Miller
Thanks a lot for your feedback.
- eliminating some races off the bat (as indicated as examples in the SIP),
- and for other cases where spores cannot eliminate races (because of a lack of an effect system) they at least force you to make the captured environment explicit which would definitely help in debugging a race condition.
On the compile-time guarantees:
When we were discussing SIP-21 in person at ScalaDays we already mentioned to you our ideas about adding type constraints to restrict the types of captured variables (I also mention it in my live talk [1]). However, all of the proposals that we've come up with or that were proposed by others are still experimental at this point. Naturally, something not well-polished should be left out of a SIP. For reasons I'll explain later, I think it's best to keep the current SIP-21 modest, and defer an addition of type constraints to a point when we have gained experience with applying spores in real code.
Adding a `Sporable` (or whatever it would be called) type class is an interesting idea. Though, it seems it might have far-reaching consequences: for all types that are reasonable to capture in a typical spore closure we/someone would have to provide a `Sporable` type class instance; the number of required type class instances might be impractically large. Though, we don't know without doing the experiment. Some people might argue that instead of explicitly enabling capturing, explicitly disabling capturing could be more practical/useful. At least this approach of disabling is along the current lines of thinking for type constraints in spores (meanwhile it's still not clear how to practically achieve this, so I'm not going to go into any more details, yet).
In some situations, requiring the types of captured variables to be `Sporable` might not be enough. For example, to make spores pickleable using the scala-pickling framework, one would need to have type class instances of `Pickler[T]` for all types `T` of captured variables (to be able to pickle the spore's environment). I don't see how this could be expressed in your current proposal.
Right now the situation is that more and more projects use Spark's "closure cleaner" [2] to null out problematic and unnecessary outer references to make sure serializing closures that should normally be serializable does not throw `NotSerializableException` at runtime. The problems with the closure cleaner are:
- errors manifest themselves at runtime
- it relies on scalac compilation details
- it's not clear whether the closure cleaner is complete; there is only anecdotal evidence that it seems to work "in enough cases".
- it helps the library to have better guarantees about the closures that it's dealing with (for example, worries about problematic outer pointers are gone);
- the library enforces that users write code in a clear style where it is appropriate;
- the library enforces the absence of certain errors (one example involving Akka and futures is given in the SIP).
--
You received this message because you are subscribed to the Google Groups "scala-sips" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-sips+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Johannes Rudolph
Heather, thanks for the clarifications.
The problem I still have is that it is not clear how the SIP should help to write correct programs in practice. And IMO that’s mainly a consequence of two things the SIP is unclear about:
- an analysis of the problem, a problem statement and an explanation how the given motivations fit into the problem statement
- a summary description of the solution and how the solution solves the practical problem
(- the “how does the feature integrate with existing features” section which would have revealed practical problems)
The motivating cases from the SIP show that there are issues related to capturing but it doesn’t explain what the exact relation really is.
Futures: What seems to be proposed is to replace the parameter types of Future's higher-order methods to require Spores instead of plain functions. Please try to do that and see what has to be changed in even the simplest Future-based applications. It turns out that being able to close over stuff is one of the big advantages of the Future API. In fact, it's hard to find a legitimate program using Futures that doesn't use capturing. With spores we are basically back in Java-land where closed-over values had to be put into final variables. Actually, it turns out the problem is much more related to the akka Actor API than to Futures. So, why change Futures?
Serialization: Capturing is still possible with spores, as is referencing other un-serializable values in other ways. So, how do spores help?
I’ll try to rephrase what a spore is: Spore is a new type extending the Function type which guarantees (with a static check at construction) that a function value of this type has an additional property. The property seems to be ‘All values captured by this function have been declared explicitly’.
It’s unclear to me how ‘have been declared explicitly’ will solve ‘problems, somehow related to capturing’.
There are some questions the SIP should answer:
- When would I use the Spore type over the regular function type?
- When would I use the spore constructor instead of an anonymous function?
- What are the disadvantages of using the solution?
- How does this new property solve the problem shown with the motivating examples? I want to know in detail why “Have been declared explicitly” will reduce bugs. What are the steps to get from buggy code using some libraries to a fixed version, how have spores helped and how much correct code had to be changed as well to support the solution?
IMO spores are not a conservative solution at all because the solution offered is much broader than the problem itself [1]. Think about what happens if the SIP is implemented like currently proposed: Either it’s an opt-in solution i.e. the developer decides when to use spores, then it won’t be used at all because you don’t know where to use it exactly to prevent bugs. Or, it’s enforced by libraries in which case all code using the libraries has to be changed. It will be a chore similarly to having to make variables final in Java, which when you do it in a mechanized way won’t help you paying attention to the details that actually are the root of the bugs.
Regarding my earlier comments about why this has to be a SIP: IIUC there’s nothing preventing spores from being just another library for everyone to try out. It’s likely that many of these questions would have come up by now if someone would have tried spores experimentally. A SIP for inclusion into the standard library should IMO be preceded by practical experiments collecting evidence for the utility of the proposed addition (because it's so easy compared to SIPs regarding compiler changes).
Here’s a simple application using Futures I tried to convert to use spores (assuming that all higher-order functions in `Future` are changed to require spores):
https://github.com/jrudolph/spray/commit/8faa2a6616b3a69e374a41ce3b8be9ff33b35b14
Immediate questions:
* for-comprehensions: do they work together with spores anyhow?
* nested spores, what are the rules?
Johannes
[1] Whatever the problem really is, I find it hard to describe. Many cases seem related to capturing `this` and then exporting code which calls instance methods from inside the class context, and therefore violating the encapsulation classes usually provide. That seems to fit the Actor.sender case but less the serialization case where the problem seems to be that it’s hard to control which implicitly created instances contain outer references and whereto exactly.
Heather Miller
Heather, thanks for the clarifications.
The problem I still have is that it is not clear how the SIP should help to write correct programs in practice. And IMO that’s mainly a consequence of two things the SIP is unclear about:
- an analysis of the problem, a problem statement and an explanation how the given motivations fit into the problem statement
- a summary description of the solution and how the solution solves the practical problem
(- the “how does the feature integrate with existing features” section which would have revealed practical problems)
The motivating cases from the SIP show that there are issues related to capturing but it doesn’t explain what the exact relation really is.
Futures: What seems to be proposed is to replace the parameter types of Future's higher-order methods to require Spores instead of plain functions.
Please try to do that and see what has to be changed in even the simplest Future-based applications. It turns out that being able to close over stuff is one of the big advantages of the Future API. In fact, it's hard to find a legitimate program using Futures that doesn't use capturing.
With spores we are basically back in Java-land where closed-over values had to be put into final variables. Actually, it turns out the problem is much more related to the akka Actor API than to Futures. So, why change Futures?
Serialization: Capturing is still possible with spores, as is referencing other un-serializable values in other ways. So, how do spores help?
I’ll try to rephrase what a spore is: Spore is a new type extending the Function type which guarantees (with a static check at construction) that a function value of this type has an additional property. The property seems to be ‘All values captured by this function have been declared explicitly’.
It’s unclear to me how ‘have been declared explicitly’ will solve ‘problems, somehow related to capturing’.
There are some questions the SIP should answer:
- When would I use the Spore type over the regular function type?
- When would I use the spore constructor instead of an anonymous function?
- What are the disadvantages of using the solution?
- How does this new property solve the problem shown with the motivating examples? I want to know in detail why “Have been declared explicitly” will reduce bugs. What are the steps to get from buggy code using some libraries to a fixed version, how have spores helped and how much correct code had to be changed as well to support the solution?
IMO spores are not a conservative solution at all because the solution offered is much broader than the problem itself [1]. Think about what happens if the SIP is implemented like currently proposed: Either it’s an opt-in solution i.e. the developer decides when to use spores, then it won’t be used at all because you don’t know where to use it exactly to prevent bugs. Or, it’s enforced by libraries in which case all code using the libraries has to be changed. It will be a chore similarly to having to make variables final in Java, which when you do it in a mechanized way won’t help you paying attention to the details that actually are the root of the bugs.
Regarding my earlier comments about why this has to be a SIP: IIUC there’s nothing preventing spores from being just another library for everyone to try out. It’s likely that many of these questions would have come up by now if someone would have tried spores experimentally. A SIP for inclusion into the standard library should IMO be preceded by practical experiments collecting evidence for the utility of the proposed addition (because it's so easy compared to SIPs regarding compiler changes).
Here’s a simple application using Futures I tried to convert to use spores (assuming that all higher-order functions in `Future` are changed to require spores):
https://github.com/jrudolph/spray/commit/8faa2a6616b3a69e374a41ce3b8be9ff33b35b14
Immediate questions:
* for-comprehensions: do they work together with spores anyhow?
* nested spores, what are the rules?
Johannes
[1] Whatever the problem really is, I find it hard to describe. Many cases seem related to capturing `this` and then exporting code which calls instance methods from inside the class context, and therefore violating the encapsulation classes usually provide. That seems to fit the Actor.sender case but less the serialization case where the problem seems to be that it’s hard to control which implicitly created instances contain outer references and whereto exactly.
--
You received this message because you are subscribed to the Google Groups "scala-sips" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-sips+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Philipp Haller
Heather, thanks for the clarifications.
The problem I still have is that it is not clear how the SIP should help to write correct programs in practice. And IMO that’s mainly a consequence of two things the SIP is unclear about:
- an analysis of the problem, a problem statement and an explanation how the given motivations fit into the problem statement
- a summary description of the solution and how the solution solves the practical problem
this is avoided, since the result of this.sender is assigned to the spore’s local value from when the spore is created. The spore conformity checking ensures that within the spore’s closure, only from and d are used.this is avoided, since helper has to be assigned to a local value (here, h) so that it can be used inside the spore’s closure. As a result, fun can now be serialized without runtime errors, since h refers to a serializable object (a case class instance).(- the “how does the feature integrate with existing features” section which would have revealed practical problems)
The motivating cases from the SIP show that there are issues related to capturing but it doesn’t explain what the exact relation really is.
Given the above class definitions, serializing the fun member of an instance of Main throws aNotSerializableException. This is unexpected, since fun refers only to serializable objects: x (an Int) andhelper (an instance of a case class).
Here is an explanation of why the serialization of fun fails: since helper is a field, it is not actually copied when it is captured by the closure. Instead, when accessing helper its getter is invoked. This can be made explicit by replacinghelper.toString by the invocation of its getter, this.helper.toString. Consequently, the fun closure capturesthis, not just a copy of helper. However, this is a reference to class Main which is not serializable.
Futures: What seems to be proposed is to replace the parameter types of Future's higher-order methods to require Spores instead of plain functions. Please try to do that and see what has to be changed in even the simplest Future-based applications. It turns out that being able to close over stuff is one of the big advantages of the Future API. In fact, it's hard to find a legitimate program using Futures that doesn't use capturing. With spores we are basically back in Java-land where closed-over values had to be put into final variables. Actually, it turns out the problem is much more related to the akka Actor API than to Futures. So, why change Futures?
Serialization: Capturing is still possible with spores, as is referencing other un-serializable values in other ways. So, how do spores help?
I’ll try to rephrase what a spore is: Spore is a new type extending the Function type which guarantees (with a static check at construction) that a function value of this type has an additional property. The property seems to be ‘All values captured by this function have been declared explicitly’.
It’s unclear to me how ‘have been declared explicitly’ will solve ‘problems, somehow related to capturing’.
There are some questions the SIP should answer:
- When would I use the Spore type over the regular function type?
- When would I use the spore constructor instead of an anonymous function?
- What are the disadvantages of using the solution?
- How does this new property solve the problem shown with the motivating examples? I want to know in detail why “Have been declared explicitly” will reduce bugs. What are the steps to get from buggy code using some libraries to a fixed version, how have spores helped and how much correct code had to be changed as well to support the solution?
IMO spores are not a conservative solution at all because the solution offered is much broader than the problem itself [1]. Think about what happens if the SIP is implemented like currently proposed: Either it’s an opt-in solution i.e. the developer decides when to use spores, then it won’t be used at all because you don’t know where to use it exactly to prevent bugs. Or, it’s enforced by libraries in which case all code using the libraries has to be changed. It will be a chore similarly to having to make variables final in Java, which when you do it in a mechanized way won’t help you paying attention to the details that actually are the root of the bugs.
Regarding my earlier comments about why this has to be a SIP: IIUC there’s nothing preventing spores from being just another library for everyone to try out. It’s likely that many of these questions would have come up by now if someone would have tried spores experimentally. A SIP for inclusion into the standard library should IMO be preceded by practical experiments collecting evidence for the utility of the proposed addition (because it's so easy compared to SIPs regarding compiler changes).
Here’s a simple application using Futures I tried to convert to use spores (assuming that all higher-order functions in `Future` are changed to require spores):
https://github.com/jrudolph/spray/commit/8faa2a6616b3a69e374a41ce3b8be9ff33b35b14
Immediate questions:
* for-comprehensions: do they work together with spores anyhow?
* nested spores, what are the rules?
Johannes
[1] Whatever the problem really is, I find it hard to describe. Many cases seem related to capturing `this` and then exporting code which calls instance methods from inside the class context, and therefore violating the encapsulation classes usually provide. That seems to fit the Actor.sender case but less the serialization case where the problem seems to be that it’s hard to control which implicitly created instances contain outer references and whereto exactly.
--
Alex Boisvert
Johannes Rudolph
I apologize, my tone was inviting a defensive answer and I didn't get my point through. I’ll try to rephrase my arguments.
My main complaint: All boils down to how to decide when to use spores. I say: for your argumentation in the SIP that spores prevent errors you have invoked an oracle to decide when to use them. More specifically, the SIP takes two examples we know are broken and then applies spores to 'reveal' the errors. When and how to use spores to prevent errors (on code of unknown correctness) is not something which should be left to the user documentation but it should be the central part of the motivation. Lacking an oracle, you are left with two choices: use spores everywhere which is not acceptable, or don’t use them at all.
My formal objection: Language changes and library changes in Scala usually are evaluated to comply to a high-standard. One of the main criteria usually is that a new feature was proven to compose well with other features of the language. The most obvious interaction of spores would be with itself (i.e. nested spores), other functions, with methods, with by-name parameters, with for-comprehensions and wrt eta expansion. This is missing from the proposal.
My admission: It’s indeed hard to argue with Roland’s example. `Props.apply` seems to be the poster child for spores. What you usually want to build a Props from is 1) a number of constant values and 2) a constructor for an actor. The `Props.apply` overload with the by-name parameter is just too powerful and invites dangerous captures. Spores seem to fit there quite nicely. My objection is that even there a spore doesn't catch all errors in its domain. But maybe that’s something I have to live with: there are solutions which are not applicable everywhere and which don’t catch all errors but which may still be useful for some purposes.
In summary my objections are:
* spores only help if applied at the right places, but I don’t know where to apply them
* spores interaction with other language features is unspecified and, as others have noted, not obviously trouble-free
* spores indicate only some errors related to capturing and always require the developer to be alert not to duct-tape around the issue
Trying to be more constructive I created an example (showcasing the last point from my list) which assumes that `Props.apply` would require spores (I created a wrapper `SProps.apply` which actually does). The situation is simple, it’s an akka application which has a Boot class which holds some buffers used during booting which should be eligible for GC after booting. During booting a series of actors is created. The task here is to avoid leaking the Boot instance into the actor Props. I used spores with imagined best practices to avoid the common pitfalls:
Questions for evaluation: Which one of those actor creations avoided capturing the Boot instance? How did spores help? How does the syntax compare with the original one?
Chris Marshall
Lukas Rytz
I apologize, my tone was inviting a defensive answer and I didn't get my point through. I’ll try to rephrase my arguments.
My main complaint: All boils down to how to decide when to use spores. I say: for your argumentation in the SIP that spores prevent errors you have invoked an oracle to decide when to use them. More specifically, the SIP takes two examples we know are broken and then applies spores to 'reveal' the errors. When and how to use spores to prevent errors (on code of unknown correctness) is not something which should be left to the user documentation but it should be the central part of the motivation. Lacking an oracle, you are left with two choices: use spores everywhere which is not acceptable, or don’t use them at all.
Roland Kuhn
Eric Torreborre
Heather Miller
system.actorOf(SProps {val l = this.logger() => new LoggingActor(l)})
martin odersky
Hi Johannes, all,First off, thanks, all, for all the feedback everyone's provided so far. All of your input has allowed us to come up with a potential revision of the SIP (the rest of this email covers its main points).Johannes– thanks for rephrasing your arguments, that clarifies a lot. Btw, I really appreciate your efforts to try and apply spores in various situations- it'll help us with our next revision of the SIP!Regarding the interaction of spores with other language features: you're right that as it stands spores and for-comprehensions don't integrate (causing your rewriting of your initial future-based code). The question of for-comprehensions specifically has come up several times– it's a question I/we have actually been working hard to answer for several days (solving it in a satisfying way requires re-thinking some aspects of spores). I'm happy to say that, after some careful thought, I think we have some cool ideas.In order to more seamlessly support for-comprehensions and other closure-based features, we've been thinking about whether or not it'd be a good idea to provide a way to implicitly convert closures to spores (Eugene has been trying really hard to convince us!). This implicit conversion + a twist could give us a way to provide intuitive support for for-comprehensions. Additionally, as a side effect, it should enable us to reduce the syntactical overhead that you, Roland, and others have voiced concerns about.About your SProps experiment: using an implicit macro for converting a (correctly-shaped) closure to a spore, and making SProps a macro, too, one should be able to writesystem.actorOf(SProps {val l = this.logger() => new LoggingActor(l)})The earlier-mentioned "twist" is to introduce the suggested `capture` syntax, but only for local vals. Selections and method invocations (e.g., `this.sender`) that should be evaluated upon spore creation would still have to go into the list of val defs of the spore. The rationale is that this restriction would make sure that we re-use the existing way of expressing the same evaluation semantics:{val l = this.logger() => new LoggingActor(l)}andspore {val l = this.logger() => new LoggingActor(l)}have the same runtime behavior. That's nice because this way, we don't introduce a new syntax for expressing this evaluation semantics.
Roland Kuhn
- that it would be wise to add the implicit conversion from function literal to spore as outlined below
- that unproblematic values should be captured without any overhead (i.e. “capture()” should not be required, since it does not add more information)
- that unproblematic values include all stable values (or paths, if I understand that term correctly), not only local vals
- it should be possible to explicitly mark problematic types so that capturing them is rejected
Havoc Pennington
Reading over this thread, I'm not sure exactly when a spore type
should be used, or what specifically defines a "problematic" value as
Roland puts it. We've talked about examples such as serialization and
concurrency, but I'm not sure how to identify a problem via general
principle (vs. by plugging it into examples).
I wonder if the general principle has to do with whether the current
stack frame remains present when the closure runs?
I talked about that in this blog post (some of you have seen it before):
http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/
For that post I defined sync vs. async in a very specific way, which
is whether the closure is executed with the current stack frame still
around.
Talking about "same stack (frame)" execution vs. "another stack",
"another stack" could conceptually be another thread, or this same
thread but in deferred fashion (stack is popped back to the main loop,
which then executes the closure).
It's possible that serialization fits into the same scheme; another
JVM, after all, is a different stack...
Perhaps this terminology (same-stack / different-stack) would be a way
to clarify the intended semantics and when to use which
types/annotations/APIs/whatever.
If we look at Actor methods which are dangerous to capture, such as
`sender` and `self`, they are dangerous precisely because they are
valid only on the current stack (they are presumably set up by the
stack frame that invokes the actor's behavior). If these were somehow
annotated as "same stack only", then a macro could know that they
cannot be captured in a "different stack" closure. So those `sender`
and `self` methods could be precisely described as "stack dependent"
or "valid only for this same stack" or something of that nature.
Similarly, the subject of my blog post above is to argue that most
APIs should define the execution model of closures they accept - given
foo(closure: A=>B), it should be defined whether `closure` runs with
foo's stack frame still around. So at least for new APIs without
legacy headaches, it might be reasonable to hope that they could
specify this (through the type system, an annotation, or at least
docs).
Some complications:
- some current APIs do NOT promise one way or the other
- if the closure is pure, then it is probably OK not to care what
stack the closure runs on (impure closures can be broken either way -
they can rely on same-stack or rely on different-stack)
- an API could legitimately leave the closure's execution stack
unspecified by requiring the closure to be pure
- for Future, same-vs-different stack is deterministic - but it's
defined by the ExecutionContext and thus configurable
In the context of Actor we've talked a lot about closures which are
invalid when moved to a different stack, but I don't think we've
talked much about closures which are invalid if kept on our own stack
- that's possible too though, e.g. deadlocks can happen, and you can
find mutable-state scenarios that break.
If this same-stack vs. different-stack definition of the problem is
useful, then it seems like we'd want to start to know, for each method
or value, whether its value is stack-dependent... and for each
closure, whether it will execute with the current stack frame
present... and then different warnings and macros could use that
information...
I don't know if this way of thinking about it is consistent with what
you guys have in mind for spores or not, but this is how I typically
think about closures, so I find myself trying to understand spores in
these terms.
Havoc
Johannes Rudolph
Hi Heather,
thanks for having this open discussion and addressing the syntax concerns. I’ve updated both of my examples to the new syntax [1,2]. Indeed, with the new syntax the original for-comprehension can be restored so that seems like progress.
That said: purposely, I tried to refrain from discussing any technical details of the proposal (it’s itching me all over to do that) because I’m still not convinced that it solves the root of the problems you state as a motivation (and I tend to agree with Roland that maybe many of them can’t be solved right now). That’s what my SProps example was about: for each example where spores could prevent a problem, the error which isn’t found by spores is just one abstraction away. *)
With the syntax changes this might actually be amplified by the fact that the least intrusive change needed to make your code compile again is usually the wrong one, i.e. wrapping the wrong expression with `capture`. As Roland noted when going from functions to spores wrapping with `capture` means no semantic change so this alone cannot directly resolve a bug. To be fair, in some cases you will end up with `capture(this)` which in some cases is a marker that something is wrong. The point is you cannot know without looking at the type of `this` and even then you may come to the conclusion that `capture(this)` is the way to go. Then you have to mark this with a comment which then again is not machine-checkable any more.
I repeat myself here: If the proposal were just about serialization, it could work because serializability actually _is_ a property of captured values and you could somehow encode it in the types. Also, it’s quite clear how serializability composes: a compound value is serializable if all its parts are (roughly). And a closure is just an implicit compound value so you could check serializability of the compound by checking the serializability of its components. IIUC that’s what that part of cloud haskell is about.
More generally speaking, the problems arising are less about the fact *that* you capture than about *what* you capture. That’s nothing you can properly determine or decide locally at the point where you capture but something which 1. has to be propagated in the types and must be composable in some way and 2. is often dependent on the actual context of capturing. So, Roland’s suggestion to explicitly opt out some types from being captured seems like the minimal version of something like that. I guess it would work in the simple cases but missing any composability it still makes it too easy for problems to hide somewhere.
Johannes
*) It may be ok for a solution to catch just a subset of errors (it's this way with most solutions). The proposal should then at least be clear about it to make it easier to understand the trade-offs.
[1] https://github.com/jrudolph/spray/compare/wip;sporification
[2] https://github.com/jrudolph/sporophyte/blob/master/src/main/scala/sporophyte/Main.scala--
You received this message because you are subscribed to the Google Groups "scala-sips" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-sips+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Roland Kuhn
yes, your characterization looks correct and very useful to me: “problematic closure” boils down to “executed not below originating stack frame”, which means that a `.foreach` of a sequential collection is fine while a parallel collection is problematic.
I find it appealing to visualize the lexical area of code within the closure to be hoisted out of the normal “on-stack” plane, and the cliff which forms is delimited in the source code by the spore { … } marker. In the end it is similar to type inference in that explicit hoisting can be preferable in places while automatic (checked) hoisting avoids clutter where that is not needed.
Regards,
Roland
Suminda Dharmasena
Suminda Dharmasena
Would be a nicer syntax than what is posposed. Also you are explicit on what the captured arguments are.