Re: [scala-user] Re: Scala compiler crash

[Jason Zaugg]

On Wed, Aug 21, 2013 at 5:07 AM, Ian Clarke <ian.c...@gmail.com> wrote:

On Tuesday, August 20, 2013 5:09:16 PM UTC-5, Jason Zaugg wrote:

First of all, a heads up regarding CPS: We don't have resources assigned for maintenance of the CPS compiler plugin. In fact, we're thinking of declaring it deprecated. It has some architectural issues (most notably: intimate relationship with typechecking under the annotation-driven pluggable subtyping logic) that means it has a long tail of hard- or impossible-to-fix bugs and limitations.

I would plead that this plugin not be deprecated if at all possible, our project is entirely dependant on it, and I think Greg's is too.  Moreover, there are very few programming languages that support continuations, and even fewer that support portable continuations.  

While portable continuations might seem to be an esoteric feature, in actuality they are extremely relevant to many of the real-world challenges that programmers face today, in particular the problem of building scalable distributed and fault tolerant software.  In my humble opinion, portable continuations are our best chance of achieving the holy grail of allowing software to be written that can be scaled across tens, hundreds, or maybe even thousands of computers - just as easily as writing software to run on a single CPU.  The existence of the continuations plugin was what convinced me to base our project on Scala.

Typically, useful functionality is only deprecated in an API when there is a better alternative.  Deprecating the continuations plugin in favor of an unspecified Futures-based system may seriously screw quite a few people who made quite early bets on Scala.  I hope you take that into consideration in your decision to deprecate it.

[moving to scala-internals] 

I would argue that while continuations are a great feature, the implementation in Scala isn't suitable for building production systems. We know of folks that have tried this at a large scales and are now keen to migrate to async.

BTW, this isn't meant in any way as a slight on the implementation of original CPS plugin. The problems arise when trying to comprehensively integrate it into the rest of the compiler. Even if you can live with domain-unspecific type errors that your API users will encounter, you ought to be worried about the unsolved (and unsolvable) bugs (I'd wager that there are as many more again undiscovered or unreported), and the way that workarounds for the plugin have leaked into the regular compiler (typer and pattern matcher in particular.) For example, failing to account for the fact that `T @cps` is not a subtype of `Any` is an easy way for the compiler to break CPS (well, in theory any AnnotationChecker based compiler plugin; but there are very, very few of them!)

With all of this in mind, we have to find a way to communicate our view of CPS to potential new users (we'd rather that *more* projects didn't start depending on it) while providing some sort of migration path for your projects. The plugin could be maintained by a third party, for example, we're not planning to remove the extension points from the compiler. Or maybe we can suggest ways that you could use macros as your user API like async and effectful [1].

I'll also ask Adriaan to chime in. I don't want to push this discussion too much as I helped to develop scala-async so I might be biased.

-jason

[1] https://github.com/pelotom/effectful

[Ian Clarke]

On Wednesday, August 21, 2013 1:55:39 AM UTC-5, Jason Zaugg wrote:

With all of this in mind, we have to find a way to communicate our view of CPS to potential new users (we'd rather that *more* projects didn't start depending on it) while providing some sort of migration path for your projects. The plugin could be maintained by a third party, for example, we're not planning to remove the extension points from the compiler. Or maybe we can suggest ways that you could use macros as your user API like async and effectful [1].

So long as you are thinking hard about how to avoid leaving existing CPS plugin users high and dry then I feel somewhat better, but I'm curious as to whether macros and async/effectful can offer the functionality we need (effectively serializing the state of a thread transparently to the programmer)?

I'm not familiar with Scala macros but will read up on them.

Ian.

[Adriaan Moors]

Hi Ian, Greg,

I agree completely with Jason. If you're using CPS in production and would like support,

please get in touch and we'll try to work something out.

The community is of course more than welcome to maintain the CPS plugin,

but we see scala-async as a much, much nicer way of solving these issues.

Our experience has been that users of the continuations plugin have been eager to move to scala-async.

Also note that CPS has been effectively unmaintained for about 2 years now:

src/continuations/plugin/ was touched by about 50 commits in that period.

cheers

adriaan

[Paul Phillips]

On Wed, Aug 21, 2013 at 10:48 AM, Adriaan Moors <adriaa...@typesafe.com> wrote:

Also note that CPS has been effectively unmaintained for about 2 years now:

src/continuations/plugin/ was touched by about 50 commits in that period.

As the author of about half of those commits, let me say you might have been better off with half as many. What I mean to say is, "what they said."

[Ian Clarke]

On Wed, Aug 21, 2013 at 12:48 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

I agree completely with Jason. If you're using CPS in production and would like support,

please get in touch and we'll try to work something out.

Unfortunately we're an open source project, we don't have financial resources to pay for something like that, I'm afraid.

Our experience has been that users of the continuations plugin have been eager to move to scala-async.

I will investigate scala async.  I'm curious though, does it allow you to serialize the "state" of some code and transmit it across the network (ie. a "portable continuation")?

Ian.

--
Ian Clarke

Blog: http://blog.locut.us/

[Adriaan Moors]

Our experience has been that users of the continuations plugin have been eager to move to scala-async.

I will investigate scala async.  I'm curious though, does it allow you to serialize the "state" of some code and transmit it across the network (ie. a "portable continuation")?

It doesn't. To be clear: we'll make sure the CPS plugin is released for 2.11, but we're unlikely to fix bugs in it ourselves.

 

[Meredith Gregory]

Dear Adriaan, and All,

Thanks for all the helpful information!

Biosimilarity, LCC (Biosim), my company, had already concluded that we needed to own our own version of the CPS plugin. After deploying a production level project that used the plugin we realized that there were several desiderata in the plugin and the way we used it that we needed to address. We were planning on making it generally available to the community once we'd gotten it to the point where it was more help than harm. ;-)

Biosim would be happy to collaborate with TypeSafe and the community to help craft a CPS solution factored in a way that was flexible enough to support scala-async yet also address users like Swarm and SpecialK -- if that would be of interest.

Best wishes,

--greg

On Wed, Aug 21, 2013 at 10:48 AM, Adriaan Moors <adriaa...@typesafe.com> wrote:

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
7329 39th Ave SW

Seattle, WA 98136

+1 206.650.3740

http://biosimilarity.blogspot.com

[Adriaan Moors]

Hi Greg,

Great! 

I'll let Jason fill in the gaps in my understanding,

but I believe scala-async uses a state machine translation,

because it doesn't require the full generality of the cps transform. 

(It does do an ANF transform, so I'm sure there's potential for code reuse.)

Also, part of the motivation for the 2.11 efforts regarding modularization and stabilization of the platform

is to facilitate both the development and the coordination of modules on top of the Scala[c] "platform".

cheers

adriaan

[Ian Clarke]

That is great Greg, I will do what I can to help although I find that simply using the CPS plugin to be a bit of a mindbender so I'm not sure how much help I can be on developing it :-)

Ian.

[Philipp Haller]

Hi Greg,

This sounds very interesting!

On Wed, Aug 21, 2013 at 9:18 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

I'll let Jason fill in the gaps in my understanding,

but I believe scala-async uses a state machine translation,

because it doesn't require the full generality of the cps transform. 

(It does do an ANF transform, so I'm sure there's potential for code reuse.)

Exactly.

The Async SIP [1] contains a fully worked-out example of both the ANF transform and the state machine translation. To compare with the CPS plugin, we have a number of tests [2] that show what the current restrictions of Async are.

Cheers,

Philipp

 

[1] http://docs.scala-lang.org/sips/pending/async.html

[2] https://github.com/scala/async/blob/master/src/test/scala/scala/async/neg/NakedAwait.scala

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Sébastien Doeraene]

Hi all,

I really hope some CPS plugin will continue to exist. Async is a better, nicer solution under the assumption of lexical scoping. But capturing the full state of a thread across library boundaries is sometimes very useful, and only the CPS plugin provides that.

I also have but two weeks ago figured out that the CPS plugin was the solution to a difficult design issue I had in one of my personal projects. That project cross-compiles under Scala and Scala.js, and in JavaScript you don't have threads. CPS is the only solution I found to design the project in a sane way, considering the 1-thread-only limitation. Btw, yes, the CPS plugin works with Scala.js!

Cheers,

Sébastien

[Tiark Rompf]

Hi all,

just to add my 2 cents, I think an unfortunate decision was made to focus development effort on something less powerful, less expressive (async) instead of investing resources to fix continuations once and for all. Most of the issues like error messages etc. would be easily fixed by integrating the CPS logic into the compiler proper. Making the underlying monad implementation swappable would have made it trivial to add a state-machine implementation as a special case.

Cheers,

- Tiark

[Simon Ochsenreither]

just to add my 2 cents, I think an unfortunate decision was made to focus development effort on something less powerful, less expressive (async) instead of investing resources to fix continuations once and for all. Most of the issues like error messages etc. would be easily fixed by integrating the CPS logic into the compiler proper. Making the underlying monad implementation swappable would have made it trivial to add a state-machine implementation as a special case.

Btw, could you have a look at https://groups.google.com/d/topic/scala-internals/ZW_kunE7kiA/discussion?

[Pierre Schaus]

Hi All,

I was just informed bout this discussion on CPS compiler.

I'm working on a state of the art constraint programming solver https://bitbucket.org/oscarlib/

Our project is 100K loc and the search tree exploration heavily relies on continuations (we have implemented this paper: Pascal Van Hentenryck and Laurent Michel. “Nondeterministic control for hybrid search” Constraints Journal, 2006.)

As already said above, I would plead that this plugin not be deprecated if at all possible, since or project is also dependant on it.  

CPS plugin is also what convinced us to use Scala...

Note that in OscaR we also use continuations to implement a discrete event simulation engine.

We use mainly use the most basic version of continuations:

shift { k: (Unit => Unit) => 

}

Any alternative for that?

Thanks,

Pierre

[Miles Sabin]

On Thu, Aug 22, 2013 at 12:30 PM, Tiark Rompf <tiark...@epfl.ch> wrote:
> just to add my 2 cents, I think an unfortunate decision was made to focus
> development effort on something less powerful, less expressive (async)
> instead of investing resources to fix continuations once and for all. Most
> of the issues like error messages etc. would be easily fixed by integrating
> the CPS logic into the compiler proper. Making the underlying monad
> implementation swappable would have made it trivial to add a state-machine
> implementation as a special case.

+100

Cheers,


Miles

--
Miles Sabin
tel: +44 7813 944 528
skype: milessabin
gtalk: mi...@milessabin.com
g+: http://www.milessabin.com
http://twitter.com/milessabin

[Miles Sabin]

On Wed, Aug 21, 2013 at 6:48 PM, Adriaan Moors
<adriaa...@typesafe.com> wrote:
> The community is of course more than welcome to maintain the CPS plugin,
> but we see scala-async as a much, much nicer way of solving these issues.

I'm afraid this isn't a great way to drum up community support for a
project. Do we even have any assurance that the annotation based
extension mechanisms will continue to exist?

[Som Snytt]

Another way to look at it is, this is a great opportunity to contribute, because you know your code won't be immediately superseded by something lying around in a paulp branch or by something elegant retronym knocked together based on an idea phaller sketched at dinner.

Well, I guess there aren't many paulp branches left, anyway.

[Paul Phillips]

This thread reads like a who's who of people working on interesting things, so I'm sure we're sorry to disappoint. (Not that I was involved in the decision - I would have done it years ago.) But you guys have no idea what a burden the implementation is. There's still plenty of very basic stuff in the compiler which doesn't work. By very basic I mean things which affect everyone. This thread notwithstanding, the usage of continuations is tiny. There is no justification available for pouring time into the continuations plugin under those circumstances. And it just being there at all, even if we make absolutely no effort to improve it, is still PLENTY of burden.

[Ian Clarke]

On Fri, Aug 23, 2013 at 9:24 AM, Paul Phillips <pa...@improving.org> wrote:

This thread reads like a who's who of people working on interesting things, so I'm sure we're sorry to disappoint. (Not that I was involved in the decision - I would have done it years ago.) But you guys have no idea what a burden the implementation is. There's still plenty of very basic stuff in the compiler which doesn't work. By very basic I mean things which affect everyone. This thread notwithstanding, the usage of continuations is tiny. There is no justification available for pouring time into the continuations plugin under those circumstances. And it just being there at all, even if we make absolutely no effort to improve it, is still PLENTY of burden.

I understand your view Paul, and I am the first to admit that I know next to nothing about Scala internals.

However, I can't agree that there is no justification.  I think the justification is that Scala is a platform, and one of the greatest responsibilities of a platform is not to make people regret building on it by removing features that have no alternative, as is the case with the CPS plugin.  

A number of people, myself included, chose Scala specifically because it had this capability.  If it were taken away now this would not only makes us all regret that decision, but it will make people question the wisdom of betting on Scala in the future for other projects.

So I think there is justification, although you might argue that it's not strong enough given the resources that would be required.

Regardless, Greg has stated his willingness to put his resources (and considerable brain power) into this, and I will certainly do what I can to support him - the critical thing is that he not be made to regret that decision.

Ian.

[martin odersky]

I think it would be good to re-think the approach to continuations. Async has two important advantages over the CPS plugin:

1) It's a macro, and a blackbox macro to boot, so we can assume the compiler will support it in the future. The cps plugin is a compiler plugin, and these tend require a much higher amount of maintenance to keep them in sync with future compiler versions. 

2) It's syntactic, triggered by async/await wrappers. This makes it much easier to disentangle from the compiler internals. By contrast, the cps plugin requires fairly tricky integration of annotation checkers in the type checker. A lot of that code was written specifically for the continuations plugin (and a couple other, more experimental ones). Many of us believe that we have not yet discovered all the possible corner cases in this integration.

But of course, continuations subsume async and are much more powerful. So one potentially interesting question for potential contributors would be:

 - can one implement cps transform as a macro as opposed to a compiler plugin?

 - can the source language itself be syntactic, prompted by wrapper calls instead of annotations?

The two questions are independent of each other.

Of course, another route would be to maintain the current plugin. But if there is a change of guard in maintainers, that might also be a good opportunity to think about the questions above.

Cheers

 - Martin

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967

[Paul Phillips]

On Fri, Aug 23, 2013 at 7:35 AM, Ian Clarke <ian.c...@gmail.com> wrote:

However, I can't agree that there is no justification.  I think the justification is that Scala is a platform, and one of the greatest responsibilities of a platform is not to make people regret building on it by removing features that have no alternative, as is the case with the CPS plugin.  

Hey, I'm with you in spirit. There are about a billion ways to make people regret building on it though. These things are always tradeoffs.

[Adriaan Moors]

In saying "you're welcome to maintain", it was implied that we'll be

happy to work with anyone who decides to maintain it.

We're very grateful for every single contribution from the community,

and will go to great lengths to make it as easy for you to contribute as possible.

(As demonstrated, I think, by our investment in PR validation infrastructure,

 modularization, build cleanup,...)

We have limited resources, and pouring our time in improving the CPS plugin,

while developing the async macro would be a disservice to the 99% of the community

that isn't using the CPS plugin. Availability bias may make this hard to believe,

but I'd be shocked to see more than 1% of Scala developers using the plugin.

In developing the 2.10 pattern matcher, I spent a significant amount of time with the

CPS plugin implementation, and I seriously doubt it would be a trivial matter to

make it as efficient, user-friendly and maintainable as the async macro (in practice).

On Fri, Aug 23, 2013 at 2:15 AM, Miles Sabin <mi...@milessabin.com> wrote:

[Tiark Rompf]

On Aug 23, 2013, at 4:57 PM, martin odersky <martin....@epfl.ch> wrote:

I think it would be good to re-think the approach to continuations. Async has two important advantages over the CPS plugin:

It's obvious that these are advantages in terms of making life easier for the compiler team. Whether they are advantages for Scala users seems debatable.

1) It's a macro, and a blackbox macro to boot, so we can assume the compiler will support it in the future. The cps plugin is a compiler plugin, and these tend require a much higher amount of maintenance to keep them in sync with future compiler versions. 

2) It's syntactic, triggered by async/await wrappers. This makes it much easier to disentangle from the compiler internals. By contrast, the cps plugin requires fairly tricky integration of annotation checkers in the type checker. A lot of that code was written specifically for the continuations plugin (and a couple other, more experimental ones). Many of us believe that we have not yet discovered all the possible corner cases in this integration.

But of course, continuations subsume async and are much more powerful. So one potentially interesting question for potential contributors would be:

 - can one implement cps transform as a macro as opposed to a compiler plugin?

 - can the source language itself be syntactic, prompted by wrapper calls instead of annotations?

These questions seem largely rhetorical. Of course one can implement a (method-local) cps transform as a macro. Of course one can use surface syntax. The real question is will this lead to something robust and general enough to implement interesting abstractions on top of it: nondeterministic and probabilistic computation, coroutines, reactive programming etc.

There still needs to be logic that traces which pieces of code are affected by continuations and which aren't. My hunch is that covering anything beyond the most simple cases (suspendable closures? by-name arguments? pattern matching? exceptions?) will result in an ad hoc and brittle effect type system spread across a collection of macros. I'm not convinced this is the best option in the long run.

Cheers,

- Tiark

[Meredith Gregory]

Dear All,

This dialogue has been very useful!

Biosim's goals go considerably past what the delimited continuations plugin supports. As an example, we are serializing continuations and restoring them from disk. This raises a whole host of questions that are simply not in the purview of the current plugin. However, we have been able to work out some back of the envelope schemes for factoring various levels of functionality.

Additionally, we welcome the opportunity to engage the Scala compiler more fully. It's been something we knew we had to do for quite some time.

In short, our aim and intent is to work with various members of the community to re-architect and reimplement the existing CPS plugin. We'd like it to be as useful as possible. So, if it could provide a substrate for the async work, that's excellent! Certainly, they should be able to co-exist. As preparation for that effort we'd like to hold a kind of townhall hangout on air to get a sense of the community requirements, what's necessary, what's possible and who's in and who's part of the peanut gallery. ;-)

If people could send me some indication as to their interest in participating in that discussion with some indication of preferred times i will endeavor to set something up that as many as possible can attend.

Best wishes,

--greg

[Meredith Gregory]

P.S. Just to be clear, we are happy to take on the burden to maintain this for users for the foreseeable future. We are not looking to increase TypeSafe's burdens. We are hoping to realize some of the potential of the open source ecosystem and increase the scope of Scala without TypeSafe having to pick up the tab. We've really benefitted from the Scala effort. If this becomes a way we can deliver some value back to the community, that's only right.

[Adriaan Moors]

Thanks, Greg! We'll be happy to participate in the hangout.

have a great weekend,

adriaan

[Paul Phillips]

On Fri, Aug 23, 2013 at 3:18 PM, Tiark Rompf <tiark...@epfl.ch> wrote:

It's obvious that these are advantages in terms of making life easier for the compiler team. Whether they are advantages for Scala users seems debatable.

What's good for General Motors is good for the country!

[Miles Sabin]

On Fri, Aug 23, 2013 at 6:36 PM, Adriaan Moors
<adriaa...@typesafe.com> wrote:
> In saying "you're welcome to maintain", it was implied that we'll be
> happy to work with anyone who decides to maintain it.

That only really works if we have a positive answer to my question "Do

we even have any assurance that the annotation based extension

mechanisms will continue to exist?" Do we?

Speaking for myself, I have no interest in contributing to a community
CPS plugin if it's likely that the underlying mechanisms which support
it might go away. Maybe I misread your earlier mail, but it sounded to
me as though that mechanism was a major contributor to your headaches,
hence destined the chop.

[Adriaan Moors]

It's the cps plugin itself that's causing head aches, not the plugin infrastructure.

We won't deprecate the plugin infrastructure in 2.11, so it'll at least be around until 2.12.

We make roughly the same guarantees for (whitebox) macros and large parts of the reflection API.

I'm a bit surprised at the reactions in this thread, to be honest.

We're just coming out and being explicit about what's been the status quo of the cps plugin for years now.

We're not removing it. We're just trying to spare new users unpleasant surprises.

[Miles Sabin]

On Sun, Aug 25, 2013 at 6:27 AM, Adriaan Moors
<adriaa...@typesafe.com> wrote:
> It's the cps plugin itself that's causing head aches, not the plugin
> infrastructure.
> We won't deprecate the plugin infrastructure in 2.11, so it'll at least be
> around until 2.12.

OK, that's somewhat reassuring.

> I'm a bit surprised at the reactions in this thread, to be honest.
> We're just coming out and being explicit about what's been the status quo of
> the cps plugin for years now.
> We're not removing it. We're just trying to spare new users unpleasant
> surprises.

The CPS plugin has been around for a while, and I'd been hoping that
it would follow the same trajectory into the core language as
dependent method types, particularly given that we now have the SIP-18
mechanisms available to enable or disable it at the language rather
than the tooling level. The fact that the plugin shipped with the
compiler lent quite a bit of support to that hope. Not to mention that
Akka (at least at one point? still?), used the plugin to support
dataflow concurrency.

As the other contributions to this thread show, I'm not the only one
that's formed those sort of expectations.

[Adriaan Moors]

On Sun, Aug 25, 2013 at 3:01 AM, Miles Sabin <mi...@milessabin.com> wrote:

The CPS plugin has been around for a while, and I'd been hoping that
it would follow the same trajectory into the core language as
dependent method types, particularly given that we now have the SIP-18
mechanisms available to enable or disable it at the language rather
than the tooling level. The fact that the plugin shipped with the

compiler lent quite a bit of support to that hope. 

I think I've said it before, but again: I'm sorry we set the wrong expectations.

This is what I'm trying to correct, for lack of a better option.

This is open source software: if there's no maintainer, well, there's no maintainer.

We, as Typesafe, need to focus our limited resources where they will bring most good.

Would you really put cps at the top of the list, above the much more user-friendly async,

incremental compilation performance, a better optimizer, modularization and

overall working towards full binary compatibility?

 

Not to mention that Akka (at least at one point? still?), used the plugin to support
dataflow concurrency.

They are using it. Again: we're not removing it. Just indicating it's not maintained. 

Unless new information pops up, I don't see any value in reiterating our position in this thread.

cheers

adriaan

[Miles Sabin]

On Sun, Aug 25, 2013 at 7:03 PM, Adriaan Moors
<adriaa...@typesafe.com> wrote:
> Would you really put cps at the top of the list, above the much more
> user-friendly async, incremental compilation performance, a better optimizer,
> modularization and overall working towards full binary compatibility?

Given that the CPS plugin already existed and the async functionality
could have been built on top of it (granted, with some effort, but not
obviously much more effort than went into async), yes, definitely.

[Paul Phillips]

On Sun, Aug 25, 2013 at 11:23 AM, Miles Sabin <mi...@milessabin.com> wrote:

Given that the CPS plugin already existed and the async functionality
could have been built on top of it (granted, with some effort, but not
obviously much more effort than went into async)

Your word placement reveals you are conscious of the distinction between "not obviously much more effort" and "obviously not much more effort". The former is easy to agree with but irrelevant. Jason must have a clearer idea of the relative difficulties than you or I.

[Miles Sabin]

I said that as one of the people who has built async-like
functionality on top of the CPS plugin (several people have done it
independently). I know from first hand experience that given Tiark's
work 90% of the rest is pretty close to trivial. The last last 10%,
maybe not so trivial, but whether that's "not obviously" or "obviously
not" less effort than rehashing the hard bit (ie. the CPS transform,
even leaving out the effect typing) probably only Tiark can judge. My
gut feeling is that the wrong call was made.

But whatever, so long as the underlying infrastructure isn't going
away, I guess I have no option but to find some time to put into a
community supported version of the plugin.

[Paul Phillips]

On Sunday, August 25, 2013, Miles Sabin wrote:

I said that as one of the people who has built async-like
functionality on top of the CPS plugin (several people have done it
independently). I know from first hand experience that given Tiark's
work 90% of the rest is pretty close to trivial. The last last 10%,
maybe not so trivial, but whether that's "not obviously" or "obviously
not" less effort than rehashing the hard bit (ie. the CPS transform,
even leaving out the effect typing) probably only Tiark can judge. My
gut feeling is that the wrong call was made.

I don't think any of that is the hard part. Something I am in a position to judge is how much time is required to modify compiler code in nontrivial ways. The voracious time sink is uncovering the unknown-unknown interactions upon which existing code unintentionally depends. When you hack async-like functionality on top of CPS for your own use, you get to say "done" whenever you want. We have different requirements.

The last five years, the amount of time I've spent on anything a fraction as interesting as "CPS transform" or "effect typing" is at best 5%.

[Miles Sabin]

On Sun, Aug 25, 2013 at 10:57 PM, Paul Phillips <pa...@improving.org> wrote:
> On Sunday, August 25, 2013, Miles Sabin wrote:
> I don't think any of that is the hard part. Something I am in a position to
> judge is how much time is required to modify compiler code in nontrivial
> ways. The voracious time sink is uncovering the unknown-unknown interactions
> upon which existing code unintentionally depends. When you hack async-like
> functionality on top of CPS for your own use, you get to say "done" whenever
> you want. We have different requirements.

Fine. And how much easier does it get when you move the hard part from
a compiler plugin to a macro?

[Paul Phillips]

On Sun, Aug 25, 2013 at 3:04 PM, Miles Sabin <mi...@milessabin.com> wrote:

Fine. And how much easier does it get when you move the hard part from
a compiler plugin to a macro?

I don't know; I know that even if it isn't easier, effort expended on macros is more likely to pay off beyond the immediate task than is effort expended on the plugin architecture.

[Miles Sabin]

So, can we get this story straight: exactly what is under threat here
... the CPS plugin or the plugin architecture?

[Tiark Rompf]

It's always easier to start afresh than diving into an existing codebase, especially when the functionality it implements is not exactly trivial (like delimited continuations). 

The emphasis is on "start" though, because bringing a new implementation to maturity requires solving all the tricky issues again without necessarily leading to more maintainable code (google "second system effect" and "rewrites considered harmful"). 

Cheers,

- Tiark

[Paul Phillips]

I only speak for myself, now more than ever, so I don't know what if anything fits that description. I'm only telling you how things look from my let's say "unique" vantage. 

[Paul Phillips]

I used to have the "avoid starting fresh at all costs" religion, and then I met the pattern matcher. Not to compare the continuations code to the pattern matcher - but no longer do I fear starting over (a position I am taking to the extreme in about 3 weeks) now that I appreciate how much it can hurt to fail to start over.

[Meredith Gregory]

Dear All,

i love the idea of having two different ways to get async. i especially like the idea of having a mark to hit with the async over a refactored cps plugin. My hope is that if people can get all they want from async via async-over-cps and they can have backtracking, constraint-satisfaction and a host of other features, they will make a natural choice. With enough people making that choice support for the cps plugin will also be obviously in the best interest of the community.

Really, i think this is an excellent situation for all to benefit. There's plenty of work. Lot's of interesting, cool ideas and lots of interesting, brilliant people to engage them. In my personal perspective, i think Oleg's work on delimited continuations makes things crystal clear. This is a feature we won't want to live without; it's also so powerful that we really aren't equipped to engage it without type-based reasoning and support from tools like a compiler. However, for the same reasons i like playing music with other musicians is that they play things i wouldn't play, i like engaging computation with other computer scientists and programmers -- they do things i wouldn't do! With an open and welcoming attitude and gratitude for the abundance of opportunity here, we should have a blast!

Best wishes,

--greg

[Pierre Schaus]

that isn't using the CPS plugin. Availability bias may make this hard to believe,

but I'd be shocked to see more than 1% of Scala developers using the plugin.

You are surely right! 

Although I have the (maybe wrong) feeling that those 1% users may be important for the community.

Maybe those 1% are building important libraries used by many users, maybe those 1% are the ones trying hard to convince others that Scala is a great language.

By the way, IMHO another explanation of why continuations are not more used:

It is simply not documented in the book "Programming in Scala" (it is only explained on a few technical blogs and in the book Scala for the impatient). 

At UCL (Belgium) I taught to my students  what continuations is about using Scala.

Many students found that very impressive and powerful and they loved Scala!

It means that if I still want to teach continuations, I probably need to choose another language...

[Jason Zaugg]

On Sun, Aug 25, 2013 at 7:27 AM, Adriaan Moors <adriaa...@typesafe.com> wrote:

It's the cps plugin itself that's causing head aches, not the plugin infrastructure.

We won't deprecate the plugin infrastructure in 2.11, so it'll at least be around until 2.12.

I'd argue that AnnotationCheckers part of the plugin infrastructure does contribute headaches. Here's the latest: https://issues.scala-lang.org/browse/SI-7782.

-jason

[Meredith Gregory]

Dear Jason and Adriaan,

i just want to make sure i understand. Does TypeSafe plan to deprecate both the plugin infrastructure and the AnnotationCheckers? Are these features not to be relied on as supported past 2.11?

Best wishes,

--greg

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jason Zaugg]

Okay, that regression turned out to be unrelated to AnnotationCheckers.

-jason 

[Adriaan Moors]

We have no plans to deprecate the plugin/annotation checkers infrastructure in 2.11.

The only deprecations still pending for 2.11 will be to clean up dark corners of the library.

If the plugin infrastructure proves useful during 2.11, we're unlikely to deprecate it in 2.12,

but that's a discussion to be had in about a year.

In the mean time, note that we do not guarantee binary compatibility for the compiler classes.

We also don't have any regression testing in place for third-party compiler plugins.

We'd be happy to work with you to set that up, if you're up for some jenkins hacking :-)

The modularization effort is intended to force us to make the compiler a viable stable platform, but we're only just getting started with that.

Also, we're actively working on tooling to better detect binary incompatibilities.

[Adriaan Moors]

Glad to hear you had fun teaching continuations using Scala!

Again, deprecation != removal. You'll be able to use Scala 2.11 for years to come.

(Scala 2.9 -- released over 2 years ago -- is still commonly used, and was last updated 6 months ago.

 It won't be until well into 2016 that Scala 2.11 reaches that age.)

If indeed there's a huge demand for the cps plugin, I'm sure someone will step up and release it for 2.12 when the time comes.

It'll have its own repository on github with all the infrastructure to go with it.

[Meredith Gregory]

Dear Adriaan,

Many thanks! That's very helpful.

How does this relate to Scala virtualized? Is this going to be supported going forward? i'm mostly asking because i could really use access to for-comprehension syntax prior to desugaring. This can potentially impact our plans for a range of projects including the rearchitected cps plugin.

Best wishes,

--greg

[Adriaan Moors]

Greg,

The for comprehension syntax desugaring happens during parser, both in plain and virtualized scala (last I checked, anyway),

so I don't think that would help you. We may consider some form of language virtualization in 2.12,

but that's really up to the Scala team at EPFL.

cheers

adriaan

[Meredith Gregory]

Dear Adriaan,

Thanks! That is just one of those "It seemed like a good idea at the time" decisions.

Best wishes,

--greg

[Paolo G. Giarrusso]

On Tuesday, August 27, 2013 12:06:13 AM UTC+2, Adriaan Moors wrote:

Greg,

The for comprehension syntax desugaring happens during parser, both in plain and virtualized scala (last I checked, anyway),

so I don't think that would help you.

Scala-virtualized is about desugaring *more* constructs ("essentially" all) with the same spirit as for-comprehensions, so that you can overload those other constructs exactly like you overload for-comprehensions (by overloading map, flatMap and withFilter).

[Meredith Gregory]

Dear Paolo,

Thanks!

Basically, i want access to the for-comprehension before it is de-sugared in order to do syntactic analysis that will result in slightly different de-sugaring patterns depending on what's encountered. In our domain

for( ptn <- src if cond( ptn ) ) yield { e }

we need to determine what of cond and e can be sent to compilation for native storage processing and what must be done in memory. A very large chunk of this can be done syntactically.

Best wishes,

--greg

[Paolo G. Giarrusso]

On Tuesday, August 27, 2013 10:53:56 PM UTC+2, leithaus wrote:

Dear Paolo,

Thanks!

Basically, i want access to the for-comprehension before it is de-sugared in order to do syntactic analysis that will result in slightly different de-sugaring patterns depending on what's encountered. In our domain

for( ptn <- src if cond( ptn ) ) yield { e }

we need to determine what of cond and e can be sent to compilation for native storage processing and what must be done in memory. A very large chunk of this can be done syntactically.

Well, if you *really really really* want to do that syntactically, you'd need to move those desugaring out of the parser - something which Paul Phillips wanted to do anyway for ages.

But can't you do that after desugaring? After all, I'd usually report a bug if this code behaved differently:

src withFilter (ptn => cond(ptn)) map (ptn => e)

because I could well need to refactor the first code into the second (say, because I need to call some other function in between), and I don't want performance to drop for a simple refactoring.

And depending on your case, you can probably get a macro to do what you want.

The other approach is to use Scala-virtualized and LMS, which gives you a normalized tree representation (again the same for both pieces of code, and the same even for more complex refactorings). LMS is then actually used (in the Delite project) to turn such code into (for instance) GPU code, cluster code, and so on, so I think your use case might also fit there.

However, I would ask the maintainers before using LMS in production, since it's documented to be in alpha stage (http://stanford-ppl.github.io/Delite/index.html). After all, it's currently an (extremely cool) research prototype.

Do note please that I'm biased pro-LMS as I do research on it, but macros can be a plausible alternative in many cases, and they currently enjoy more support.

Best,

Paolo

[Stefan Zeiger]

On 2013-08-27 23:51, Paolo G. Giarrusso wrote:
>
> The other approach is to use Scala-virtualized and LMS, which gives
> you a normalized tree representation (again the same for both pieces
> of code, and the same even for more complex refactorings). LMS is then
> actually used (in the Delite project) to turn such code into (for
> instance) GPU code, cluster code, and so on, so I think your use case
> might also fit there.

Another alternative is YinYang
(https://infoscience.epfl.ch/record/185832) which works with plain Scala
instead of scala-virtualized. We're considering it for a new embedding
for Slick: https://github.com/slick/slick/tree/tmp/yy/cg

-sz

[Eugene Burmako]

Speaking of compiler plugins, their interface and power. Let's discuss that at https://groups.google.com/forum/#!topic/scala-internals/q8y9k4EoBNg.

[Hasan Amjad]

A bit late to the party but I am in somewhat the same situation as Sebastien wrt to lexical scoping and the need to execute in a single thread.

I am hoping to introduce Scala at my firm but the prospect of losing the delimited continuations feature at some unspecified point in the future makes this harder.

Unfortunately I do not have time to contribute personally. My firm might be able to contribute in terms of paying someone else to maintain it, but that will not happen unless the future of delimited continuations in Scala (as plugin or macro or...) is assured and they have proved their worth in our production code (a bit of a chicken and egg problem there, sorry).


On Thursday, August 22, 2013 9:58:12 AM UTC+1, Sébastien Doeraene aka sjrd wrote:

Hi all,

I really hope some CPS plugin will continue to exist. Async is a better, nicer solution under the assumption of lexical scoping. But capturing the full state of a thread across library boundaries is sometimes very useful, and only the CPS plugin provides that.

I also have but two weeks ago figured out that the CPS plugin was the solution to a difficult design issue I had in one of my personal projects. That project cross-compiles under Scala and Scala.js, and in JavaScript you don't have threads. CPS is the only solution I found to design the project in a sane way, considering the 1-thread-only limitation. Btw, yes, the CPS plugin works with Scala.js!

Cheers,

Sébastien

On Thu, Aug 22, 2013 at 9:45 AM, Philipp Haller <hal...@gmail.com> wrote:

Hi Greg,

This sounds very interesting!

On Wed, Aug 21, 2013 at 9:18 PM, Adriaan Moors <adriaa...@typesafe.com> wrote:

I'll let Jason fill in the gaps in my understanding,

but I believe scala-async uses a state machine translation,

because it doesn't require the full generality of the cps transform. 

(It does do an ANF transform, so I'm sure there's potential for code reuse.)

Exactly.

The Async SIP [1] contains a fully worked-out example of both the ANF transform and the state machine translation. To compare with the CPS plugin, we have a number of tests [2] that show what the current restrictions of Async are.

Cheers,

Philipp

 

[1] http://docs.scala-lang.org/sips/pending/async.html

[2] https://github.com/scala/async/blob/master/src/test/scala/scala/async/neg/NakedAwait.scala

On Wed, Aug 21, 2013 at 11:53 AM, Meredith Gregory <lgreg.m...@gmail.com> wrote:

Dear Adriaan, and All,

Thanks for all the helpful information!

Biosimilarity, LCC (Biosim), my company, had already concluded that we needed to own our own version of the CPS plugin. After deploying a production level project that used the plugin we realized that there were several desiderata in the plugin and the way we used it that we needed to address. We were planning on making it generally available to the community once we'd gotten it to the point where it was more help than harm. ;-)

Biosim would be happy to collaborate with TypeSafe and the community to help craft a CPS solution factored in a way that was flexible enough to support scala-async yet also address users like Swarm and SpecialK -- if that would be of interest.

Best wishes,

--greg

On Wed, Aug 21, 2013 at 10:48 AM, Adriaan Moors <adriaa...@typesafe.com> wrote:

Hi Ian, Greg,

I agree completely with Jason. If you're using CPS in production and would like support,

please get in touch and we'll try to work something out.

The community is of course more than welcome to maintain the CPS plugin,

but we see scala-async as a much, much nicer way of solving these issues.

Our experience has been that users of the continuations plugin have been eager to move to scala-async.

Also note that CPS has been effectively unmaintained for about 2 years now:

src/continuations/plugin/ was touched by about 50 commits in that period.

cheers

adriaan

On Wed, Aug 21, 2013 at 7:08 AM, Ian Clarke <ian.c...@gmail.com> wrote:

On Wednesday, August 21, 2013 1:55:39 AM UTC-5, Jason Zaugg wrote:

With all of this in mind, we have to find a way to communicate our view of CPS to potential new users (we'd rather that *more* projects didn't start depending on it) while providing some sort of migration path for your projects. The plugin could be maintained by a third party, for example, we're not planning to remove the extension points from the compiler. Or maybe we can suggest ways that you could use macros as your user API like async and effectful [1].

So long as you are thinking hard about how to avoid leaving existing CPS plugin users high and dry then I feel somewhat better, but I'm curious as to whether macros and async/effectful can offer the functionality we need (effectively serializing the state of a thread transparently to the programmer)?

I'm not familiar with Scala macros but will read up on them.

Ian.

--
L.G. Meredith
Managing Partner
Biosimilarity LLC
7329 39th Ave SW

Seattle, WA 98136

+1 206.650.3740

http://biosimilarity.blogspot.com

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Simon Ochsenreither]

So for everyone interested on starting to look into continuations, here is a probably an easy issue to get started: https://groups.google.com/d/topic/scala-internals/ZW_kunE7kiA/discussion
I think adding tests and having a better documentation about what works and which limitations exist would also be a nice starting point.

[Meredith Gregory]

Dear To-be-continued,

i've sent out an invite for a google hangout for a town hall discussion of a community supported delimited continuations plugin. i attempted to put everyone who seemed to have a stake in the discussion on the invite list. If i've left you off, just reply to this email and i'll add you.

Best wishes,

--greg

On Sun, Sep 8, 2013 at 9:33 AM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

So for everyone interested on starting to look into continuations, here is a probably an easy issue to get started: https://groups.google.com/d/topic/scala-internals/ZW_kunE7kiA/discussion
I think adding tests and having a better documentation about what works and which limitations exist would also be a nice starting point.

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Hasan Amjad]

Please add me, thank you.

You received this message because you are subscribed to a topic in the Google Groups "scala-internals" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scala-internals/9Ts3GLsXuOg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scala-interna...@googlegroups.com.

[ejc]

I'd like to watch the discussion if I'm available.

Thanks,

Eric

[Thorpe, Michael]

If there's a video transcript / ability to watch but not participate I'd be interested

Thanks,

Michael

[s...@n-side.com]

Hello All,

any update on the discussion you had ?

Thank you,

Sébastien

[Meredith Gregory]

Dear Sebastian,

Thanks for the prod! We had a good meeting with a lot of representatives from TypeSafe as well as Miles Sabin and myself.

My take away is that the infrastructure that a community supported delimited continuations plugin might need is as stable as we could hope given the nature of an open source project.

My hunch is that if Biosim++ ( = Biosim + whoever wants to help us with this ) creates a revision of the existing plugin that gains in popularity, then there will be continued support for the necessary infrastructure support.

It is tricky because there are several interacting elements: 

• compiler plugin architecture
• type checker
• annotations
• delimited continuations compilation plugin
• Scala type system
• delimited continuations type and effect system. 

We touched on this briefly, but i suspect that -- in full generality -- there needs to be a recursive or re-entrant relationship between type checker and a given plugin -- especially in the case that the plugin works with an extended type system that embeds into the existing type system and generates code. Tiark had a different take that suggested that with the delimited continuations plugin, at least, one could just do a two pass approach. 

i want to have one more meeting in which projects that are dependent on the plugin -- as well as folks who are hoping to use async -- are more represented. The aim of this meeting would be to get a better sense of requirements from folks in the field actually using this stuff. We've got a rough idea of what's likely to remain and for how long. We don't have a sense -- apart from Biosim's admittedly idiosyncratic use of the plugin -- of how or what people are actually using it for and what they really need supported. 

If people don't speak up about this to Biosim, we can only go with the requirements we know.

Best wishes,

--greg

[Adriaan Moors]

As announced back in August, I just released scala-continuations-plugin and scala-continuations-library 1.0.0-RC1 against Scala 2.11.0-M7 (on sonatype, on its way to maven central).

The repository: https://github.com/scala/scala-continuations. There's a standard sbt build and Travis CI is keeping an eye out.

The project will also be part of an integration build (more on that later).

Just like the other scala modules, these jars will be included in the Scala 2.11 distribution.

Note that the project is marked as "unsupported" to indicate that Typesafe does not provide support for it.

In other words, the Scala team at Typesafe will be manning the PR queue until a maintainer steps up.

cheers

adriaan

[Simon Ochsenreither]

Hi!

There is currently work going on to move Avian's continuation support to delimited continuations. Joel, the person working on it is asking for some input about it here: https://groups.google.com/d/msg/avian/yrdlmjLy1Ng/CIYxJ7QqoEUJ
It would be great if the continuation experts here could add some expertise and answer the questions raised in the last post of that thread and in the commits linked in this thread, e. g.:

I've been told by knowledgeable people that it is impossible to
implement composable continuations (AKA delimited continuations AKA
shift/reset) in terms of call-with-current-continuation.  Since I
don't yet understand why that is, I figured it would help my
understanding to attempt it and see how it fails.

and

I'm mostly curious about what tests we could add to

ComposableContinuations.java which will fail using this implementation.

There's nothing like a concrete, failing test to focus the mind.

Even if you never intend to use Avian, keep in mind that it will be a hugely helpful testing platform where you can compare what the continuation plugin does with the natively supported continuations on Avian. E. g. "is this supposed to work?" can be answered in a few seconds if you can just switch out the implementation.

Please help to get the implementation of delimited continuations nailed down in Avian. If this works out, I'll look into adding Avian support to the CPS plugin (e. g. instead of doing the CPS tranformation, just add the right calls to avian.Continuations.shift/reset).

Thanks,

Simon

[Anthony Di Franco]

Regarding the first question, let me refer you to the section "call/cc implements shift? A good question" here:

"In the POPL 1994 paper Filinski showed that ``that the composable-continuations construct can itself be represented using ordinary, non-composable first-class continuations and a single piece of state..."

"Unfortunately the context of the POPL94 remarkable result, its goals and the scope, and hence its side-conditions are often forgotten. The POPL94 conclusions are frequently taken beyond their intended application area, causing problems and confusion. This article reminds of the POPL94 motivations and restrictions. Practical language systems invariably violate the side-conditions, making the POPL94 result inapplicable. Using the POPL94 implementation of shift in terms of call/cc then leads to confusion because the implementation no longer confirms to the semantics of shift. We demonstrate two simple counter-examples of the difference between the predicted, on the basis of the semantics of shift, results and the observed ones."

"The problem arises precisely because call/cc captures the whole continuation rather than its prefix. Such a behavior of call/cc is not a problem within the POPL94 framework since it presupposes that shiftis the only effect (with all others implemented in terms of it). Even call/cc cannot be used except for its appearance in the implementation of shift. In practice, effectful operations such as exceptions or dynamic binding are implemented independently, breaking the main assumption of POPL94 and exposing the problem of capturing the whole continuation. In these practical circumstances, call/cc does not implement shift or other control features. They have to be provided as primitives..." (etc.)

[Simon Ochsenreither]

It works.

  def fib2(n: Int): Int = {
    def fib0(n: Int, sub1: Int, sub2: Int): Int = n match {
      case x if (math.abs(x) < 2) => shift { k: (Int => Int) => k(x) }
      case x => fib0(x - sub1, sub1, sub2) + fib0(x - sub2, sub1, sub2)
    }
    val s = math.signum(n)
    fib0(n - s, s, s * 2)
  }

  val fibonacciNumbers = List(Int.MinValue, 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811, 514229, 832040, 1346269, 2178309, 3524578, 5702887, 9227465, 14930352, 24157817, 39088169)
  (1 to 20) foreach (n => println(reset { fib2(n) }.ensuring(_ == fibonacciNumbers(n), (n, reset { fib2(n+1) }, fibonacciNumbers(n)))))

Note the absence of any kind of @cps annotations.

[Adriaan Moors]

Very cool! (I'm assuming this is on avian?)

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.