Scala STM for 2.9?

[martin odersky]

All,

We will release the first RC for Scala 2.9 soon (propbably next week). I know it's late, but I wanted to ask whether you think it would be worthwhile to ship the Scala STM API with that RC?  Is the API stable enough that you feel it should be included?

At the moment I would not yet ship an implementation for the API (AFAIK there are two), just the interfaces. But we should do it only if these interfaces are stable. It's always hard to change an interface that's already used by clients.

Cheers

 -- Martin

[Daniel Spiewak]

I'll leave the stability question to Nathan, since he's vastly more familiar with the area than I am.  I would like to question why you want to ship the API without the implementation?  That would seem to reduce the usefulness nearly to zero.

Daniel

[Krishna Sankar]

I have the same question. What would the programmers be able to do without an impl ? Usually folks learn by hands-on programming, nobody ever reads the manuals or look at the api (unless of course they are stuck ;o))

Cheers

<k/>

[Nathan Grasso Bronson]

Martin,

I think that both the API and the reference implementation are ready, provided that someone on the Akka team agrees.

I don't think that there's any need to hold back the reference implementation, except perhaps to delay the work of migrating the unit tests from scalatest to partest.

Regards,

  Nathan

From: martin odersky <ode...@gmail.com>
Reply-To: <scala-stm-e...@googlegroups.com>

[Peter Veentjer]

Hi Nathan,

I would also not release the STM api without an implementation. I'll reinspect it this weekend. One of the things I'm not completely clear about is the
numbers of objects being created. For the Multiverse STM it is limited to a single object per transaction (the closure). Everything else is pooled. Object
creation (gc) just kills scalability.

Do you know out of your head how many object creations are mandatory?

[Peter Veentjer]

The STM in the Akka project is going to implement the Scala STM API in a few steps.

In the next release it will have roughly the same names and structure, but the interfaces
are not implemented. That will be something of the proceeding releases.

[Nathan Grasso Bronson]

Peter,

The only object that is mandatory during either a successful or failed transaction is the closure object that represents the atomic block.

InTxn instances can be reused, and I was careful to structure the NestingLevel class (the first-class representation of the current attempt) so that it can be created lazily. In addition, the Ref and Ref.View traits can be implemented by the same concrete class.

Thanks,

  Nathan

[Nathan Bronson]

Porting the scalatest tests to partest wasn't too difficult. I've now
got a local build of scala 2.9 that includes scala-stm and all of its
tests.

- Nathan

[Daniel Spiewak]

In case there was any doubt, I'm very much in favor of including the STM API and implementation in Scala 2.9.  My only concern is that it is a little late in the 2.9 cycle to be making additions like this.  However, given that the STM has been stabilizing and gaining acceptance in its own right (and it is a strictly additive change), I don't see this as being an issue for the larger 2.9 library and core.

Daniel

[Daniel Sobral]

Does it generate its own jar, like swing, or is it being stuck inside the main library jar? If it were to be placed in a separate jar, I'd expect there would be little to complain about putting it on Scala main distribution.

Now, one can also think of scala.dbc, which is abandonware. It doesn't bother people much because it doesn't appear on scaladoc and it has a separate jar file. So that serves as a counter-example. But I see STM much more in the same light as actors -- there are alternative implementations nowadays, but its presence on the Scala distribution is a significant factor in drawing people to Scala.

Side note: when I decided to learn Scala, I thought it came with an STM implementation. It was a big disappointment to learn this was not true, though the rest of the language more than made up for it. :-)

--
Daniel C. Sobral

I travel to the future all the time.

[Sciss]

hi all,

although i'm not actively involved in any of the implementations, i would like to say that i pretty much share the thoughts of daniel.

on the one hand, the question is if really more packages should be bundled with scala-library or if not the approach of scala-swing is better. after all, the main point is to have an official standard for the API. on the other hand, like actors, having the STM readily there will be a great advantage. and i suppose the stm would run fine on other systems such as android.

it also signalised that scala shines when it comes to concurrency -- we have java.util.concurrent for free, we have scala actors [*], and now we also have STM which is not only incredibly useful in practice, but also good PR regarding scala's competing languages (clojure, haskell).

given that i have been working for over a year now successfully with CCSTM, i'm pretty confident that the ccstm based implementation of the scala STM api should be good.

i also think it doesn't make sense to add the API without the implementation.

however, i would think that the parties trying to do other implementations (akka, multiverse) should be checking very carefully that by releasing the API now they're not going to run into future problems when they are actually doing their implementations.

2c, -sciss-

[*] couldn't that actual be made an API with different implementations? like the original scala actors, lift actors, akka actors etc.?

[Nathan Bronson]

From a usage perspective I think that like actors the STM belongs in
scala-library, because it is useful for all Scala platforms.

The primary practical concern that I see is an increase in the size of
scala-library. Compressed, the STM API is about 300K; the reference
implementation is another 230K. (A lot (most?) of the bulk comes from
the forwarding stubs generated when a class extends a trait. Extending
mutable.Map results in > 200 stubs, for example.) For reference
scala-library.jar is about 9M. The STM would be stripped out for
Android targets if it wasn't used.

The biggest potential future problem for other implementers is the API's
inclusion of features that require support for partial rollback, namely
failure atomicity (rolling back on exception) and modular blocking
(retry/orAtomic, AKA retry/orElse). We discussed this on the mailing
list a while ago and agreed that it was worth the effort.

- Nathan

[David Pollak]

I'd recommend waiting.

If the actor library is a guide... well the library has a bad interface and a worse implementation but because it's baked into the scala distribution there's substantial momentum and ego in not improving it.

I think it's worth spending a year with the stm stuff outside the main distro so it can be gotten right.

[Daniel Spiewak]

In principle, I agree.  However, CCSTM has been around for quite a while now, much more than a year.  Some bits of the API have been around even longer still (see my article on STM).  I'd have to check to see exactly how long Scala STM itself has been around, but it feels like nearly a year.  Granted, some things have changed in that time, but it seems like we're settling down on a refined and stable interface and implementation.

We could stay out of the main distro for another release cycle (targeting Scala 2.10?), but I'm not sure how much that will accomplish.  I'm sure we'll find one or two corners to sand, but I don't think anything major is going to crop up.  As it stands, we have an opportunity here to carry our momentum forward and catapult STM into the Scala standard library.  That's huge.  A lot of people right now are looking at Scala, seeing that it doesn't have STM in the standard library and thinking twice about using it.  That's not to imply that STM is critical to Scala's success or anything like that, but I think there is a lot of temptation right now to draw comparisons between Scala and Clojure.  Clojure has an STM built into the core, Scala doesn't.  We have an opportunity to change that here.  Let's not undervalue that.

I think the actor library is a bad parallel in this instance.  It was built quite a while ago, mostly in academic isolation and partially as a proof of concept.  Part of why the actor library has a bad interface is it was built before a lot of what we now call "good interface" had emerged in Scala.  It's important to remember that Scala is no longer changing anywhere near as rapidly as before.  We can safely expect "good APIs" to remain "good" for many releases to come.  As for the actor library implementation, well, I've bled too much on that one to give it a free pass.  Here again though, the actor library was developed in isolation with limited testing and even less industrial feedback.  As Sciss pointed out, Scala STM has been applied to projects.  It's been hammered, refined and tested substantially more thoroughly than the actor library, the JSON support, or indeed many of the older frameworks we find in the standard library.

So, I hear the note of caution you're sounding, but I think it's unwarranted in this case.  It's time for Scala STM to take its rightful place in the standard library.

Daniel

[Nathan Grasso Bronson]

Scala-stm's presence on the scala-tools repo makes the barrier for usage pretty low (although not zero) and there is no performance difference between scala-stm as a JAR and scala-stm inside the standard lib. This means that what we're primarily considering is the signal from the core Scala team about the STM API's stability and long-term maintenance future.

Perhaps a compromise between full commitment and status quo is to release scala-stm 1.0, based on 0.3-snapshot. This captures my personal belief in the quality of the API, and my commitment to support it, without asking too much of others. There is nothing to stop full STM integration from occurring during a minor release.

There's also the question of language comparisons (X has STM, but Scala doesn't). I consider it a good thing that the ScalaSTM API can be implemented entirely as a Scala library.

 - Nathan

From: Daniel Spiewak <djsp...@gmail.com>
Reply-To: <scala-stm-e...@googlegroups.com>
Date: Sat, 12 Mar 2011 22:12:30 -0600
To: <scala-stm-e...@googlegroups.com>

[Daniel Spiewak]

Martin, we haven't heard from you since the initial post.  Could you provide some insight as to the decision making process here?  We've provided our input, but Scala 2.9 RC1 is now out, so it looks like our window is rapidly closing, if not already closed, on getting this (highly anticipated) feature into the current release cycle.

Daniel

[martin odersky]

On Fri, Mar 25, 2011 at 9:47 PM, Daniel Spiewak <djsp...@gmail.com> wrote:

Martin, we haven't heard from you since the initial post.  Could you provide some insight as to the decision making process here?  We've provided our input, but Scala 2.9 RC1 is now out, so it looks like our window is rapidly closing, if not already closed, on getting this (highly anticipated) feature into the current release cycle.

Sorry, I corresponded privately with Nathan and Peter Ventjeer. I did not realize that the list was not part of this.

The upshot is that we'll wait one minor release (i.e. a couple of months). We still need to figure out whether we should just ship the API or also a reference implementation. Once that's done, we can ship in 2.9.1.

The plan is certainly that we will ship a STM with Scala. It just was a bit late for this release. But that means we should not slack off now. The right time to add new functionality to trunk is not just before a release but just after one. This gives us time to thoroughly test features before we have to freeze them. So, as soon as 2.9 is out, we should get started to integrate the STM API in trunk. Then we should figure out whether we should also commit to a reference implementation or whether we should offer a choice (e.g. CCSTM, Multiverse), and how that should be done.

Cheers

 -- Martin

[Daniel Spiewak]

Understood.  Thanks for the clarification!

Daniel

[Jonas Bonér]

I think it is the right thing to do to NOT ship the impl. It is what Sun has done in Java for years. Ship the interfaces, create a reference impl that people can  download and then open up for other vendors. F.e look at JAX-RS. The interfaces are shipped and then people can download Jersey if they like. Same thing with JTA. EJB, JPA etc. etc. 

Shipping an impl will make it harder to use something else, like Multiverse. It also blesses one impl over another which I find very unfortunate. In Akka f.e. we are using Multiverse. Shipping CCSTM with Scala would make it clash with Multiverse. When you create a TX and get auto imports in Eclipse, you will have a choice of which impl to use. If one of them is 'scala.stm' something then people will naturally go for that. It locks things in instead of trying to create an open platform.

I'd like to see Scala take this route for all additional APIs that are added.

/Jonas

--
Jonas Bonér

Specialist at Large
work: http://scalablesolutions.se
code: http://akka.io
blog: http://jonasboner.com
twtr: @jboner

[Sciss]

but then: what would be the advantage then to have the API at all shipped instead of having it, too, as a separately downloadable package? i mean, if i want to add an implementation, let's say to CCSTM, i could as well add the API as a second dependancy with zero effort, no?

best, -sciss-

[Jonas Bonér]

On 26 March 2011 19:59, Sciss <con...@sciss.de> wrote:

but then: what would be the advantage then to have the API at all shipped instead of having it, too, as a separately downloadable package? i mean, if i want to add an implementation, let's say to CCSTM, i could as well add the API as a second dependancy with zero effort, no?

It would be the natural thing to do if it is the standard. It would become the standard by putting it in the Scala dist, as an API. 

[Peter Veentjer]

After have given it some thought, perhaps it is better to release the API without the implementation. Or perhaps even wait with releasing the API until the Multiverse implementation implements it. A good API can't be extracted in one go, but needs to have different implementations to mold it in the right form.

On Fri, Mar 11, 2011 at 7:49 PM, Peter Veentjer <alarm...@gmail.com> wrote:

[Nathan Grasso Bronson]

Jonas,

I understand your concerns about blessing one implementation over another, but I think we've made the footings as equal as possible while keeping the initial friction low.

The API is structured so that the user only interacts with generic classes. All of the implementations (reference + Multiverse) will appear to the end user as scala.concurrent.stm.* (except where they use an implementation-specific extension, of which CCSTM doesn't have any). This is important so that the entire program (including all of its libraries) will end up using a single STM implementation.

The only way in which CCSTM is blessed by the API is that it is used as a fallback at runtime if the system property "scala.stm.impl" is not defined. Any code or library that was developed using the reference implementation will also run using Multiverse (once it implements the API) without recompilation. Only one class from the reference implementation is public, and that is only so that it can be created via Class.forName.

If we don't include a reference implementation, then localized or exploratory use of the STM will be harder. Ideally if someone answers a "how do I …" question with a snippet of 10 lines that includes a transaction, that code will just work in the REPL. If that code is then run inside Akka it will transparently run using Multiverse.

 - Nathan

[Nathan Grasso Bronson]

Peter,

It would definitely be better if there was a second implementation of the API before it is included in a standard library release, while the API is still easier to change. Can we do this and still commit to 2.9.1?

 - Nathan

From: Peter Veentjer <alarm...@gmail.com>
Reply-To: <scala-stm-e...@googlegroups.com>

Date: Sat, 26 Mar 2011 20:34:16 +0100
To: <scala-stm-e...@googlegroups.com>

[Daniel Spiewak]

I feel very strongly on this point: we need to include a default implementation.  I think Java framework examples make this point beautifully.  Shipping the API without the implementation may be the more generic, "enterprisy" (I hate that adverb) thing to do, but it dramatically reduces the utility of the API.  In the case of Java, this ended up being an almost universally despised design decision.  In fact, you're the first person I've met who actually likes it!

Consider, for example, the JAXP library.  Almost everyone working on the JVM has to do XML processing at some point.  So, of course, they turn to the built-in support.  Until recently (well, a few years ago), the "built in support" consisted entirely of an API with no implementation.  Worse than that, the lack of implementation wasn't known statically, so people would write code, try to run it and then be mystified by the reflection errors which cropped up.  It always baffled me that I would need to download some third party library which I would never use directly (and thus would be seen as a "dead" dependency) to enable functionality which was supposed to be built into rt.jar!  The JCE framework (cryptography) still suffers from this problem to a large degree, since anyone who does any real work with JCE needs to go out and download Bouncycastle.  This also raises non-coding issues.  For example, I have (more than once) had to justify to management why our code base was dependent on some random third party framework.  (I've had this problem with Xerces too, just not recently)

Judging by developer mindshare, the "batteries not included" approach to language framework design has really gone the way of the dodo.  Developers want to start hacking using a language and expect all of its built-in APIs to work as advertised out of the box.

So, this is something that I'm going to argue quite strenuously: we need to include a default implementation for Scala STM, and more generally, Scala must continue to go that route moving forward.  This is even more important to me than the question of which implementation will be included.

Daniel

[Peter Veentjer]

On Sun, Mar 27, 2011 at 10:46 PM, Daniel Spiewak <djsp...@gmail.com> wrote:

I feel very strongly on this point: we need to include a default implementation.  I think Java framework examples make this point beautifully.  Shipping the API without the implementation may be the more generic, "enterprisy" (I hate that adverb) thing to do, but it dramatically reduces the utility of the API.  In the case of Java, this ended up being an almost universally despised design decision.  In fact, you're the first person I've met who actually likes it!

Perhaps it is better to wait releasing the Scala STM API, since there is only a single implementation. A good API can imho only be created after multiple implementations. In a few weeks I'm going to have much more time for getting Multiverse implementation of the Scala API up and running and see what works and what doesn't work.
 

[Daniel Spiewak]

I feel very strongly on this point: we need to include a default implementation.  I think Java framework examples make this point beautifully.  Shipping the API without the implementation may be the more generic, "enterprisy" (I hate that adverb) thing to do, but it dramatically reduces the utility of the API.  In the case of Java, this ended up being an almost universally despised design decision.  In fact, you're the first person I've met who actually likes it!

Perhaps it is better to wait releasing the Scala STM API, since there is only a single implementation. A good API can imho only be created after multiple implementations. In a few weeks I'm going to have much more time for getting Multiverse implementation of the Scala API up and running and see what works and what doesn't work.

I agree on this point (didn't have a chance to reply to that message yet).  :-)  I was under the impression that the Multiverse implementation was more or less at parity with the CCSTM backend.  I think we need to wait on pushing the API into the stdlib until we get the Multiverse impl to that level (or at least close).  Without at least two implementations, we don't really know if the API is sufficiently general.

Daniel

[Jonas Bonér]

Hi Nathan. 

On 26 March 2011 22:28, Nathan Grasso Bronson <nbro...@cs.stanford.edu> wrote:

Jonas,

I understand your concerns about blessing one implementation over another, but I think we've made the footings as equal as possible while keeping the initial friction low.

So far yes. 

 

The API is structured so that the user only interacts with generic classes. All of the implementations (reference + Multiverse) will appear to the end user as scala.concurrent.stm.* (except where they use an implementation-specific extension, of which CCSTM doesn't have any). This is important so that the entire program (including all of its libraries) will end up using a single STM implementation. 

The only way in which CCSTM is blessed by the API is that it is used as a fallback at runtime if the system property "scala.stm.impl" is not defined. Any code or library that was developed using the reference implementation will also run using Multiverse (once it implements the API) without recompilation. Only one class from the reference implementation is public, and that is only so that it can be created via Class.forName.

This sounds good. But we still have a problem if a system is using two libraries that are using two different impls of Scala STM (both perhaps even use extensions provided). 

[Jonas Bonér]

On 27 March 2011 22:46, Daniel Spiewak <djsp...@gmail.com> wrote:

I feel very strongly on this point: we need to include a default implementation.  I think Java framework examples make this point beautifully.  Shipping the API without the implementation may be the more generic, "enterprisy" (I hate that adverb) thing to do, but it dramatically reduces the utility of the API.  In the case of Java, this ended up being an almost universally despised design decision.  In fact, you're the first person I've met who actually likes it!

I like it because almost every reference impl Sun has created have been crap. 

JTA, JPA, EJB, Servlet spec etc etc.  

 

Consider, for example, the JAXP library.  Almost everyone working on the JVM has to do XML processing at some point.  So, of course, they turn to the built-in support.  Until recently (well, a few years ago), the "built in support" consisted entirely of an API with no implementation.  Worse than that, the lack of implementation wasn't known statically, so people would write code, try to run it and then be mystified by the reflection errors which cropped up.  It always baffled me that I would need to download some third party library which I would never use directly (and thus would be seen as a "dead" dependency) to enable functionality which was supposed to be built into rt.jar!  The JCE framework (cryptography) still suffers from this problem to a large degree, since anyone who does any real work with JCE needs to go out and download Bouncycastle.  This also raises non-coding issues.  For example, I have (more than once) had to justify to management why our code base was dependent on some random third party framework.  (I've had this problem with Xerces too, just not recently)

Judging by developer mindshare, the "batteries not included" approach to language framework design has really gone the way of the dodo.  Developers want to start hacking using a language and expect all of its built-in APIs to work as advertised out of the box. 

I see this point and actually agree that it is ideal but then I'd like to see all the potential issues ironed out first. I'm not talkin about issues in CCSTM here which is right down mind-blowing good stuff (as Multiverse) but in making sure that multiple impls can co-exist in the same runtime (without OSGi). 

 

So, this is something that I'm going to argue quite strenuously: we need to include a default implementation for Scala STM, and more generally, Scala must continue to go that route moving forward.  This is even more important to me than the question of which implementation will be included.

It hasn't worked out that well so far; actors, xml etc. Perhaps because they were put in premature. I think that we (as discussed) should wait until we have at least 2 impl implementing the API so we can ensure not having to deprecate stuff in the upcoming releases. 

[Nathan Grasso Bronson]

I agree that having two implementations of the API would be best, and I'm glad to hear that Peter will be able to devote more time to that in the near future.

The issue of single versus multiple impls per JVM is separate from the inclusion of the reference implementation. I think it will be best if the reference implementation supports only the base API with no extensions. This means that there will be no barrier to moving from the RI to Multiverse. There might be a barrier moving in the other direction, but that's fine.

I haven't had time to digest Daniel's implicits idea, hopefully I can read it thoroughly tomorrow. At the moment I'm doubtful that supporting multiple impls per application is a good idea.

 - Nathan

From: Jonas Bonér <jo...@jonasboner.com>
Reply-To: <scala-stm-e...@googlegroups.com>
Date: Mon, 28 Mar 2011 08:11:52 +0200
To: Daniel Spiewak <djsp...@gmail.com>
Cc: <scala-stm-e...@googlegroups.com>, martin odersky <martin....@epfl.ch>
Subject: Re: [scala-stm-expert-group] Scala STM for 2.9?

[Sciss]

in any case it would be very useful if the other implementations could oversee the API as soon as possible, because i would like to use it at this very moment, and i have just converted some stuff from the previous CCSTM API to the new ScalaSTM API and i would like to minimize this effort for an upcoming version of ScalaSTM. BTW, thanks for the 0.3 release, it's a good thing to have this condensed in a stable version!

i understand that people are really busy, but hasn't this group been created some 9 or 10 months ago? i would have thought there was enough time to at least take a look at the efforts which in my perception have been mostly driven by nathan's work, and figure out whether it was adoptable for the other projects. i find the complaints now about the missing second implementation a bit late.

best, -sciss-

[Jonas Bonér]

I don't think we should rush it. If ScalaSTM is close to beta-final then are you not fine just using that? 

[Nathan Grasso Bronson]

Sciss,

Go ahead and use scala-stm 0.3. If adding Multiverse leads us to trim a feature you use from the API (and therefore the reference implementation) then I will split out a non-blessed CCSTM that includes those features. Alternately, we'll come up with a shim that will involve minimal work for you (perhaps just an import).

Thanks

[Jonas Bonér]

Hi guys.

The Scala 2.9 train have passed long ago and it is too late to jump on the Scala 2.10 train as well. 

Should we aim for Scala 2.11? 

Should we create a SIP? 

Should we skip it completely and let Scala STM continue to be an independent project? 

WDYT? 

/Jonas

--
Jonas Bonér
CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner

[Daniel Sobral]

This and IO have kind of been waiting for the incubator project,
haven't they? Or, at least, that's what I think the situation is.

[Josh Suereth]

I think they're more waiting for heavy users.  Incubator is supposed to help get users to test the designs and work out kinks.   If IO had a ton of vocal users, I bet you dollars to peanuts it'd be in Scala 2.10 or slotted for 2.11.

- Josh

[Daniel Sobral]

On Wed, May 23, 2012 at 9:58 AM, Josh Suereth <joshua....@gmail.com> wrote:
> I think they're more waiting for heavy users.  Incubator is supposed to help
> get users to test the designs and work out kinks.   If IO had a ton of vocal
> users, I bet you dollars to peanuts it'd be in Scala 2.10 or slotted for
> 2.11.

True, but I think visibility plays a heavy part here. Libraries don't
get as many users as frameworks, as they don't build communities as
easily. That's why gem, pip and cpan are so popular: they make
libraries discoverable in the absence of communities. Scala is missing
that, despite repeated efforts to create it.

[Daniel Spiewak]

I guess the question I would ask is the following: what do we expect to discover if Scala STM were moved into the incubator?  I would posit that most people who really need an STM in Scala now have already looked at Scala STM.  The audience we're missing are the people who don't know they need an STM, or perhaps the people who's need is less acute.  However, this group of people are no more likely to use an incubator project (that they may or may not know about) than a third party library.

In other words, I don't think an incubator stage would gain us very much at this point.  We've been hemming and hawing about this framework for two years now.  Heck, even Java stdlib enhancements tend to move faster!  I say we just drop it into the Scala standard library and move forward from there.

Daniel

[Nathan Bronson]

There is no technical barrier to moving the STM into the standard
library (proof by example:
https://github.com/nbronson/scala/commit/4e08a1bee01e389029add5cfb00b3406953cb943
). At the moment the STM itself doesn't gain extra powers or
efficiencies by being in the stdlib, it is just a matter of build and
distribution convenience (both pros and cons) and of an official
blessing. So far I have managed to keep a single source image for all
of the targeted Scala versions, but that might be harder in the
future, at which point inclusion in the stdlib would be convenient.

Not too many people ask questions on this list, which means the STM
doesn't have very many direct users, direct users don't have very many
problems, or direct users don't ask when they have a problem.
Multi-threaded programming is tricky, so even though I've tried to
make high-quality documentation I suspect the first case.

- Nathan

--
Nathan Grasso Bronson
ngbr...@gmail.com

[Sciss]

perhaps the list name should be changed? i certainly thought it was to discuss between the experts how to develop the stm api. or maybe a separate scala-stm-users list would stimulate people to participate with user questions?

best, .h.h.

[Nathan Bronson]

Changing the list name is a good idea. We probably don't need two. How
about "scala-stm"?

[Jonas Bonér]

On Wed, May 23, 2012 at 5:29 PM, Daniel Spiewak <djsp...@gmail.com> wrote:

I guess the question I would ask is the following: what do we expect to discover if Scala STM were moved into the incubator?  I would posit that most people who really need an STM in Scala now have already looked at Scala STM.  The audience we're missing are the people who don't know they need an STM, or perhaps the people who's need is less acute.  However, this group of people are no more likely to use an incubator project (that they may or may not know about) than a third party library.

In other words, I don't think an incubator stage would gain us very much at this point.  We've been hemming and hawing about this framework for two years now.  Heck, even Java stdlib enhancements tend to move faster!  I say we just drop it into the Scala standard library and move forward from there.

+1

[Jonas Bonér]

On Thu, May 24, 2012 at 12:02 AM, Nathan Bronson <ngbr...@gmail.com> wrote:

There is no technical barrier to moving the STM into the standard
library (proof by example:
https://github.com/nbronson/scala/commit/4e08a1bee01e389029add5cfb00b3406953cb943
). At the moment the STM itself doesn't gain extra powers or
efficiencies by being in the stdlib, it is just a matter of build and
distribution convenience (both pros and cons) and of an official
blessing. So far I have managed to keep a single source image for all
of the targeted Scala versions, but that might be harder in the
future, at which point inclusion in the stdlib would be convenient.

Not too many people ask questions on this list, which means the STM
doesn't have very many direct users, direct users don't have very many
problems, or direct users don't ask when they have a problem.

Putting it int the standard lib will give it a *much* larger exposure and many more users will most likely follow. 

[Jonas Bonér]

I agree. Go for scala-stm.

[Josh Suereth]

But you lose the ability to make changes.   I'm not so sure it's all unicorns and win going into the STD. Library....

[Jorge Ortiz]

what about including scala-stm.jar and scala-io.jar in the standard distribution, but not the standard library? (including hosting scaladocs on scala-lang.org, etc)

e.g., "bless" it, but don't bundle it.

--j

[Josh Suereth]

You still have to maintain binary compatibility with scala distros then.

I'm not against it, but e.g. I'm not trying to push scala-arm in the distro immediately either....

[Daniel Spiewak]

I think people would just ignore those jars.  I know I never used scala-swing or any of the other toys that were included in this way.

Daniel

[martin odersky]

Putting it into the standard library is not so easy. Who will maintain it? Who will maintain it when the first maintainer goes away? Who will ensure binary compatibility for minor releases. Is it even desirable that all these libraries develop in lockstep with Scala, with binary changes allowed only for every major Scala version (in the future there might be a 2 years gap between such versions)? If we agree to that principle, who will make sure it is met? These are in practice very tricky questions. And we still suffer from the fact that, for instance, XML does not have an effective maintainer yet we have to keep it in stdlib. 

Also, the argument that we should put a module into stdlib because it is currently not used much is putting the horse before the cart. stdlib needs to be a consolidator for stuff that everybody uses; it can't be a promoter for things that some of us believe are underused.

For these reasons I think it would be best to have a set of "semi-blessed" modules that we build and ship alongside the standard library, but that have a different status. They will not automatically have the same guarantees of backwards compatibility and forwards maintenance. But they give the assurances that they build with trunk, have active maintainers, and have a certain standard of documentation. 

As first candidates for such a community library I'd see at least scala-stm, scala-arm, scala-io, scalacheck, and some testing frameworks.

We really need to get going on that now, I think.

Cheers

 - Martin

[Josh Suereth]

I'm working on it.

[Daniel Sobral]

On Thu, May 24, 2012 at 5:05 PM, Josh Suereth <joshua....@gmail.com> wrote:
> But you lose the ability to make changes.   I'm not so sure it's all
> unicorns and win going into the STD. Library....

In what sense you lose the ability to make changes? Can STM change now
at will? If it can, then no one is using it -- only experimenting with
it. There's much to be said about not being able to make changes
because there are people using it for production code.

[Daniel Sobral]

On Thu, May 24, 2012 at 5:45 PM, Daniel Spiewak <djsp...@gmail.com> wrote:
> I think people would just ignore those jars.  I know I never used
> scala-swing or any of the other toys that were included in this way.

Ah, but you'll use scala-reflect.jar, and that will lead you to use other jars.

You never used scala-swing because you didn't build swing
applications, or, if you did, it was something small enough that you
went with Java API. If we transition to a model where the library _is_
split on many jar files (parsers and xml are prime candidates for
splittling, for example), people will get used to adding more
dependencies.

[Josh Suereth]

On Thu, May 24, 2012 at 4:59 PM, Daniel Sobral <dcso...@gmail.com> wrote:

On Thu, May 24, 2012 at 5:05 PM, Josh Suereth <joshua....@gmail.com> wrote:

> But you lose the ability to make changes.   I'm not so sure it's all
> unicorns and win going into the STD. Library....

In what sense you lose the ability to make changes? Can STM change now
at will? If it can, then no one is using it -- only experimenting with
it. There's much to be said about not being able to make changes
because there are people using it for production code.

In the sense that you don't have the expectation of binary compatibility now.

I hate to be a negative nancy (but I have to for my own library):   You need to improve users, I feel, before hitting the standard library.    When the "blessed" library bit is ready (i.e. the build utlity I'm working on is finished), then we can move you into there, but it's far too early for the standard library.

I think it'd be better to do some PR, evangelize and give more talks about STM.  Hype people up about it.  Don't rely on the standard library to be your PR.  You have to promote this project yourselves.  You have something to promote, it's stable.  You should be able to get users.   Perhaps you need to promote *other libraries* to use yours to improve users.   Either way, the users will stabilize the API, and you can get them today, with some effort.

The way to get more users is promotion.  Even if it hits the standard library, we still need a way to make people aware of it.  The standard library isn't a crutch to get users of a feature, and it should not be used that way.  It's the same for scala-arm and scala-io.   I'd love to push an IO library more than anything.  BUT, I don't think scala-io has solidified enough to warrant usage, and Scala-Arm only has 600 downloads over the past year.

I don't think that's enough for standard library inclusion.  It may be enough for a blessed status, but certainly not going into the core.

Trust me, I think you're better off promoting the library before going into the standard.

- Josh

[Sciss]

i do use scala-swing, and i find it great that i can include it on a per project basis. (however, there are lot of aspects of scala-swing which i don't like, it could have been a much more radical and powerful library...)

i think the important message is to say, this is the way to do STM in scala. and i wouldn't mind having scala-xml and scala-actors as separate libraries, even scala-collection-mutable.

on the other hand i would disagree with martin's cause-effect hypothesis ("stdlib needs to be a consolidator for stuff that everybody uses") -- i think concurrency is an essential part of the language support, and it's a perfectly valid assumption that stm is as important as actors. so if i had to choose what is in stdlib, i would say collection and concurrency (actors+stm). maybe it would be worth to look at the average clojure project, since clojure has the stm built-in -- so how frequently is it actually used? how frequently are other concurrency tools used?

best, .h.h.

[Josh Suereth]

I also want to point out that one reason the continuations-plugin is not on by default in Scala 2.10 is lack of usage.   The download stats for that are worse than scala-arm.

Ironically,  scala-swing and scala-arm have *the same* download statistics.  So it makes a good argument that scala-arm has enough users to be included as a separate artifact...

Do you have your download statistics since moving to oss.sonatype.org?   That could make the argument more compelling if you have a bunch of silent, happy users.

- Josh

[Nathan Bronson]

One argument for putting the STM into the stdlib is so that other
parts of the stdlib can use it, if there is a need. There is certainly
not a need for 2.10; I don't have a good feel for the number of
potential future uses.

ScalaSTM would have more users if it had more promotion, but I come
from a long line of introverted non-salespeople. Sonatype doesn't seem
to have statistics for May yet (that I can find). There were 400
downloads in the fraction of April where it was hosted on sonatype,
which is about 1/8 as popular as the maven-scala-plugin.

- Nathan

[Josh Suereth]

And more popular than the continuations plugin...

[Jonas Bonér]

I agree Martin. I think this is a great idea. Like 'Haskell Platform' and 'OCaml Batteries Included'? 

[Jonas Bonér]

On Thu, May 24, 2012 at 11:18 PM, Josh Suereth <joshua....@gmail.com> wrote:

On Thu, May 24, 2012 at 4:59 PM, Daniel Sobral <dcso...@gmail.com> wrote:

On Thu, May 24, 2012 at 5:05 PM, Josh Suereth <joshua....@gmail.com> wrote:

> But you lose the ability to make changes.   I'm not so sure it's all
> unicorns and win going into the STD. Library....

In what sense you lose the ability to make changes? Can STM change now
at will? If it can, then no one is using it -- only experimenting with
it. There's much to be said about not being able to make changes
because there are people using it for production code.

In the sense that you don't have the expectation of binary compatibility now.

I hate to be a negative nancy (but I have to for my own library):   You need to improve users, I feel, before hitting the standard library.    When the "blessed" library bit is ready (i.e. the build utlity I'm working on is finished), then we can move you into there, but it's far too early for the standard library.

I think it'd be better to do some PR, evangelize and give more talks about STM.  Hype people up about it.  Don't rely on the standard library to be your PR.  You have to promote this project yourselves.  You have something to promote, it's stable.  You should be able to get users.   Perhaps you need to promote *other libraries* to use yours to improve users.   Either way, the users will stabilize the API, and you can get them today, with some effort.

This is so true. Spot on. 

 

The way to get more users is promotion.  Even if it hits the standard library, we still need a way to make people aware of it.  The standard library isn't a crutch to get users of a feature, and it should not be used that way.  It's the same for scala-arm and scala-io.   I'd love to push an IO library more than anything.  BUT, I don't think scala-io has solidified enough to warrant usage, and Scala-Arm only has 600 downloads over the past year.

I don't think that's enough for standard library inclusion.  It may be enough for a blessed status, but certainly not going into the core.

I totally agree.

 

Trust me, I think you're better off promoting the library before going into the standard.

- Josh

[Jonas Bonér]

On Thu, May 24, 2012 at 11:35 PM, Sciss <con...@sciss.de> wrote:

i do use scala-swing, and i find it great that i can include it on a per project basis. (however, there are lot of aspects of scala-swing which i don't like, it could have been a much more radical and powerful library...)

i think the important message is to say, this is the way to do STM in scala. and i wouldn't mind having scala-xml and scala-actors as separate libraries, even scala-collection-mutable.

scala-actors and akka-actors will be separate jars from Scala 2.10 and forward.

I agree that scala-xml as separate jar would be *so* nice. 

scala-collection-mutable have to stay in though.

[Aleksandar Prokopec]

On 5/24/12 11:35 PM, Sciss wrote:
> i do use scala-swing, and i find it great that i can include it on a per project basis. (however, there are lot of aspects of scala-swing which i don't like, it could have been a much more radical and powerful library...)
>
> i think the important message is to say, this is the way to do STM in scala. and i wouldn't mind having scala-xml and scala-actors as separate libraries, even scala-collection-mutable.

I think scala-collection-mutable might be pushing it too far. This is
the sort of stuff you use every day.

Cheers,
Alex

> on the other hand i would disagree with martin's cause-effect hypothesis ("stdlib needs to be a consolidator for stuff that everybody uses") -- i think concurrency is an essential part of the language support, and it's a perfectly valid assumption that stm is as important as actors. so if i had to choose what is in stdlib, i would say collection and concurrency (actors+stm). maybe it would be worth to look at the average clojure project, since clojure has the stm built-in -- so how frequently is it actually used? how frequently are other concurrency tools used?
>
> best, .h.h.
>

--
Aleksandar Prokopec,
Doctoral Assistant
LAMP, IC, EPFL
http://people.epfl.ch/aleksandar.prokopec

[Sciss]

On 25 May 2012, at 08:49, Aleksandar Prokopec wrote:

> On 5/24/12 11:35 PM, Sciss wrote:
>> i do use scala-swing, and i find it great that i can include it on a per project basis. (however, there are lot of aspects of scala-swing which i don't like, it could have been a much more radical and powerful library...)
>>
>> i think the important message is to say, this is the way to do STM in scala. and i wouldn't mind having scala-xml and scala-actors as separate libraries, even scala-collection-mutable.
>
> I think scala-collection-mutable might be pushing it too far. This is the sort of stuff you use every day.

it's true, and i guess it's used a lot under-the-hood, like in builders. i just meant to say, if there are modules that stand on their own feet and are virtually not required by other parts of the standard library, it would be good to isolate them, so cross dependencies are minimised. for stuff like proguard/android i guess this has huge benefits.

best, ..h.h..


>
> Cheers,
> Alex