Scala Meeting report, 2011-08-23

[Adriaan Moors]

We are currently publishing a summary of each of the weekly meetings of the Scala Core Team.

This information is made available as a service to the community. It is by necessity rather brief and gives only a rough approximation of the main points of discussions explored during each meeting; it should not be taken as a source of reliable information, nor as a record of concrete or firm decisions, nor as anything other than a record of a simple discussion.

The summary that follows is primarily intended for Scala contributors and maintainers. If you are not a contributor to the Scala system, the information below is unlikely to be very useful to you, and you might lack some of the necessary background to make sense of all the discussion items.

We do not have enough people on the team to be able to write a more complete record, and we might also not have the resources to discuss every point in detail afterwards. Nevertheless, we hope that this record, cursory as it is, is better than nothing.

Scala Meeting report, 2011-08-23

Attending

Toni: back from vacation
Martin: reflection, code lifting, reifying types and symbols,
  considering putting the whole typechecker in the reflection(.internal) framework
  so that trees can be transformed and re-typechecked (don’t want to do that manually)
Grzegorz: more manifests hacking
Adriaan: bringing virtualized scalac up to date with vanilla scala, tried xsbt build of scala
Lukas: effects plugin, refining polymorphic effects
Vlad: inlining exception handlers works
Miguel: discovered new program shapes that affect erasure, need explicit outer type params
Tiark: delite
Phil: kojo article going out (useful for presentations -- “scala is easy to get started with”), 
optimizing or tiobe ranking (are there other sites that rank PLs?), Scala Days 2012 to be held in London at the Barbican
Vojin: Bugfixes in distributed collections
Hubert: error trees instead of throwing TypeError in inferencer, but not in implicit search for now, 
merging current work with trunk (lots of conflicts with current reflect refactoring), 
working on build manager in eclipse to work around sbt's lack of support for multiple output directories
Josh: sbt build, patch for bug in scaladoc (review by ureche)

Discussion

2.9.1 RC4
fixes
  - suggested fixes from trunk by Ismael Juma (collections performance issues)
  - Continuations plugin fixes
  - VerifyErrors: are they fixed?
     - errors in compiler code, but not in the library
  - repl&sbt issue
release message should contain final-release-by date: Aug 31st


Scala community extension lib
exploring libs to include:
- scala-arm
- scala-io: in good shape, docs, artifacts on scala-tools, but not widely used yet, SBT: http://jesseeichar.github.com/scala-io/getting-started.html
- STM
  - releasable CCSTM-based codebase should be right around the corner
  - good quality code
- graph
- specialized numeric
  - fast, but not backwards compatible
  - it overlaps with existing code in stdlib
  - can we give people that don’t care about backwards compatiblity a way to drop this in and give it a whirl?
- logging
  - too many choices
    - pick one?
    - define an api?
- testing frameworks
  - scalacheck
  - scalatest

  - specs
- DB wrappers (which ones?)
- configuration
- anti-xml: check back when more stable

TODO:
- public document [Josh]
  - requirements to be fulfilled by a project
  - contents of the listings (e.g., our estimate of a module's stability)
- build system
  - paul has a typesafe jenkins instance ready for use
- where should the code for the build go?
  - put it up under scala on github
- stuff in trunk could be pulled out and live in the Scala Cell [Paul]
  - e.g., scaladoc tools?


Bugs

optimiser

in principle, can’t optimise array loads away, since that may mask RuntimeExceptions
have varying -Ox options where higher values of x allow changing behaviour when it comes to exceptions, for example?
"assume whole-program compilation" option: the optimiser may be able to do more (lex tried that back in the day)
signatures

name mangling cannot be fixed in isolation
Grzegorz only wants to generate better signatures (inner class tables), not change name mangling

[Francois]

We are currently publishing a summary of each of the weekly meetings of the Scala Core Team.

[...]

Thanks Scala Team for these really insightful informations!

-- 
Francois ARMAND
http://fanf42.blogspot.com
http://www.normation.com

[Francois]

[...]

Scala community extension lib
exploring libs to include:
- scala-arm
- scala-io: in good shape, docs, artifacts on scala-tools, but not widely used yet, SBT: http://jesseeichar.github.com/scala-io/getting-started.html
- STM
  - releasable CCSTM-based codebase should be right around the corner
  - good quality code
- graph
- specialized numeric
  - fast, but not backwards compatible
  - it overlaps with existing code in stdlib
  - can we give people that don’t care about backwards compatiblity a way to drop this in and give it a whirl?
- logging
  - too many choices
    - pick one?
    - define an api?
- testing frameworks
  - scalacheck
  - scalatest

  - specs
- DB wrappers (which ones?)
- configuration
- anti-xml: check back when more stable

What about:

- JSON serialization (lift-json is a good candidate here)
- scalaz (it seems to be a good candidate for the domain it take care off)


Thanks,

[Doug Tangren]

What about:

- JSON serialization (lift-json is a good candidate here)
- scalaz (it seems to be a good candidate for the domain it take care off)

 
https://github.com/codahale/jerkson - not `pure` scala but easier to use than lift-json
 

[Josh Suereth]

Scala has JSON serialization in its library (albeit not as efficient or friendly as lift-json and others).  

As for Scalaz, I personally would want to wait for the Scalaz7 API to stabilize.  I think the type-trait domain has some readjustments to make.

[Ismael Juma]

On Fri, Aug 26, 2011 at 4:29 PM, Josh Suereth <joshua....@gmail.com> wrote:
> Scala has JSON serialization in its library (albeit not as efficient or
> friendly as lift-json and others).

Is this library good enough to remain in the standard library by the
way? I've heard of quite a few problems.

Best,
Ismael

[Francois]

> Scala has JSON serialization in its library (albeit not as efficient
> or friendly as lift-json and others).

Oh, you right... I forgot about that one, sorry.

> As for Scalaz, I personally would want to wait for the Scalaz7 API to
> stabilize. I think the type-trait domain has some readjustments to make.

There is no urgency, just a question to know if it was considered. So
thanks for the reply !

[Josh Suereth]

I, personally, would certainly consider a new JSON library. As Ismael states, the one in there is known to be slow.

- Josh

[Daniel Spiewak]

- JSON serialization (lift-json is a good candidate here)

Given the proliferation of good-quality JSON libraries in the community, I'm not sure this is something that should be tackled by the standard library.  As for Lift JSON, it's…fast.  :-)  I have significant issues with the API though, so I'm quite against its whole-sale inclusion.  As I said though, there are a lot of good community libraries that handle this, so I don't think it's a priority for the stdlib.

- scalaz (it seems to be a good candidate for the domain it take care off)

It seems to me that scalaz doesn't tie itself to backwards compatibility.  The framework needs to continue to evolve and grow, and I don't know that it can do that within the constraints of the standard library.  So, I'm not a fan of this idea either.  :-)

Daniel

[Ismael Juma]

On Fri, Aug 26, 2011 at 4:52 PM, Daniel Spiewak <djsp...@gmail.com> wrote:
> Given the proliferation of good-quality JSON libraries in the community, I'm
> not sure this is something that should be tackled by the standard library.
> As for Lift JSON, it's…fast.  :-)  I have significant issues with the API
> though, so I'm quite against its whole-sale inclusion.  As I said though,
> there are a lot of good community libraries that handle this, so I don't
> think it's a priority for the stdlib.

This discussion is not about the standard library though, it's about
the "Scala community extension lib". Having said that, I would also
wait a bit (but not forever) before making a choice when it comes to
JSON as things are still evolving there.

Best,
Ismael

[Simon Ochsenreither]

Hi everyone!

Josh: sbt build, patch for bug in scaladoc (review by ureche)

Any news when people can expect the switch from Ant to SBT by default?

- scala-io: in good shape, docs, artifacts on scala-tools, but not widely used yet, SBT: http://jesseeichar.github.com/scala-io/getting-started.html

�- How platform-dependent is it? I'm pretty wary including anything that big with deep requirements of Java classes while constantly seeing how much trouble this causes for other ports of the language.
�- As far as I understand having some decent IO library in Scala requires at least Java 7. So what would be the compatibility story here? Optional Jar files shipped by default, which are allowed to require a more up-to-date Java version?

I'm not against having it, but I certainly lack the necessary information to form an opinion. Assuring that Scala IO isn't totally tied to the Java platform and having a clear overview over which dependencies have to be rewritten for other platforms would be great.

- STM
� - releasable CCSTM-based codebase should be right around the corner
� - good quality code

I heard there are actually multiple (compatible?) implementations around the corner, is that correct?

- specialized numeric
� - fast, but not backwards compatible
� - it overlaps with existing code in stdlib
� - can we give people that don�t care about backwards compatiblity a way to drop this in and give it a whirl?

Please yes. Integrate the improvements from d_m (http://www.azavea.com/blogs/labs/2011/06/scalas-numeric-type-class-pt-2/ ). This would probably take a lot of immediate pressure away from all the current performance problems which led to the thoughts about hard inline requirements in the latest SID.
I'm at the moment trying to SI-4658 and not having a division operation is a pita. Isn't it possible to fix it, even if it creates problems with backward compatibility? (Migration manager, maybe?)

- logging
� - too many choices
� � - pick one?
� � - define an api?

I'm not really sure if "logging" makes sense at all. There are so many different use cases that I'm not sure that something defined in the source code is general enough. People log performance with VisualVM and JVMTI, people use aspects to manage other logging requirements and much more.

- DB wrappers (which ones?)

What's the status of Slick, "a common framework for connecting with databases and distributed collections"? It was mentioned in some talks (e. g. Scala eXchange) but there seems to be no information available on the internet about it.

- anti-xml: check back when more stable

Any plans on shipping an optional scala-xml package like scala-swing meanwhile, which can be left out of deployment if not required? (I know it is actually needed for compilation, but e. g. on Android platforms cutting down the filesize could lift the lengthy requirement of running ProGuard even for development builds.

Thanks and bye,

Simon

[Matthew Pocock]

Thanks for sending these mails out. I really appreciate them.

On 26 August 2011 10:14, Adriaan Moors <adriaa...@epfl.ch> wrote:

We are currently publishing a summary of each of the weekly meetings of the Scala Core Team.

 

Bugs

optimiser

in principle, can’t optimise array loads away, since that may mask RuntimeExceptions

Array loads are safe to optimize away if the index is legal. You need to be able to compare the expression giving the array length and the expression for the index, and be able to prove that the index is always be in-range. We don't really care about the out-of-range case, so the procedure is really identifying things where we can prove the index is in range, without implying that failure means it is out of range. I have no idea how hard it would be to implement this, but we could probably start with the common cases of constants and counters with known min/max.

Matthew

--

Dr Matthew Pocock

Visitor, School of Computing Science, Newcastle University

mailto: turingate...@gmail.com

gchat: turingate...@gmail.com

msn: matthew...@yahoo.co.uk

irc.freenode.net: drdozer

tel: (0191) 2566550

mob: +447535664143

[Paul Phillips]

On 8/27/11 4:25 AM, Simon Ochsenreither wrote:
> - As far as I understand having some decent IO library in Scala
> requires at least Java 7.

I reiterate that this is ridiculous. Please stop saying it. There is
no truth to it.

[Simon Ochsenreither]

Hi,

>> - As far as I understand having some decent IO library in Scala
>> requires at least Java 7.
>
> I reiterate that this is ridiculous. Please stop saying it. There is
> no truth to it.

Well, then we have to disagree here.

Personally I view requirements like copying a file or traversing a
directory to be not so much ridiculous. Both is not possible in
versions before Java 7.

Bothering with `java.io.File` and related jokes is a pure waste of time.

Thanks and bye,

Simon

[Paul Phillips]

On 8/27/11 8:32 AM, Simon Ochsenreither wrote:
> Personally I view requirements like copying a file or traversing a
> directory to be not so much ridiculous. Both is not possible in versions
> before Java 7.

Somehow a significant percentage of the world's software was written
before last tuesday. Everyone wants their flying car and their ponies,
but to say that you can't write an I/O library without java7 is like
saying you couldn't write a todo list app before the ipad 2 came out.

And, what are you talking about? You can't traverse a directory or copy
a file? How has all the traversing and copying been going on then? Who
cares if there's a single *java* function to do those things? The point
of a library is that you have the opportunity to write your own
functions. That's what it's for.

[Simon Ochsenreither]

> And, what are you talking about? You can't traverse a directory or copy
> a file? How has all the traversing and copying been going on then? Who
> cares if there's a single *java* function to do those things?

I'm not speaking about single java function vs. building my own stuff,
I'm concerned about things just not being possible at all.

> The point of a library is that you have the opportunity to write
> your own functions. That's what it's for.

The problem is that you can't just write "your own functions" because
the VM has abstracted the file system away too much and it lacked the
intrinsics to support appropriate features before 7. It is not like
.NET where people can just drop down to C++/"unsafe"/call Win32 stuff
and play with C structs to get the data they need.

So please go ahead and tell me how to copy a file in your opinion
before Java 7.

The only thing you can do is creating a new, empty file and moving
bytes from the old one to the new one either with Readers/Writers or
with the Channel API. Both make you loose everything associated with
the original file like metadata and attributes.

How would you traverse a directory without blowing up the VM if some
unfortunate symlink points to a parent folder?

It's things like that which just don't work without shipping our own
binary .dll/.so or using a runtime which decides to support the
necessary intrinsics, but I'll eat my hat if you come up with a
solution which obviously hasn't been found by everyone else since Java
1.0. :-)

Thanks and bye,

Simon

[Paul Phillips]

On 8/27/11 10:15 AM, Simon Ochsenreither wrote:
> So please go ahead and tell me how to copy a file in your opinion before
> Java 7.

I'm sure you can find the function I wrote which copies files. I am
intimately aware of the imperfections of java I/O. Everything has
imperfections, including the underlying OSes, so it's not like you are
entering some promised land even if the java runtime is bug free. None
of this has anything to do with whether you can ship a decent I/O
library: you can.

For yourself, you can pick anything you want and say you can't ship a
decent XXX library without it. It is your privilege. However those of
us responsible for the distribution have to deal with the world as it
exists. (We "go to war with the army we have", not "the army we might
like to have", or "the army we might have at some future date.") The
"army we have" is still java5. Sooner or later it will be java6. It
will not be java7 for a long while yet.

[Michael Bayne]

On Sat, Aug 27, 2011 at 10:15 AM, Simon Ochsenreither
<si...@ochsenreither.de> wrote:
> How would you traverse a directory without blowing up the VM if some
> unfortunate symlink points to a parent folder?

I find File.getCanonicalPath to be useful in this circumstance.

-- m...@samskivert.com

[Josh Suereth]

On Sat, Aug 27, 2011 at 7:25 AM, Simon Ochsenreither <si...@ochsenreither.de> wrote:

Hi everyone!

Josh: sbt build, patch for bug in scaladoc (review by ureche)

Any news when people can expect the switch from Ant to SBT by default?

There's a list of TODOs here:  https://wiki.scala-lang.org/display/SIW/SBT+Build

Once everyone has a chance to try it out and the "must have" TODOs are done, I don't see any reason to keep Ant.   That's right, speak up with features/issues before it's too late, because once that list is gone, I'll be pushing to commit the SBT build into trunk.

 

- scala-io: in good shape, docs, artifacts on scala-tools, but not widely used yet, SBT: http://jesseeichar.github.com/scala-io/getting-started.html

 - How platform-dependent is it? I'm pretty wary including anything that big with deep requirements of Java classes while constantly seeing how much trouble this causes for other ports of the language.

Let's not start FUD here.   GO LOOK at scala-io, it's very well done.  The core abstractions are done such that a JVM implementation is easy, but you could override Reads/Writes traits for alternative platform.  I think Jesse did a great job here, so please go look at it.

 

 - As far as I understand having some decent IO library in Scala requires at least Java 7. So what would be the compatibility story here? Optional Jar files shipped by default, which are allowed to require a more up-to-date Java version?

I'm not against having it, but I certainly lack the necessary information to form an opinion. Assuring that Scala IO isn't totally tied to the Java platform and having a clear overview over which dependencies have to be rewritten for other platforms would be great.

Again, Scala-IO was designed so that a Java7 implementation can be used or a Java <7 implementation.  By trying to support Java 7 and Java <7 I think you gain a huge benefit in being cross-platform capable. 

- STM

  - releasable CCSTM-based codebase should be right around the corner

  - good quality code

I heard there are actually multiple (compatible?) implementations around the corner, is that correct?

- specialized numeric

  - fast, but not backwards compatible

  - it overlaps with existing code in stdlib

  - can we give people that don’t care about backwards compatiblity a way to drop this in and give it a whirl?

Please yes. Integrate the improvements from d_m (http://www.azavea.com/blogs/labs/2011/06/scalas-numeric-type-class-pt-2/ ). This would probably take a lot of immediate pressure away from all the current performance problems which led to the thoughts about hard inline requirements in the latest SID.
I'm at the moment trying to SI-4658 and not having a division operation is a pita. Isn't it possible to fix it, even if it creates problems with backward compatibility? (Migration manager, maybe?)

I think you under-estimate the scope of the problem the inline class SID is trying to solve.   Specialized Numeric actually factor only marginally into the inline SID.

- logging
  - too many choices
    - pick one?

    - define an api?

I'm not really sure if "logging" makes sense at all. There are so many different use cases that I'm not sure that something defined in the source code is general enough. People log performance with VisualVM and JVMTI, people use aspects to manage other logging requirements and much more.

I highly disagree.   We can agree to disagree here.  If we're going to choose testing frameworks or other things with myriads of choices, you can't choose which verticle to level this complaint against.   As for logging, I think we can do far better than SLF4j and Log4j in Scala.  In the meantime, maybe we can get a standard syntax (like CC-STM is doing) and then work on more powerful backends later.  Logging is something that, in every project I've been on, is of high importance but little effort is spent on it.

- DB wrappers (which ones?)

What's the status of Slick, "a common framework for connecting with databases and distributed collections"? It was mentioned in some talks (e. g. Scala eXchange) but there seems to be no information available on the internet about it.

I think choosing an external DB wrapper is on hold until Slick enters the fray.   Giving JDBC wrappers more LINQ-like abilities could drastically alter design and principles of these libraries.

- anti-xml: check back when more stable

Any plans on shipping an optional scala-xml package like scala-swing meanwhile, which can be left out of deployment if not required? (I know it is actually needed for compilation, but e. g. on Android platforms cutting down the filesize could lift the lengthy requirement of running ProGuard even for development builds.

I know a few of us have modularization efforts.  You'll have to keep waiting patiently while momentum builds, unless you want to get the ball rolling. 

Thanks and bye,

Simon

Hopefully I answered all the questions/concerns! 

[Erik Osheim]

On Sun, Aug 28, 2011 at 12:45:57PM -0400, Josh Suereth wrote:
> I think you under-estimate the scope of the problem the inline class SID is
> trying to solve. Specialized Numeric actually factor only marginally into
> the inline SID.

Agreed.

In fact, the specialized-numeric work I've done *can't* solve the
inline class problem. I've created a compiler plugin which does do
inlining for Numeric specifically *because* there isn't another option.
I certainly don't think the work I've done obviates the need for Josh's
SID. I (and other authors) will benefit immensely from inline classes
(as well as better type trait/type class support).

-- Erik

P.S. Simon, your division issue raises a different question: is there a
possibility that the interface of scala.math.Numeric can be changed?
Many (most?) would-be users of Numeric hit the same issue as you did,
but changing the interface (even just by adding division) has
compatibility implications.

[Grzegorz Kossakowski]

On 28 August 2011 18:45, Josh Suereth <joshua....@gmail.com> wrote:

On Sat, Aug 27, 2011 at 7:25 AM, Simon Ochsenreither <si...@ochsenreither.de> wrote:

Hi everyone!

Josh: sbt build, patch for bug in scaladoc (review by ureche)

Any news when people can expect the switch from Ant to SBT by default?

There's a list of TODOs here:  https://wiki.scala-lang.org/display/SIW/SBT+Build

Once everyone has a chance to try it out and the "must have" TODOs are done, I don't see any reason to keep Ant.   That's right, speak up with features/issues before it's too late, because once that list is gone, I'll be pushing to commit the SBT build into trunk.

I guess it's already on your radar but I thought it's worth mentioning: on my box 'test' command barely uses 20% of CPU as opposed to the old partest implementation that was utilizing around 90% of CPU.

--
Grzegorz Kossakowski

[Josh Suereth]

That is strange because I see from 40-80% CPU utilization on my box.  Any chance you could dig into that a little further?  It's hard to debug something you can't see, and I don't see any reason why SBTRunner would act differently, thread-wise, than the ant "Partest" task...

 

--
Grzegorz Kossakowski

[Grzegorz Kossakowski]

On 28 August 2011 20:51, Josh Suereth <joshua....@gmail.com> wrote:

That is strange because I see from 40-80% CPU utilization on my box.  Any chance you could dig into that a little further?  It's hard to debug something you can't see, and I don't see any reason why SBTRunner would act differently, thread-wise, than the ant "Partest" task...

I'm far from being an expert in this area but will try to attach Yourkit and maybe obtain thread dump. I guess I'll have time to dig into at the end of the week (I'm super busy with Scala+GWT stuff at the moment).

--
Grzegorz Kossakowski

[Matthew Pocock]

Hi,

On 28 August 2011 18:27, Erik Osheim <er...@plastic-idolatry.com> wrote:

 

I (and other authors) will benefit immensely from inline classes
(as well as better type trait/type class support).

Class in-lining, esp for @specialized type classes, would be really great.

 

Matthew

-- Erik