Yammer moving away from Scala
Jeff Nowakowski
Email critique of Scala: https://gist.github.com/1406238
A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
It's a very detailed email from an employee at Yammer on why they are
migrating from Scala back to Java. I know it was intended to be private,
but it's gone public now, and there's nothing personal in the email anyways.
Stefan Wagner
Hash: SHA1
Am 30.11.2011 19:24, schrieb Jeff Nowakowski:
> I'm surprised I haven't seen this mentioned by now:
>
> Email critique of Scala: https://gist.github.com/1406238
Well - did you notice, what's wrong with his first example (for vs. while)?
scala>
var start = System.currentTimeMillis();
var total = 0;for(i <- 0 until 100000) { total += i };
var end = System.currentTimeMillis();
println(end-start);
println(total);
114
scala>
scala<
var start = System.currentTimeMillis();
var total = 0;var i=0;while(i < 100000) { i=i+1;total += i };
var end = System.currentTimeMillis();
println(end-start);
println(total);
8
The second one is faster, but both are wrong! :) There is an int-overrun
happening, so it has to be:
var total = 0L;for(i <- 0 until 100000) { total += i };
and
var total = 0L;var i=0;while(i < 100000) { i=i+1;total += i };
to avoid that, but there is a far faster solution:
val n = 100000L
val result= n*n/2-n
But calculating the result isn't the aim of the example? Not? What is
it? In far more complex computations, the costs of looping often vanish.
- --
Tsch���--->...Stefan
- ---------------------------
Don't visit my homepage at:
http://home.arcor-online.net/hirnstrom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7WheQACgkQQeATqGpDnRrOpgCgk65bcGVgSMuIl+Gge++YJC+U
gb8An3vS3A2tpb7FZKSFDtw/LOzZJTGX
=VQrp
-----END PGP SIGNATURE-----
Rex Kerr
1. Don't ever use a for-loop. Creating a new object for the loop closure,passing it to the iterable, etc., ends up being a forest of invokevirtual calls,even for the simple case of iterating over an array.
Indeed. For loops are slow. After "create the pipe operator for yourself", this is probably the second-most-common question/answer pair on StackOverflow. JVMs cannot reliably optimize even specialized range code in for loops, especially if they're nested.
2. Don't ever use scala.collection.mutable. Replacing a scala.collection.mutable.HashMap with a java.util.HashMap in a wrapper produced an order-of-magnitude performance benefit for one of these loops.
This is also true in some cases. The Scala HashMap does an embarrassing amount of boxing of key-value pairs. For example, the += method takes a tuple (often created by the user to begin with), then accesses the two elements, packaging them into an Entry object, which it finally passes off to HashTable. For simple mappings this can give a ~10x slowdown. I wouldn't say to _never_ use scala.collection.mutable (e.g. ArrayBuffer is pretty close to java.util.ArrayList in performance), but there are some gotchas lurking in there, with HashMap being one of the worst offenders.
3. Don't ever use scala.collection.immutable. Replacing a scala.collection.immutable.HashMap with a java.util.concurrent.ConcurrentHashMap in a wrapper also produced a large performance benefit for a strictly read-only workload. Replacing a small Set with an array for lookups was another big win, performance-wise.
Here, I disagree. What is your use case, and how sure are you that you are _actually_ not modifying that java.util.concurrent.ConcurrentHashMap anywhere? You can often write less-formally-safe code that performs better, but that doesn't make it a good idea unless you identify this as a key bottleneck, at which point you fix the key bottleneck and take away the lesson that _you fix key bottlenecks_, not that you _always use less-formally-safe code_.
4. Always use private[this]. Doing so avoids turning simple field access into aninvokevirtual on generated getters and setters. Generally HotSpot would end upinlining these, but inside our inner serialization loop this made a hugedifference.
That's not nearly as big of an issue as always using final val in classes if it can in fact be final (even in places where it seems redundant, such as inside a class itself marked final). Anyway, using private[this] is a good practice for other reasons: it prevents classes from messing with each others' internals, which makes it harder to reason about the code.
5. Avoid closures [...] we stopped seeing lambdas as free andstarted seeing them as syntactic sugar on top of anonymous classes and thusacquired the same distaste for them as we did anonymous classes
This one I also disagree with. Understand the relative cost of closures compared to the code that they enclose. Use appropriately. Also, in deeply nested loops, create the functions by hand at the outer level; the compiler doesn't notice that it can move closure creation into the outside loop. For example, if you have an array of options of strings, using arr.foreach(_.foreach(x => ...)) takes about twice as long as arr.foreach(_.foreach(f)) where f is defined outside.
Anyway, these criticisms seem, for the most part, reasonably well-founded, except that the bottom line is that you miss out on many of Scala's best features in your critical inner loops. Unless your entire code base is critical inner loops, using Java is unlikely to make your life easier. (There is a lot more to remember when writing high-performance Scala than when writing high-performance Java, however.)
--Rex
Aleksey Nikiforov
Also, in my opinion, the critisims layed against SBT is entirely justified. I think the idea of having a Scala based build tool is great, and the incremental compilation is great, and the console is great. However the custom DSL is extremely hard to use, and the implementation source code is impossible to read. Additionally many features just do not work as expected, either by design or due to bugs.
√iktor Ҡlang
When I first dicovered that for loops
Just to be very clear, there are no "for loops", are you talking about for comprehensions or just "foreach"?
were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.
Also, in my opinion, the critisims layed against SBT is entirely justified. I think the idea of having a Scala based build tool is great, and the incremental compilation is great, and the console is great. However the custom DSL is extremely hard to use, and the implementation source code is impossible to read. Additionally many features just do not work as expected, either by design or due to bugs.
--
Viktor Klang
Akka Tech Lead
Aleksey Nikiforov
On Wed, Nov 30, 2011 at 9:27 PM, Aleksey Nikiforov <lex...@gmail.com> wrote:When I first dicovered that for loops
Just to be very clear, there are no "for loops", are you talking about for comprehensions or just "foreach"?
Maybe I'm too used to scala-user... didnt realize this was on debate. I am respectfully taking myself out of this dicussion.
martin odersky
When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.
√iktor Ҡlang
You are not a compiler, you are a human being who can infer from the context. So you know EXACTLY what I mean. Your question serves no useful purpose but adds the flames.2011/11/30 √iktor Ҡlang <viktor...@gmail.com>On Wed, Nov 30, 2011 at 9:27 PM, Aleksey Nikiforov <lex...@gmail.com> wrote:When I first dicovered that for loops
Just to be very clear, there are no "for loops", are you talking about for comprehensions or just "foreach"?
What are you talking about, I asked a perfectly sensible question.
Are you talking about optimizing foreach or are you talking about something more general when it comes to desugaring for comprehensions, you definitely don't need to get all ballistic for no reason.
Maybe I'm too used to scala-user... didnt realize this was on debate. I am respectfully taking myself out of this dicussion.
Jesper Nordenberg
> When I first dicovered that for loops were not optimized I was surpised.
> When I discovered a resistance to optimizing them I was shocked. The
> decision to do nothing about for loops baffles me to this day.
For comprehensions are just syntactic sugar. Optimizing just them is not
very useful, instead effort should be put into generic optimization of
HOF's (and other inlining).
/Jesper Nordenberg
Aleksey Nikiforov
Russ Paielli
I used to religiosly compile all my code with -optimize. However at some point I found that it was inlining excessively and resulted in slower bytecode. I have not tried -optimize for a long time. My understanding is that if -optimize worked so well, it would be enabled by default. If it's not the default, there are good reasons for it.
I assume that compiling with -optimize takes longer, and that is why it is not the default. If it takes no longer to compile (and it works reliably), then yes it might as well be the default.
--Russ P.
On Wed, Nov 30, 2011 at 3:35 PM, martin odersky <martin....@epfl.ch> wrote:On Wed, Nov 30, 2011 at 9:27 PM, Aleksey Nikiforov <lex...@gmail.com> wrote:When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this. I wonder whether Yammer ever tried compiling with -optimize.Cheers-- Martin
--
http://RussP.us
Grzegorz Kossakowski
>
> I assume that compiling with -optimize takes longer, and that is why it is
> not the default. If it takes no longer to compile (and it works reliably),
> then yes it might as well be the default.
It does take significantly longer to compile large code bases with
-optimize enabled.
--
Grzegorz Kossakowski
Erik Osheim
> > You are not a compiler, you are a human being who can infer from the
> > context. So you know EXACTLY what I mean. Your question serves no useful
> > purpose but adds the flames.
> >
>
> What are you talking about, I asked a perfectly sensible question.
> Are you talking about optimizing foreach or are you talking about something
> more general when it comes to desugaring for comprehensions, you definitely
> don't need to get all ballistic for no reason.
If you needed clarification because you were confused then Aleksey
wrongly took offense at your reply. If you were taking the opportunity
to snipe at him for using slightly inexact terminology then his
reaction would make sense.
Martin (and most of us I think) understood what he meant, and Martin
referred to a "standard for loop" in his reply. I don't think anyone
would reply to Martin Odersky pointing out the fact that Scala doesn't
have for loops and asking for clarification.
Online, it's easy to perceive hostility online when none exists. It's
also easy to be casually rude to people. Let's all try to lighten up a
bit.
-- Erik
Paul Butcher
The point that he makes that (to me, at least) resonates most strongly relates to the Scala community. I don't know how to fix it, but we really do need to fix it.
It must be possible to create a supportive and welcoming environment for people coming to the language. Right now, the Scala community is nothing like that.
We've been using Scala heavily over the last 18 months and are in the process of selecting which language we'll use for a major new development. There are two things that count strongly against Scala - the binary compatibility issue and the poisonous nature of community that surrounds the language.
Personally, I'm strongly in favour of Scala. I'd rather stick pins in my eyes than use Java and I don't see any credible choice other than Scala. But my case is severely weakened by by the squabbles, flame wars and just plain rudeness that constantly breaks out here.
How do we (can we) address this?
--
paul.butcher->msgCount++
Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: pa...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher
Aleksey Nikiforov
It does not matter to me if it is a syntactic sugar or if they are comprehensions. What matters to me is to be able to write the code:
var sum = 0; for (i <- 0 until array.length) sum += array(i)
And have it work as fast as possible. Scala fails that expectation. It fails it so much that this is one of the most frequent questions on stack overflow. Also, it appears to be the number 1 on the list of problems with Scala as viewed by the employee at Yammer who wrote the email:
1. Don't ever use a for-loop.
It is nice to have for-comprehensions optimized. But I really do not care about for-comprehensions. I want something that looks like for loops and works like for loops. 99% of my code uses emulated for-loops via while-loops. It's painful and error prone, so I take this issue very personally.
Not providing proper for-LOOPs is akin to saying: Scala is not meant for the applications that require for-loops. Which sends the message that Scala is not meant for performance-sensitive applications. So guess what, the folks at Yammer got the message and went back to Java.
You can tell me that they are comprehensions, or a more general case should be optimized. But I do not care about those uses at all. Ranges and closures can never be optimized to the level of while loops. And I need the LOOPs. And so do many other people not interested in functional aspects of Scala.
√iktor Ҡlang
On Wed, Nov 30, 2011 at 10:37:02PM +0100, √iktor Ҡlang wrote:If you needed clarification because you were confused then Aleksey
> > You are not a compiler, you are a human being who can infer from the
> > context. So you know EXACTLY what I mean. Your question serves no useful
> > purpose but adds the flames.
> >
>
> What are you talking about, I asked a perfectly sensible question.
> Are you talking about optimizing foreach or are you talking about something
> more general when it comes to desugaring for comprehensions, you definitely
> don't need to get all ballistic for no reason.
wrongly took offense at your reply. If you were taking the opportunity
to snipe at him for using slightly inexact terminology then his
reaction would make sense.
No, I'm still unsure if we're talking about optimizing foreach or there are some special desugaring-optimizations of for comprehensions that are being discussed.
Martin (and most of us I think) understood what he meant, and Martin
referred to a "standard for loop" in his reply. I don't think anyone
would reply to Martin Odersky pointing out the fact that Scala doesn't
have for loops and asking for clarification.
I've been wasting too much time in discussions where people _thought_ they had the same definition of a word and then after a lot of gripe suddenly they found out that they were talking about different things, so I tend to want clarification in case I'm unsure. I see that as responsible, adult, behavior.
Online, it's easy to perceive hostility online when none exists. It's
also easy to be casually rude to people. Let's all try to lighten up a
bit.
Exactly, there was no grounds for which to become upset.
-- Erik
Josh Suereth
This is exactly the thinking that baffles me. There are for-loops in Java. Not comprehensions, LOOPs. Anyone coming from Java will expect for LOOPs or simlar functionality.On Wed, Nov 30, 2011 at 3:43 PM, Jesper Nordenberg <mega...@yahoo.com> wrote:
Aleksey Nikiforov skrev 2011-11-30 21:27:For comprehensions are just syntactic sugar. Optimizing just them is not very useful, instead effort should be put into generic optimization of HOF's (and other inlining).
When I first dicovered that for loops were not optimized I was surpised.
When I discovered a resistance to optimizing them I was shocked. The
decision to do nothing about for loops baffles me to this day.
/Jesper Nordenberg
It does not matter to me if it is a syntactic sugar or if they are comprehensions. What matters to me is to be able to write the code:
var sum = 0; for (i <- 0 until array.length) sum += array(i)
And have it work as fast as possible. Scala fails that expectation. It fails it so much that this is one of the most frequent questions on stack overflow. Also, it appears to be the number 1 on the list of problems with Scala as viewed by the employee at Yammer who wrote the email:So, they were looking for loops as well, not comprehensions. Why not send them an email and ask them if perhaps they were "talking about for comprehensions or just foreach"?! If you think I was overreacting, I would LOVE to see their response.
1. Don't ever use a for-loop.
It is nice to have for-comprehensions optimized. But I really do not care about for-comprehensions. I want something that looks like for loops and works like for loops. 99% of my code uses emulated for-loops via while-loops. It's painful and error prone, so I take this issue very personally.
Aleksey Nikiforov
I'm pretty surprised by this. I don't think while loops *always* beat for-expression for speed in all cases...
Jesper Nordenberg
> If you are not iterating over collections, then while-loops always beat
> the for-comprehensions. Also I have custom collections, which works best
> when you use index access isntead of iterators due to other JVM
> optimizations.
Hmmm, what benefits do you see in using Scala over Java if you only use
plain old imperative loops?
/Jesper Nordenberg
Russ Paielli
The point that he makes that (to me, at least) resonates most strongly relates to the Scala community. I don't know how to fix it, but we really do need to fix it.On 30 Nov 2011, at 18:24, Jeff Nowakowski wrote:
> I'm surprised I haven't seen this mentioned by now:
>
> Email critique of Scala: https://gist.github.com/1406238
>
> A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
>
> It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.
It must be possible to create a supportive and welcoming environment for people coming to the language. Right now, the Scala community is nothing like that.
We've been using Scala heavily over the last 18 months and are in the process of selecting which language we'll use for a major new development. There are two things that count strongly against Scala - the binary compatibility issue and the poisonous nature of community that surrounds the language.
I find the part about the "poisonous nature of community" hard to believe. First, I didn't realize that it is any more "poisonous" than any other online community. And even if it were, I can't imagine that would be much of a factor in selecting a language for a business. Then again, I'm lucky enough to have the freedom to select Scala without any approval from anyone else, and I'm the only one in my immediate organization who currently uses it, so maybe I just can't see the problem you are referring to.
Personally, I'm strongly in favour of Scala. I'd rather stick pins in my eyes than use Java and I don't see any credible choice other than Scala. But my case is severely weakened by by the squabbles, flame wars and just plain rudeness that constantly breaks out here.
In defense of Victor, he did preface his question with "Just to be very clear, ...". I think the hostile reply was an overreaction, but that sort of thing is common in online discussions.
--Russ P.
http://RussP.us
Aleksey Nikiforov
Hmmm, what benefits do you see in using Scala over Java if you only use plain old imperative loops?
/Jesper Nordenberg
Raoul Duke
>> eyes than use Java and I don't see any credible choice other than Scala. But
>> my case is severely weakened by by the squabbles, flame wars and just plain
>> rudeness that constantly breaks out here.
i wonder to what degree Scala is powerful enough to let various
disparate communities think they can use it, and then they come to
loggerheads on the lists? Like, there's the "i wish i were doing
haskell" crowd that will poo poo you if you don't already know the
typeclassopedia by heart; there's the performance nuts; there's the
people who really just kinda want a slightly less painful java but not
much more than that; there's yadda yadda yadda.
with java nobody is under any mistaken impression that it is or can be
anything other than plain old java. if you add on something like guava
or apache functional stuff you know you are out in la la land and your
noize only shows up on those mailing lists for only like-minded
suckers to read.
fun.
√iktor Ҡlang
This is exactly the thinking that baffles me. There are for-loops in Java. Not comprehensions, LOOPs. Anyone coming from Java will expect for LOOPs or simlar functionality.On Wed, Nov 30, 2011 at 3:43 PM, Jesper Nordenberg <mega...@yahoo.com> wrote:
Aleksey Nikiforov skrev 2011-11-30 21:27:For comprehensions are just syntactic sugar. Optimizing just them is not very useful, instead effort should be put into generic optimization of HOF's (and other inlining).
When I first dicovered that for loops were not optimized I was surpised.
When I discovered a resistance to optimizing them I was shocked. The
decision to do nothing about for loops baffles me to this day.
/Jesper Nordenberg
It does not matter to me if it is a syntactic sugar or if they are comprehensions. What matters to me is to be able to write the code:
var sum = 0; for (i <- 0 until array.length) sum += array(i)
And have it work as fast as possible. Scala fails that expectation. It fails it so much that this is one of the most frequent questions on stack overflow. Also, it appears to be the number 1 on the list of problems with Scala as viewed by the employee at Yammer who wrote the email:So, they were looking for loops as well, not comprehensions. Why not send them an email and ask them if perhaps they were "talking about for comprehensions or just foreach"?! If you think I was overreacting, I would LOVE to see their response.
1. Don't ever use a for-loop.
So you allow yourself to overreact because someone else might overreact even more?
I'm trying to understand the topic at hand so there can be a meaningful discussion, this is scala-debate after all.
It is nice to have for-comprehensions optimized. But I really do not care about for-comprehensions. I want something that looks like for loops and works like for loops. 99% of my code uses emulated for-loops via while-loops. It's painful and error prone, so I take this issue very personally.
Something like that should be a prime use case for a macro, to expand it into a while construct, I mean.
Not providing proper for-LOOPs is akin to saying: Scala is not meant for the applications that require for-loops.
So your only problem with foreach is that it does not have the raw performance that you need? Otherwise the for-comprehension sugar is just fine? So "proper" in this sentence means "high performance".
The question is how deep the rabbit hole goes, what if you add a guard to a for comprehension, should the optmizations be applied anyway, because then we're just not simply talking about "for loops" anymore, and people might wonder why the "for loop" users have so much better performance than the "for loop with guard" users.
This is why I suspected that the discussion was about foreach, but it would really be about optimizing the desugaring of for comprehensions, which is why I needed to ask what we actually were talking about.
Which sends the message that Scala is not meant for performance-sensitive applications. So guess what, the folks at Yammer got the message and went back to Java.
When writing high performance code you _always_ have to use the right tool for the job, if that means dropping into a little bit of Java, or into sun.misc.Unsafe, into cache-line padding classes, write your own custom data structures or doing object pooling. But that Scala would be unsuited for high performance code just not true, I think 45 million asynchronous messages processed per second on a 6 core box is pretty decent performance.
You can tell me that they are comprehensions, or a more general case should be optimized. But I do not care about those uses at all. Ranges and closures can never be optimized to the level of while loops. And I need the LOOPs. And so do many other people not interested in functional aspects of Scala.
So essentially you'd just want Javas good old for(... ; ....; ...) ?
As I said earlier, if nothing else, it could most probably be implemented using macro expansion in the future. So it is just a special sugar for a while construct.
Cheers,
√
Raoul Duke
> As I said earlier, if nothing else, it could most probably be implemented
> using macro expansion in the future. So it is just a special sugar for a
> while construct.
feature request:
#PRAGMA JAVA
...
#PRAGMA ENDJAVA
:-)
Erik Osheim
> Hmmm, what benefits do you see in using Scala over Java if you only
> use plain old imperative loops?
Tons, of course. :)
At my job we do much of our work on Arrays (gasp) using while loops
(gasp) and yet we benefit from many awesome features in Scala:
* cleaner-looking code
* traits
* unified primitive/boxed types
* much better generics
* type classes via implicits
* pattern matching/extractors
These are only a subset of course.
I understand why a lot of us are defensive about this issue (no one
likes to be criticized, or have their favorite language criticized).
But there are legitimate reasons to want fast imperative-style loops
besides while loop, and legitimate Scala users who need or would like
to see this kind of use optimized.
-- Erik
√iktor Ҡlang
That I not true, I got flamed on SO for using obscure language features like instance initializers and qualified "this" when some guy wanted the golfiest snippet to traverse a list + this in pure Java:
for (X x : new ArrayList<X>(xs){{ add(EnclosingClassX.this); }}) {
// Profit!
}
So apparently you can be one of the weird guys in the Java world as well.
fun.
Jesper Nordenberg
> There is nothing wrong with imperative programming.
Actually there is, but let's not get into that discussion. Personally I,
and I think many others, find it much more important that code written
in a functional style is optimized to get the same performance as
corresponding imperative code.
/Jesper Nordenberg
Aleksey Nikiforov
Raoul Duke
> On Wed, Nov 30, 2011 at 5:00 PM, Jesper Nordenberg <mega...@yahoo.com>
>> Actually there is, but let's not get into that discussion. Personally I,
see what i mean?
scala might have cast the net too wide...
Josh Suereth
martin odersky
Josh Suereth
√iktor Ҡlang
Totally agree,
real world beats benchmarks any day of the week.
-- Martin
Jesper Nordenberg
> That's a nice dream. But I challenge you to find a language
> implementation where this holds in the real world, not just in some
> select benchmarks.
Not sure what you're getting at. With proper inlining, unboxed types,
type inference etc. you can match imperative performance for functional
code. This can quite easily be done in C++11 for example (GHC also has
language extensions which allow these types of optimizations, but I
don't have much experience using those).
/Jesper Nordenberg
martin odersky
Bradley Buchsbaum
Bradley R. Buchsbaum
Rotman Research Institute
3560 Bathurst St.
Toronto, ON Canada M6A 2E1
email: bbuch...@rotman-baycrest.on.ca
Rex Kerr
On Wed, Nov 30, 2011 at 9:27 PM, Aleksey Nikiforov <lex...@gmail.com> wrote:When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.
That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this.
By "very close" I assume you mean "works in many cases as long as the number of items is large"? I've never gotten good performance out of a for-comprehension on, for instance, short arrays or ranges. (Well, except for the ScalaCL plugin.) I think it's lingering object creation overhead...haven't chased it down.
Also, one can forgive people for not compiling with -optimize, since (1) it is slower, and (2) for years it did nothing beneficial for performance.
Anyway, let's get macro capability (or compile {} blocks) implemented and then everyone who wants to can add a jfor that works just like the Java for (or the collections library can be compilable).
--Rex
Stefan Wagner
Hash: SHA1
Am 30.11.2011 23:24, schrieb Aleksey Nikiforov:
> If you are not iterating over collections, then while-loops always beat the
> for-comprehensions. Also I have custom collections, which works best when
> you use index access isntead of iterators due to other JVM optimizations.
Not for me:
scala ForWhileBench 400000000
0 while for
1 0.574 0.147
2 1.199 0.259
3 1.309 0.44
4 1.727 0.55
5 2.151 0.671
6 2.585 0.844
7 3.025 1.004
8 3.467 1.237
9 3.963 1.379
10 4.888 1.758
The code is only marginally modified from the yammer-code, except:
a) Linebreaks
b) using Longs, to avoid Int-overrun
c) returning the result.
type I=Int
type O=Long
def fordef (max: I): O = {
var total = 0L
for (i <- 0 until max) {
total += i }
total
}
def whiledef (max: I): O = {
var total = 0L
var i = 0
while (i < max) {
i += 1
total += i
}
total
}
Ah yes - and I compiled it.
- --
Tsch���--->...Stefan
- ---------------------------
Don't visit my homepage at:
http://home.arcor-online.net/hirnstrom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7W03YACgkQQeATqGpDnRp1/wCfZRcKN/0XD0YDK20OGnBa3RoO
hb4An3I1mY5JdOhEQ4t8+CPspqD7JHJd
=hZsK
-----END PGP SIGNATURE-----
ARKBAN
keep potential Scala converts. (Conversely failing to optimize them
might alienate potential converts.) When we first started with Scala we
all likely used various Java idioms such as for loops because they were
familiar and we could only embrace so much Scala at a time. And yet
that's as afar as many people will get before making up their minds
about adopting Scala. It seems so obvious to use because we've already
fallen down the rabbit hole.
ARKBAN
Josh Suereth
ejc
> I think you should check out (if it's available) susseman's (sp?) talk at
> Strange Loop 2011.
>
A very good talk indeed!
http://www.infoq.com/presentations/We-Really-Dont-Know-How-To-Compute
Thanks,
Eric
Aleksey Nikiforov
- --
Tschööö--->...Stefan
- ---------------------------
Don't visit my homepage at:
http://home.arcor-online.net/hirnstrom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
Aleksey Nikiforov
for: 554.
while: 270.
for: 481.
while: 261.
for: 481.
while: 260.
for: 481.
while: 261.
for: 481.
while: 260.
Here is the code:
object Test {
def main(args: Array[String]) {
test()
test()
test()
test()
test()
}
def test() {
val max = 400000000
var start = 0L
start = System.currentTimeMillis
fordef(max)
val timeFor = System.currentTimeMillis - start
start = System.currentTimeMillis
whiledef(max)
val timeWhile = System.currentTimeMillis - start
println("for: " + timeFor + ".")
println("while: " + timeWhile + ".")
}
def fordef(max: Int) {
var total = 0L
for (i <- 0 until max) {
total += i
}
}
def whiledef(max: Int) {
var total = 0L
var i = 0; while (i < max) {
i += 1
}
println(total)
}
}
Aleksey Nikiforov
while (i < max) {
i += 1
total += i
}
while (i < max) {
total += i
i += 1
}
for: 556.
while: 304.
for: 481.
while: 300.
for: 481.
while: 301.
for: 481.
while: 300.
for: 480.
while: 301.
A whole 40ms or about 15% slower. Still, the while loop significantly outperforms the for counterpart.
Josh Suereth
Paul Brauner
I don't know if that's what's planned but I guess that once the effect
system is there and once "until" is annotated as pure, some
deforestation or one of its variants could be applied there.
Paul
On Wed, Nov 30, 2011 at 22:35, martin odersky <martin....@epfl.ch> wrote:
>
>
> On Wed, Nov 30, 2011 at 9:27 PM, Aleksey Nikiforov <lex...@gmail.com> wrote:
>>
>> When I first dicovered that for loops were not optimized I was surpised.
>> When I discovered a resistance to optimizing them I was shocked. The
>> decision to do nothing about for loops baffles me to this day.
>>
> That's simply not true. The position is that we should not do isolated
> ad-hoc fixes but go to the root of things. Scala's overall optimizations
> need to be good enough that they can optimize standard for-loops. And they
> are already very close of doing this. I wonder whether Yammer ever tried
> compiling with -optimize.
>
> Cheers
>
> -- Martin
>
Bernd Johannes
> On 30 Nov 2011, at 18:24, Jeff Nowakowski wrote:
> > I'm surprised I haven't seen this mentioned by now:
> >
> > Email critique of Scala: https://gist.github.com/1406238
> >
> > A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
> >
> > It's a very detailed email from an employee at Yammer on why they are
> > migrating from Scala back to Java. I know it was intended to be private,
> > but it's gone public now, and there's nothing personal in the email
> > anyways.
First: I think it's to be expected that not every adoption will succeed. One
size doesn't fit all. Scala has it's issues (as a language, as an ecosystem,
as a community) but so does <fill in another language of choice>.
Furthermore it's a little unfortunate that a personal communication leaked
into the public. But now it's on the table for all to behold. And some of the
criticism is not really new (at least to me). Many issues are already being
adressed by typesafe so there is a chance for improvement.
My take home message: many (maybe even small) burdens and inconveniences have
summed up to a point where scala was not perceived as being worth the cost.
Fair enough.
> The point that he makes that (to me, at least) resonates most strongly
> relates to the Scala community. I don't know how to fix it, but we really
> do need to fix it.
I don't think we can fix it on this medium. It's the internet, it's all about
personality. It's uncontrolled.
And yes - some of the flames are used as evidence by people who don't like
scala to show that the community is hostile. Nothing can change that (unless
there are no longer such posts - but this won't happen. It's the internet).
So what can be done?
Officially: Documentation, education, official statements from the scala
maintainer about all major topics. Provide official references and resources.
But all this has already been done or is on its way.
So I think it's ok - scala is on its way.
Community:
val wishListForCommunity = List("pragmatism", "pragmatism", "pragmatism",
"patience", "peace")
Sometimes people don't want to be educated - not even remotely. They just want
a quick copy & modify & paste solution for an obstacle to get pressing
business done. Not providing a solution but trying to convince them that their
solution outline is crap firsthand or trying to baffle them with links to
fundamental articles of some sort won't scala do a favour. If I don't find a
switch to turn an engine on I don't want to read about electricity...
Providing a solution together with a hint that there is room for improvement
will often be received better.
Greetings
Bernd
Jesper Nordenberg
> I'd really like to see case studies of large systems where this is the
> case. I am surprised to see you mention C++ because without GC you
> cannot even start implementing true functional programming.
What is true functional programming, and why would you need a GC to
implement it? C++11 supports lambda expressions with zero runtime
overhead, local type inference, type parameters, unboxed immutable data
types, scoped objects and ref counting (for automatic and exact
deallocation). I honestly don't see how it's less of a FPL than Scala is.
/Jesper Nordenberg
Russ Paielli
--Russ P.
http://RussP.us
martin odersky
/Jesper Nordenberg
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967
Jesper Nordenberg
> As you know, a key aspect of FP is immutability. How do you effectively
> "change" an immutable object? You discard it and create a new, modified
> version of it (e.g., the case class copy method). If you had to manually
> clean up all the discarded objects, wouldn't you waste half your time
> doing that and still be left will memory leaks all over the place? Or
> am I missing something?
In C++ you basically have three options:
- Use value types, which means there won't be any garbage produced. This
works well for small, non-recursive types (tuples, points, options,
records etc.)
- Use reference counting. Works well for non-cyclic data (which includes
basically all functional data types), but there is some runtime overhead
especially for objects which can be passed between threads
- Use a conservative GC like Boehm
/Jesper Nordenberg
martin odersky
Out of curiosity: I wonder how common that is? Are there well known large C++ systems that use conservative GC?
Cheers
-- Martin
Paul Brauner
Tiark Rompf
> e, I think 45 million asynchronous messages processed per second on a 6 core box is pretty decent performance.
Off-topic, but I'd love to know more about that benchmark. Can you share some details? 45M messages with complex synchronization are pretty impressive whereas 45M no-ops are not.
Cheers,
- Tiark
Stefan Wagner
Hash: SHA1
Am 01.12.2011 05:53, schrieb Aleksey Nikiforov:
> Clearly you are doing something wrong.
You're right, I'am sorry. My code labelled the cases in the wrong way.
For 200 million to 2 billion iterations, I now get:
0 while for rekdef
1 0.681 2.629 0.765
2 1.31 6.221 1.499
3 1.979 7.652 2.413
4 2.649 9.109 3.216
5 3.308 12.299 4.01
6 3.947 13.593 5.209
7 4.845 16.433 5.486
8 6.255 18.343 5.959
9 5.947 20.686 6.697
10 6.607 24.32 7.429
Rekdef is a recursive implementation:
@tailrec
def rekdef (max: I, sofar: O): O =
if (max == 0) sofar
else rekdef (max-1, sofar+max)
def rekdef (max: I): O = rekdef (max, 0L)
- --
Tsch���--->...Stefan
- ---------------------------
Don't visit my homepage at:
http://home.arcor-online.net/hirnstrom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7XVPcACgkQQeATqGpDnRon0gCfePfxq+MaZHBJOCpWj0dsLfNP
GPAAoIPGU+xrws9TRmDCgRrckmZXCaX9
=i54A
-----END PGP SIGNATURE-----
nicola...@gmail.com
scala site where:
- It is explained what you can expect from -optimize, on a server JVM.
- Advices are given to improve performance in bottlenecks.
That would help in situation like those described in the mail that
started this discussion.
Best regards,
Nicolas.
√iktor Ҡlang
On Nov 30, 2011, at 11:48 PM, √iktor Ҡlang wrote:Off-topic, but I'd love to know more about that benchmark. Can you share some details? 45M messages with complex synchronization are pretty impressive whereas 45M no-ops are not.
> e, I think 45 million asynchronous messages processed per second on a 6 core box is pretty decent performance.
Complex synchronization? The only monitor lock ever taken in the bench is in the BlockingQueue of the ThreadPoolExecutor, all else is just volatiles/CAS
Of course the benchmark isn't doing any work besides sending and receiving messages, otherwise it would benchmark something else than sending and receiving messages.
https://github.com/jboner/akka/tree/master/akka-actor-tests/src/test/scala/akka/performance/microbench
Cheers,
√
Cheers,
- Tiark
Paul Butcher
> I find the part about the "poisonous nature of community" hard to believe. First, I didn't realize that it is any more "poisonous" than any other online community.
Sadly, it really is. You're absolutely correct that misunderstandings and flame wars are endemic on the internet, but for some reason the Scala community is much more prone to them than any other I've ever participated in. Worse, we have a habit of unloading on newbies so that their first experience of trying to participate is a negative one. You don't have to look far to find many examples of exactly this.
I spent quite a lot of time using Ruby (from around 2006 to 2010). One of the things that really helped Ruby adoption was the fact that the community was welcoming and supportive. Scala would benefit hugely if we could achieve the same.
(and yes, I know that if you go search the archives, you will find examples of flame wars in the Ruby community - the infamous Zed Shaw falling out of love with Ruby being a classic example. But that doesn't change the fact that the overall impression of the community was positive, in stark contrast with Scala at the moment).
> And even if it were, I can't imagine that would be much of a factor in selecting a language for a business.
You have at least two examples of people telling you that it's a problem. Me, and the Yammer post that sparked this discussion off. And there are plenty of others.
Community is an *incredibly* important part of what makes a language successful.
> In defense of Victor, he did preface his question with "Just to be very clear, ...". I think the hostile reply was an overreaction, but that sort of thing is common in online discussions.
My post was not intended to be aimed at Victor at all (in fact, I hit "send" before his message arrived in my inbox). One of the unfortunate side-effects of the bad atmosphere that surrounds this list is that people get "jumpy" and start taking offence unnecessarily. It becomes a self-perpetuating negative cycle. We all need to participate in breaking that cycle.
--
paul.butcher->msgCount++
Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?
http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: pa...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher
Tony Morris
> people get "jumpy" and start taking offence unnecessarily. It becomes a self-perpetuating negative cycle. We all need to participate in breaking that cycle.
Of all the commentary I have seen lately, this sentence alone has the
most value.
--
Tony Morris
http://tmorris.net/
Paul Butcher
> Of all the commentary I have seen lately, this sentence alone has the
> most value.
Thank you Tony.
For the avoidance of doubt - part of what needs to happen is for those of us who are prone to taking offence unnecessarily to give the benefit of the doubt to the person who we think might be being insulting. But part of what we need to do is be careful not to say things that could be interpreted as insulting.
I believe, from your recent comment in another thread, that you feel that avoiding language which might be interpreted in this way is "useless and superficial pandering". I understand why you feel that way, but that lack of pandering on the part of some members of this list is a large part of what's caused the current problem.
Tony Morris
> On 1 Dec 2011, at 10:45, Tony Morris wrote:
>> Of all the commentary I have seen lately, this sentence alone has the
>> most value.
>
> Thank you Tony.
>
> For the avoidance of doubt - part of what needs to happen is for those of us who are prone to taking offence unnecessarily to give the benefit of the doubt to the person who we think might be being insulting. But part of what we need to do is be careful not to say things that could be interpreted as insulting.
>
> I believe, from your recent comment in another thread, that you feel that avoiding language which might be interpreted in this way is "useless and superficial pandering". I understand why you feel that way, but that lack of pandering on the part of some members of this list is a large part of what's caused the current problem.
>
existed before the opportunity to unnecessarily pander even existed.
More to the (practical) point, I also reject any possible suggestion
that these problems are difficult to overcome -- it's easy.
Tiark Rompf
2011/12/1 Tiark Rompf <tiark...@epfl.ch>On Nov 30, 2011, at 11:48 PM, √iktor Ҡlang wrote:Off-topic, but I'd love to know more about that benchmark. Can you share some details? 45M messages with complex synchronization are pretty impressive whereas 45M no-ops are not.
> e, I think 45 million asynchronous messages processed per second on a 6 core box is pretty decent performance.
Complex synchronization? The only monitor lock ever taken in the bench is in the BlockingQueue of the ThreadPoolExecutor, all else is just volatiles/CAS
Of course the benchmark isn't doing any work besides sending and receiving messages, otherwise it would benchmark something else than sending and receiving messages.
https://github.com/jboner/akka/tree/master/akka-actor-tests/src/test/scala/akka/performance/microbench
Matthew Pocock
My understanding is that if -optimize worked so well, it would be enabled by default. If it's not the default, there are good reasons for it.
On Wed, Nov 30, 2011 at 3:35 PM, martin odersky <martin....@epfl.ch> wrote:On Wed, Nov 30, 2011 at 9:27 PM, Aleksey Nikiforov <lex...@gmail.com> wrote:When I first dicovered that for loops were not optimized I was surpised. When I discovered a resistance to optimizing them I was shocked. The decision to do nothing about for loops baffles me to this day.That's simply not true. The position is that we should not do isolated ad-hoc fixes but go to the root of things. Scala's overall optimizations need to be good enough that they can optimize standard for-loops. And they are already very close of doing this. I wonder whether Yammer ever tried compiling with -optimize.Cheers-- Martin
Jesper Nordenberg
> The equivalent of
>
> { x =>
> val arr = Array.ofDim(10)
> y => arr(y) += 1; arr(y)
> }
>
> Here you have a true closure, i.e. a lambda abstraction that closes over
> its environment. Scoped objects don't help you here. Ref-counting could
> help if it's done everywhere. But ref-counting tends to have terrible
> performance in a multi-threaded environment. Also ref-counting does not
> pick up cyclic structures.
Interesting that you picked an example containing mutation and arrays when
we are discussing functional programming. In these cases it's certainly
useful to have a GC (for example ref counting), but it doesn't have anything
to do with if the code is functional or imperative.
Regarding cycles, lots of functional data types don't contain cycles (basically
all Scala immutable collection types for example). And there are lots of
other functional data types that can be value types.
Regarding multi threaded performance of ref counting, you can restrict
object sharing between threads and for objects that is shared you can mitigate
the performance hit by keeping thread specific ref counts.
Another solution is to use a conservative GC like Boehm with C++.
> You really need GC to to functional programming.
I don't agree (even if you consider ref counting as a form of GC). You
can certainly do some forms of functional programming without a GC, but
for other forms it's certainly an advantage to have one.
/Jesper Nordenberg
√iktor Ҡlang
On Dec 1, 2011, at 11:34 AM, √iktor Ҡlang wrote:2011/12/1 Tiark Rompf <tiark...@epfl.ch>
On Nov 30, 2011, at 11:48 PM, √iktor Ҡlang wrote:Off-topic, but I'd love to know more about that benchmark. Can you share some details? 45M messages with complex synchronization are pretty impressive whereas 45M no-ops are not.
> e, I think 45 million asynchronous messages processed per second on a 6 core box is pretty decent performance.
Complex synchronization? The only monitor lock ever taken in the bench is in the BlockingQueue of the ThreadPoolExecutor, all else is just volatiles/CAS
Let's not attack strawmen ...
So what did you mean by complex synchronization?
Thanks for the pointer. Of course there are multiple ways to interpret the results, but (playing the devil's advocate here) one perspective is that message exchange has an upper bound of 45 MHz on a multi-core, multi-GHz machine.Of course the benchmark isn't doing any work besides sending and receiving messages, otherwise it would benchmark something else than sending and receiving messages.
https://github.com/jboner/akka/tree/master/akka-actor-tests/src/test/scala/akka/performance/microbench
That was not the upper bound, that was after a bit of tuning, not full tuning, on a 6-core box.
This was just on a 6-core box, we're running tests on our 48-core box right now for additional optimization opportunities.
Right now we've found a bottleneck in java.util.concurrent.{ThreadPoolExecutor, LinkedBlockingQueue, ArrayBlockingQueue},
so we're working to build our own stuff for that.
Cheers,
√
Cheers,- Tiark
Sébastien Bocq
This question has been nagging at me for a while. Wouldn't it be possible for a compiler to allocate an efficient and recyclable stack-like structure for a restricted subset of a functional programming language without relying heavily on the runtime? For example, for a pure and total subset with single-threaded (linearly owned?) data access, ...? Btw, does anybody knows a mailing-list where one can ask such general question on functional programming?
Cheers,
Sébastien
√iktor Ҡlang
I read some interesting papers a couple of years ago about a language with compile-time garbage collection.
Unfortunately don't have a link, but I believe it was on LtU.
Cheers,
√
This question has been nagging at me for a while. Wouldn't it be possible for a compiler to allocate an efficient and recyclable stack-like structure for a restricted subset of a functional programming language without relying heavily on the runtime? For example, for a pure and total subset with single-threaded (linearly owned?) data access, ...? Btw, does anybody knows a mailing-list where one can ask such general question on functional programming?
/Jesper Nordenberg
Cheers,
Sébastien
martin odersky
√iktor Ҡlang
Thanks for the pointer, I think it was this paper I read: http://lambda-the-ultimate.org/node/2047
Cheers,
√
Chers-- Martin
Sébastien Bocq
Thanks for the pointers too. Region analysis looked promising, it is strange that this research was not pursued further. This reminds me of this famous quote:
"LISP programmers know the value of everything, but the cost of nothing" - Alan Perlis
Maybe the demand was not there at the time but maybe now would be a good time to get it out the fridge since functional programming seems to attract more performance minded people.
Cheers,
Sébastien
Paul Brauner
it's not always applied to garbage collection. This talk from François
Pottier (http://gallium.inria.fr/~fpottier/slides/fpottier-2007-05-linear-bestiary.pdf)
does a good job at summarizing this research type system wise.
Paul
2011/12/1 Sébastien Bocq <sebasti...@gmail.com>:
Sergei
esteem. Quite often, such people tend to project self-loathing
externally, exhibiting shocking ability to find and exaggerate flaws
in even the most beautiful things.
Surely Scala has plenty of areas for improvement, which shall be duly
noted and scheduled for prioritization and execution. Yet I would
advise the Scala team to avoid getting distracted from pursuit of a
bigger strategic goal by strongly expressed opinions regarding
features of tactical importance.
The discussions like this one here remind me of face-to-face
encounters involving Comp.Sc.Ph.D. with couple decades of experience
designing highly complex software systems vs. bright but not yet
educated and not yet experienced junior programmers working on
relatively simple "shuffle data back and forth" applications.
Scala specifically presents a "trap" to certain junior programmers, as
it may appear simple on the surface, while in fact one needs quite a
bit of genetically-determined mental capacity, education, and
experience to use Scala effectively.
On the other hand, certain junior programmers present a "trap" to
Scala community, as they may require spending a lot of attention and
hand-holding by more senior members, only to be found abandoning Scala
in the end.
I think this inherent tension explains quite of bit of the dynamics we
observe on this discussion board.
Erik Osheim
describe as "bright but not yet educated and not yet experienced junion
programmers" nor as people with "low self-esteem") think this is a real
problem, not just an anomaly.
I'm not sure what it is going to take to change the discussion from "is
there a problem?" to "how do we make the community more welcoming
and/or moderate the level of hostility" but we clearly need something.
Maybe some kind of internal poll or survey? I don't know.
Even if there is a mismatch between inexperienced newcomers and very
experienced and smart veterans, we must do better to keep discussions
friendly and relaxed (even when we disagree). In this I agree with Rex
Kerr's earlier post.
By the way Sergei, while I understand that you're trying to help, the
way you framed your email paints people who are worried about (or
affected by) the tone of our community in insulting terms.
Specifically, it describes people taking offense as either
"[projecting] self-loathing externally" or that they are "not yet
educated or experienced" or that they may lack "a bit of
genetically-determined mental capacity, education, and experience".
This is the kind of thing I'd like to see less of.
-- Erik
Josh Suereth
Stefan Wagner
Hash: SHA1
Am 01.12.2011 05:53, schrieb Aleksey Nikiforov:
> Clearly you are doing something wrong.
>
I made a new test - now I just compiled with -optimise, and here is the
result:
scala ForWhileBench 2000000000
0 while for rekdef
1 0.642 1.0 0.737
2 1.289 1.914 1.448
3 1.888 2.504 2.167
4 2.882 3.371 2.902
5 3.13 4.225 3.63
6 3.745 5.106 4.329
7 4.423 5.92 5.048
8 5.031 6.742 5.813
9 5.667 7.553 6.569
10 7.13 8.427 7.239
It's only a 20% difference between while and for, and nearly no
differnce to the tailcall recursive solution.
- --
Tsch���--->...Stefan
- ---------------------------
Don't visit my homepage at:
http://home.arcor-online.net/hirnstrom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk7X0VoACgkQQeATqGpDnRolmQCcDXI1wJrYFwOwdy9r79GCKOcW
EAkAn02w14qD6rH9S10qSqM8xRfoPtS4
=oGpb
-----END PGP SIGNATURE-----
Chris Marshall
> Subject: [scala-debate] Re: Yammer moving away from Scala
> From: hello....@gmail.com
Jori Jovanovich
On Wed, Nov 30, 2011 at 11:39:56PM +0100, Jesper Nordenberg wrote:Tons, of course. :)
> Hmmm, what benefits do you see in using Scala over Java if you only
> use plain old imperative loops?
At my job we do much of our work on Arrays (gasp) using while loops
(gasp) and yet we benefit from many awesome features in Scala:
* cleaner-looking code
* traits
* unified primitive/boxed types
* much better generics
* type classes via implicits
* pattern matching/extractors
These are only a subset of course.
I understand why a lot of us are defensive about this issue (no one
likes to be criticized, or have their favorite language criticized).
But there are legitimate reasons to want fast imperative-style loops
besides while loop, and legitimate Scala users who need or would like
to see this kind of use optimized.
Adriaan Moors
√iktor Ҡlang
a big, fat +1 on that!
Sergei
become exceedinlgy self-moderating and less intellectually challenging
(considered by some as "hostile").
I see it as a problem when a person with a liberal arts degree and
barely five years of coding experience expresses strong and mostly
wrong opinions regarding Scala vs. Java, and the experienced members
of the community start falling over themselves to placate him.
I deliberately framed my previous post to be rather direct and
inviting such strongly opinionated people to look in the mirror and
ask themselves why do they choose to post such unbalanced personal
views as an official decision of a well-known company.
You see, I've been bitten by that bug before - junior programmers
unwilling or incapable of learning how to properly use a technology,
and resorting instead to its critique based on rather irrelevant micro-
benchmarks.
The corollary to the "insecure people tend to blame others
excessively" is "very secure people tend to avoid blaming others
altogether". I'm trying to be balanced here and say what Martin and
other members of the community of comparable stature would perhaps
never say - "Don't blame the tool, learn how to use it properly
instead".
Naftoli Gugenheim
Adriaan Moors
I deliberately framed my previous post to be rather direct and
inviting such strongly opinionated people to look in the mirror and
ask themselves why do they choose to post such unbalanced personal
views as an official decision of a well-known company.
Daniel Sobral
> On Wed, Nov 30, 2011 at 3:43 PM, Jesper Nordenberg <mega...@yahoo.com>
> wrote:
>>
>> Aleksey Nikiforov skrev 2011-11-30 21:27:
>>
>>> When I first dicovered that for loops were not optimized I was surpised.
>>> When I discovered a resistance to optimizing them I was shocked. The
>>> decision to do nothing about for loops baffles me to this day.
>>
>>
>> very useful, instead effort should be put into generic optimization of HOF's
>> (and other inlining).
>>
>> /Jesper Nordenberg
>
>
>
> This is exactly the thinking that baffles me. There are for-loops in Java.
> Not comprehensions, LOOPs. Anyone coming from Java will expect for LOOPs or
> simlar functionality.
>
> It does not matter to me if it is a syntactic sugar or if they are
> comprehensions. What matters to me is to be able to write the code:
>
> var sum = 0; for (i <- 0 until array.length) sum += array(i)
The body of the loop compiles (with -optimize) to:
59: iload 8
61: iload_3
62: if_icmpne 93
(last iteration code elided)
93: iload 8
95: istore 6
97: aload 5
99: aload 5
101: getfield #50; //Field scala/runtime/IntRef.elem:I
104: aload_0
105: getfield #11; //Field XXS$$array:[I
108: iload 6
110: iaload
111: iadd
112: putfield #50; //Field scala/runtime/IntRef.elem:I
115: iload 8
117: aload 7
119: invokevirtual #53; //Method scala/collection/immutable/Range.step:()I
122: iadd
123: istore 8
125: goto 59
Java's
4: iload_2
5: aload_0
6: getfield #2; //Field array:[I
9: arraylength
10: if_icmpge 28
13: iload_1
14: aload_0
15: getfield #2; //Field array:[I
18: iload_2
19: iaload
20: iadd
21: istore_1
22: iinc 2, 1
25: goto 4
I had noticed before this strange characteristic of Scala bytecode of
breaking loop bodies in two parts, and jumping between them. I suppose
that's probably related to the inlining of closures, but, still, this
unnecessary jumping can't possible be good.
Anyway, notice how lines 104-111 in Scala are the same as lines 14-20
of Java. Where it differs is that Scala is using IntRef all over the
place. That's sum stored as IntRef on slot 5, on the heap. And look at
lines 93, 95 and 108: slot 8 is "i", but it reads it on 93, stores on
slot 6 on 95, then do the arithmetic, and next restores it from slot 6
on 108! Slot 6 need not exist at all, and it could have just read from
slot 8 at that point!
And on the elided part, it does this as it returns:
84: putfield #50; //Field scala/runtime/IntRef.elem:I
87: aload 5
89: getfield #50; //Field scala/runtime/IntRef.elem:I
92: ireturn
I even understand why it isn't optimizing this, but these lines would
be completely unnecessary if "sum" was on the stack in first place!
I'm impressed by the job done to inline the closure, but then it seems
to punt out on a minor optimization such as getting rid of IntRef. So
close, yet still so far...
Here are the test classes if anyone wants to look into it:
class XXS {
private val array = Array.range(0, 100)
def tst = { var sum = 0; for (i <- 0 until array.length) sum +=
array(i); sum }
}
class XXJ {
private int[] array;
XXJ() {
for (int i = 0; i < 100; i++)
array[i] = i;
}
int tst() {
int sum = 0;
for (int i =0; i < array.length; i++)
sum += array[i];
return sum;
}
}
>
> And have it work as fast as possible. Scala fails that expectation. It fails
> it so much that this is one of the most frequent questions on stack
> overflow. Also, it appears to be the number 1 on the list of problems with
> Scala as viewed by the employee at Yammer who wrote the email:
>
> 1. Don't ever use a for-loop.
>
> So, they were looking for loops as well, not comprehensions. Why not send
> them an email and ask them if perhaps they were "talking about for
> comprehensions or just foreach"?! If you think I was overreacting, I would
> LOVE to see their response.
>
>
> It is nice to have for-comprehensions optimized. But I really do not care
> about for-comprehensions. I want something that looks like for loops and
> works like for loops. 99% of my code uses emulated for-loops via
> while-loops. It's painful and error prone, so I take this issue very
> personally.
>
> Not providing proper for-LOOPs is akin to saying: Scala is not meant for the
> applications that require for-loops. Which sends the message that Scala is
> not meant for performance-sensitive applications. So guess what, the folks
> at Yammer got the message and went back to Java.
>
> You can tell me that they are comprehensions, or a more general case should
> be optimized. But I do not care about those uses at all. Ranges and closures
> can never be optimized to the level of while loops. And I need the LOOPs.
> And so do many other people not interested in functional aspects of Scala.
>
--
Daniel C. Sobral
I travel to the future all the time.
ARKBAN
> unwilling or incapable of learning how to properly use a technology,
> and resorting instead to its critique based on rather irrelevant micro-
> benchmarks.
Everyone has been bitten by that. In every industry. From the beginning
of human's history. That's what youth is, it questions the old and
established, sometimes correctly and sometimes incorrectly. I think you
will be happier if you accept that fact and try to work with it rather
than becoming frustrated. I'm not saying its easy.
ARKBAN
Sophie
> *Anything* can be interpreted as insulting by a person with low self-
> esteem. Quite often, such people tend to project self-loathing
> externally, exhibiting shocking ability to find and exaggerate flaws
> in even the most beautiful things.
Let me be one data point for you:
None of these apply to me : newbie, junior, not yet educated,
inexperienced, genetically short-changed.
Were I the target of some of the comments I have seen, I would be both
insulted and turned off.
As a reader of these comments targeting a third party, I am turned off.
Maintaining focus on strategic direction has nothing to do with being
pleasant and professional in a community discussion.
Sergei
importance of staying calm, professional, unacceptability of ad
hominem attacks etc. on a discussion board like this one, or in real
life for that matter.
Yet I also understand the simple fact of life that staying calm and
passive while being unjustifyably and ruthlessly attacked may get one
killed.
You would see none of the reaction I exhibited, or maybe a bit of a
calm, professional participation, if the original Yammer observation
was positioned along the lines of "Scala's relative inefficiency of
code optimization compared to Java forced us to use N more servers to
handle the load, which cost us additional $X per year. However,
Scala's productivity advantage compared to Java allowed us to do the
work with M less programmers, which is saving us $Y per year".
Having the ballpark $X and $Y, it would be easier to discuss the
relative merits of Scala and Java in the context of Yammer ecosystem.
There could be other metrics worthy of discussion, such as time to
market advantage that Scala enabled (or maybe did not enable in that
context), the resulting number of users captured (or lost to
competition), and the sum total of the lifetime values of those users
for the company.
Yet that was not the case at all. I personally perceived the original
Yammer post as a corporate blame-shifting game which attacked Scala
community unjustifiably. I may be right, or I may be wrong in that
perception. If I'm right, calmly arguing with the perpetrator of such
attack about the technical matters will do no good. And if I turn out
to be wrong, and the severity of the Scala shortcomings listed in the
Yammer post is independently confirmed, I will not shy away from
apology.
On Dec 2, 7:52 am, Sophie <itsme...@hotmail.com> wrote:
Cay Horstmann
I'm surprised I haven't seen this mentioned by now:
Email critique of Scala: https://gist.github.com/1406238
A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.
Here is a follow-up that puts it in perspective: http://eng.yammer.com/blog/2011/11/30/scala-at-yammer.html
I hope that some good will come out of this. For example, it's no good if a company has a coding rule that says "use var i = 0; while (i < n) { ... i += 1 } instead of for (i <- 0 until n)" That should be a standard optimization, not one that requires a special flag. I don't think it's rocket science to address this. Ditto with the performance of the hash map. It should not be necessary to use java.util.HashMap.
The angst about TraversableLike and builder typeclasses is understandable. I had to work through that when writing the collections chapter for Scala for the Impatient. I ended up advising readers to use Iterable or Seq, and to ignore all the other interfaces. I realize this will outrage someone somewhere. And yes, the operators have gotten a bit out of hand in the collections library. I put them all into a table, and the result wasn't pretty. (The problem isn't the operators, but their inconsistency.) There is a lot to like about the collections library, but it's not perfect. Perhaps it needs to get redone one more time, or perhaps Scaladoc needs to be smarter about giving a more user-centric view.
Some of the other issues raised surely resonate with most of us: IDEs, sbt/Maven, version incompatibilies. That will take blood, sweat, toil, and tears to work through. Eventually it will all work, but you can't blame a user to be frustrated about it.
Cheers,
Cay
martin odersky
On Wed, Nov 30, 2011 at 10:24 AM, Jeff Nowakowski <je...@dilacero.org> wrote:
I'm surprised I haven't seen this mentioned by now:
Email critique of Scala: https://gist.github.com/1406238
A sample new story on it: http://www.infoq.com/news/2011/11/yammer-scala
It's a very detailed email from an employee at Yammer on why they are migrating from Scala back to Java. I know it was intended to be private, but it's gone public now, and there's nothing personal in the email anyways.
Here is a follow-up that puts it in perspective: http://eng.yammer.com/blog/2011/11/30/scala-at-yammer.html
I hope that some good will come out of this. For example, it's no good if a company has a coding rule that says "use var i = 0; while (i < n) { ... i += 1 } instead of for (i <- 0 until n)" That should be a standard optimization, not one that requires a special flag. I don't think it's rocket science to address this. Ditto with the performance of the hash map. It should not be necessary to use java.util.HashMap.
The angst about TraversableLike and builder typeclasses is understandable. I had to work through that when writing the collections chapter for Scala for the Impatient. I ended up advising readers to use Iterable or Seq, and to ignore all the other interfaces. I realize this will outrage someone somewhere. And yes, the operators have gotten a bit out of hand in the collections library. I put them all into a table, and the result wasn't pretty. (The problem isn't the operators, but their inconsistency.) There is a lot to like about the collections library, but it's not perfect. Perhaps it needs to get redone one more time, or perhaps Scaladoc needs to be smarter about giving a more user-centric view.
Some of the other issues raised surely resonate with most of us: IDEs, sbt/Maven, version incompatibilies. That will take blood, sweat, toil, and tears to work through. Eventually it will all work, but you can't blame a user to be frustrated about it.
David Hall
> I think two of the things we need to do on the docs side is hide
> implementation details better in the ScalaDoc and put a link to the
> collections API overview
>
> http://www.scala-lang.org/docu/files/collections-api/collections.html
>
> into the scaladoc of each collection class.
I'm a little worried about further hiding implementation details in
the docs. Part of Coda's point was that all the hiding and such
promote cargo cult engineering. I'm not sure how much there is to be
done about it, but it definitely seems like trying to hide things like
type class implicits in the docs is only going to lead to arguments of
the form "Scala is so complex that they have to hide it from you!"
Maybe that's not what you meant, but it definitely seems worrisome. I
don't have any constructive ideas, unfortunately.
-- David
Paul Phillips
Yay.
> One other thing I thought of was merging the Traversable and Iterable
> layers if collections, because this would get rid of about 20% of the
> supertraits of standard collection classes.
Yes please.
martin odersky
On Fri, Dec 2, 2011 at 11:08 AM, martin odersky <martin....@epfl.ch> wrote:I'm a little worried about further hiding implementation details in
> I think two of the things we need to do on the docs side is hide
> implementation details better in the ScalaDoc and put a link to the
> collections API overview
>
> http://www.scala-lang.org/docu/files/collections-api/collections.html
>
> into the scaladoc of each collection class.
the docs. Part of Coda's point was that all the hiding and such
promote cargo cult engineering. I'm not sure how much there is to be
done about it, but it definitely seems like trying to hide things like
type class implicits in the docs is only going to lead to arguments of
the form "Scala is so complex that they have to hide it from you!"
Simon Ochsenreither
I'm sick and tired that the conclusion of “language experts” that Scala sucks is based to 90% on those three method names.
Tony Morris
Are you going to let wrong people affect the API of scala? It can only get better from here!
HamsterofDeath
Are you going to let wrong people affect the API of scala? It can only get better from here!
On Dec 3, 2011 7:55 AM, "Simon Ochsenreither" <simon.och...@googlemail.com> wrote:
I know, it sounds lame ... but can we get rid of /:, :\ and /:\?
tell them to use fold, foldLeft and foldRight if they don't want to use non letter methods.I'm sick and tired that the conclusion of �language experts� that Scala sucks is based to 90% on those three method names.
if i listened to conclusions like this, i would now be very, very confused because there are so many religions, and i am supposed to join any of them because they are all correct.
Tony Morris
Hash: SHA1
On 12/03/2011 07:07 PM, HamsterofDeath wrote:
> Am 02.12.2011 23:23, schrieb Tony Morris:
>>
>> Are you going to let wrong people affect the API of scala? It can only
>> get better from here!
>>
>> On Dec 3, 2011 7:55 AM, "Simon Ochsenreither"
>> <simon.och...@googlemail.com
>> <mailto:simon.och...@googlemail.com>> wrote:
>>
>> I know, it sounds lame ... but can we get rid of /:, :\ and /:\?
>>
>> I'm sick and tired that the conclusion of �language experts� that
>> Scala sucks is based to 90% on those three method names.
>>
> tell them to use fold, foldLeft and foldRight if they don't want to use
> non letter methods.
>
> if i listened to conclusions like this, i would now be very, very
> confused because there are so many religions, and i am supposed to join
> any of them because they are all correct.
>
You could just get on with it, like us irreligious people do. Try it!
- --
Tony Morris
http://tmorris.net/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJO2egyAAoJEPxHMY3rBz0PqusH/Au+wWCV+aoDDTo1wK+lL0Jt
ssNgiqXozm9QDfF65t7PpDCnx0AipLy1h2b2aBqY0PG5GSXJvbRKPd0oGO+JKw86
MRKo92k1FJXKZDKPfrylJfQtFmiwty3fm+Vz4xOF++vF87ZbJ/ZSpRcLqu+HzySb
jRXhobJOZtBhgiFHJ2597/BaH5+cEaTGEIEf9b9J6LL/2J9j0DLBZKVPWBBXBvUy
DA4K7WJfTHg7QnQ8LE/yhFfrGs3ydpAxFF74lEcPprkCMEwYYmMnRN5I7hM/F1OI
OIasbqxNK40G+7LfwxVLkbW/WpoysKpKR5qPBZklXVglJgDjajKMEI64Og2+IQY=
=CNMr
-----END PGP SIGNATURE-----
Tim Pigden
tostart them with the text forms. So avoid too many new symbols
intutorials - maybe if it's an obvious beginner use the text name
inscala-user or stackoverflow as an alternative when you're trying to
behelpful.
Chances are they won't get to foldLeft etc straight away
anyway,because they have to understand a bit functional programming
paradigmbefore they want to use it.
Once they get that far, and appreciate the use of the concepts,
thenchanging from textual to symbolic function names is just another
smallthing to learn.
Tim>
> On 3 December 2011 09:07, HamsterofDeath <h-s...@gmx.de> wrote:
>> Am 02.12.2011 23:23, schrieb Tony Morris:
>>
>> Are you going to let wrong people affect the API of scala? It can only get
>> better from here!
>>
>> On Dec 3, 2011 7:55 AM, "Simon Ochsenreither"
>> <simon.och...@googlemail.com> wrote:
>>>
>>> I know, it sounds lame ... but can we get rid of /:, :\ and /:\?
>>>
>>> I'm sick and tired that the conclusion of “language experts” that Scala
>>> sucks is based to 90% on those three method names.
>>
>> tell them to use fold, foldLeft and foldRight if they don't want to use non
>> letter methods.
>>
>> if i listened to conclusions like this, i would now be very, very confused
>> because there are so many religions, and i am supposed to join any of them
>> because they are all correct.
--
Tim Pigden
Optrak Distribution Software Limited
+44 (0)1992 517100
http://www.linkedin.com/in/timpigden
http://optrak.com
Optrak Distribution Software Ltd is a limited company registered in
England and Wales.
Company Registration No. 2327613 Registered Offices: Orland House,
Mead Lane, Hertford, SG13 7AT England
This email and any attachments to it may be confidential and are
intended solely for the use of the individual to whom it is addressed.
Any views or opinions expressed are solely those of the author and do
not necessarily represent those of Optrak Distribution Software Ltd.
If you are not the intended recipient of this email, you must neither
take any action based upon its contents, nor copy or show it to
anyone. Please contact the sender if you believe you have received
this email in error.
Simon Ochsenreither
Those people are not rational, and there is always the "but we still have to read other people's code" point.
Really, I know how folds work and I can explain them to people _interested_ in it. But the point is that I would prefer showing people more useful things instead of wasting my time to fight myths or to get beginners back to sanity after they go "OMGOMGOMG LINE NOISE OPERATOR OVERLOADING?!!?!?!?!?!".
I claim that we probably loose 90% of those interested in Scala when they see stuff like /: /:\ :\. We may not like it, but sometimes it might make sense to make some small changes for a greater good - we don't want people to be stuck with Java, right?
What about marking them with @migration and @bridge? We keep the methods, don't break compatibility, but make sure they stop appearing where they hurt?
Thanks and bye,
Simon
Kevin Wright
Tony Morris
Hash: SHA1
That method of teaching backfires, universally.
On 12/03/2011 07:25 PM, Tim Pigden wrote:
> If we're talking about people from java world then you might want
> tostart them with the text forms. So avoid too many new symbols
> intutorials - maybe if it's an obvious beginner use the text name
> inscala-user or stackoverflow as an alternative when you're trying to
> behelpful.
> Chances are they won't get to foldLeft etc straight away
> anyway,because they have to understand a bit functional programming
> paradigmbefore they want to use it.
> Once they get that far, and appreciate the use of the concepts,
> thenchanging from textual to symbolic function names is just another
> smallthing to learn.
>
> Tim>
>> On 3 December 2011 09:07, HamsterofDeath <h-s...@gmx.de> wrote:
>>> Am 02.12.2011 23:23, schrieb Tony Morris:
>>>
>>> Are you going to let wrong people affect the API of scala? It can only get
>>> better from here!
>>>
>>> On Dec 3, 2011 7:55 AM, "Simon Ochsenreither"
>>> <simon.och...@googlemail.com> wrote:
>>>>
>>>> I know, it sounds lame ... but can we get rid of /:, :\ and /:\?
>>>>
>>>> I'm sick and tired that the conclusion of �language experts� that Scala
>>>> sucks is based to 90% on those three method names.
>>>
>>> tell them to use fold, foldLeft and foldRight if they don't want to use non
>>> letter methods.
>>>
>>> if i listened to conclusions like this, i would now be very, very confused
>>> because there are so many religions, and i am supposed to join any of them
>>> because they are all correct.
>
>
>
- --
Tony Morris
http://tmorris.net/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEcBAEBAgAGBQJO2etbAAoJEPxHMY3rBz0Pa38H/ipe/UAKC+UkHFD4foA72n51
2+fKDrSOQKH0l+ey6sAqrVIAEZi6MDXf1ZaSvETq3RvavrGhJUqUQqO0O08vDLYe
HZdHLUaMF9ETWJovcvgE7Ob5qyH22oNzHaBez09yvi2Z35Kc/IYKgro0WYdKwnZy
xnBGIAp/qWuGwZb8yBzTblcZ+T4qTS3Wq6tIKtiIqkl8y6xInX/P6huj+y0fqrFi
F7hXVQ5X053xXTxn/fV42jo5oRqMQCq5fusrhTKPj2dvzsPfxT7Yn2K2/HeZTllr
rGAJVAi3eRwDI0Mh5OWUOeNq9mvBlmq3sI05Of3YeYOCvX82+CXl2kdEL2mHCOQ=
=pO/z
-----END PGP SIGNATURE-----
HamsterofDeath
and about those 90% - even if your number is correct and there really
are millions of software developers being scared off by things they
don't understand at first glance - i don't want people with such a
mindset having a strong influence. it'll slow down progress. look at
human history:
1. many people believe in a/are used to a
2. some claim b is much better than a
3. time passes. new, fresh and people not yet used to anything get to
choose between a and b
4. people choosing b either outperform those choosing a or not
5. new people prefer a or b depending on the outcome of 4.