Issues hindering Scala adoption in enterprises

284 views
Skip to first unread message

amulya rattan

unread,
Mar 30, 2012, 9:48:36 AM3/30/12
to scala-debate, scala-l...@googlegroups.com, scala-user
Cool developers,

I must preface this by saying I absolutely love Scala and frameworks that are built on it. I wanna marry them all, move to south of France and start a band named "Scalalala", dont ask why. But there are some serious _serious_ __serious__ problems with it that are definitely avoiding a fast adoption of Scala everywhere. 

On the face of it, comparison of Scala and Java leaves no doubt that one should make a good investment in the future and that future is Scala. One reads a Scala book, scribbles few lines of code on REPL and goes, "This is so cool! I'll go tomorrow and introduce my team to some of this awesomeness". Learning/Understanding its intricate Type system feels like an acid trip, and a good one at that. One comes out a better man. Implicits are yummy. Traits are simple and powerful. Introduction of Typeclasses from Haskell is more than welcome. Merging of OO and FP to seamlessly transition from one style to other is neat. Frameworks like Scalaz can be mind-bending, requiring a lot of effort and diligence, but you know if you apply yourself, you can understand it. And dont even get me started about Akka, its a work of art. 

Wow, so many positives!! So why isn't everyone investing their resources into it?? Here are few silly/rebuttable reasons/arguments:

1. Some people are just adverse to change: And you can't do anything about it. They want to continue living in null land and so be it.
2. Some people are just BIG java fans and don't want to use anything else: To them Java, like many other languages, is like a religion and they don't wanna convert. Java is turing-complete and you can do everything in it, just more verbosely and less runtime-safe-ly.    
3. Some people make effort but fallback on "Scala is too complex" cuz they read it is somewhere: Well this has come up many times before and doesn't need any more argumenting. For my 2 cents, Complex is just intertwining of a lot of simple things. Plus since most programmers are engineers and hence smart people, one would expect them not to shy away from complex but embrace it, break it down and see what sort of scary organism is living down there. However, thats not how the world works. And there is a flip-side to it, since some people would love to understand the "hairy" concepts but are busy otherwise.  

However, here are few arguments that are legitimate and definitely something that Scala team and contributors, perhaps need to contemplate on:

1. Compile time is super slow - And yes, running a seperate server especially for compilation doesn't really help a lot. It doesn't. And its not a single incident. Bigger your code base is in scala, worse are the times. I am afraid to pitch Scala for our products, since I am sure he'll show me this http://xkcd.com/303/ I thought we were out of C++ compilation era. I am sure scala compiler has to perform a lot of magic to convert to optimized bytecode and go through all its 25 elaborate stages. However, instead of working on all other cool features added to Scala for now, couldn't one release be dedicated to making compiler faster. Or am I just asking for too much? Also:

Recently Play framework moved to 2.0 with a very aggressive introduction of Scala in it and the response has been pretty much violent. If you have been following play's mailing list, you'd know what I am talking about. First comes the "scala is a nice language but too complex" argument which looses Play a lot of territory with early adopters moving to Django. Next people who are trying to use Scala are faced with horrible compile times in DEV mode. Here is a guy with similar issue of how using Play with Scala took all the speed away:

Now this guy at least tried using scala, but failed not because scala is complex but because Scala has very slow compilation. And he presents a _very_ legitimate problem that other organizations are facing or will face if they started using scala unless Scala team does something about it.

2. Cryptic errors - Everyone has seen them so I don't need to elaborate.

3. Those run time stack traces just leave me hanging

These three arguments are as legitimate as anything can be and as far as I know, every Scala developer has seen them. And they are all compiler-magic centric. And while Scala is all about about compiler-magic, does it leave no scope for optimization or errors that are more _day-to-day_ developer friendly than _rockstar_ developer friendly. Not everyone who is a programmer is super-smart and can go around those issues using their black-magic skills. A day-to-day developer in an organization wants to get through a task as fast as possible because that's what business end demands off of him/her. Software engineering doesn't exist on its own, there is a big layer of business on top of it and biggest business need is smallest Time_To_Market. 

If Scala needs to take off as it aspires to or as it should, it needs to cater to day-to-day developer. The developer who just wants a fast compile time, a error explaining what he did wrong or what could go wrong and a stack trace explaining what kind of muck he created. Not every organization would want to have a Typesafe subscription to ensure they avoid scala related issues. They expect their everyday developer to take care of language-related issues. 

Now I spent my time writing this to extract what the other developers in scala feel about this and to know if Scala people do consider this a major issue and have a plan to solve it very soon or dont know of any existing solution to it or simply dont give a muck about it. 

Expecting some responses.

~Amulya

Aleksey Nikiforov

unread,
Mar 30, 2012, 10:16:13 AM3/30/12
to amulya rattan, scala-debate, scala-l...@googlegroups.com, scala-user
1. Using SBT resolves 95% of slow compilation issues. Giving more ram to compiler also helps.

2. Messages eventually start to make sense.

3. Stack traces start to make sense as well (just focus on your package names and line numbers).

Daniel Sobral

unread,
Mar 30, 2012, 10:26:36 AM3/30/12
to amulya rattan, scala-debate, scala-l...@googlegroups.com, scala-user
The issues you raise are known and worked on to the extent possible,
but there are very real limits on what can be done.

On the cryptic error messages, if you compare messages produced by
2.7.x, 2.8.x, 2.9.x and 2.10.x, you'll see there has been steady
improvement. String representations of types have been altered to
better show how different types differ (for example, tuples vs
multiple parameters), common issues now come with suggestion for the
usual cause/solution, failed implicit resolution gained an annotation
so that the implicit author can write the error message, and so on.
IMHO, error messages on 2.9.x cannot be called "cryptic" anymore, with
a few exceptions. Nowadays, questions about error messages on Stack
Overflow are usually answered with "Yes, the problem is exactly what
the error message is saying, and the solution too." If you (or anyone
else!) feel otherwise, perhaps it is time to put forward examples of
error messages that are still deemed cryptic.

Mind you, one can get error messages that are difficult to comprehend
when one uses advanced features such as the ones Martin wants to put
behind a fence with SIP-18, but the problem is not so much the error
message, but the code that has been written.

With regards to compiler speed, this has been a constant focus of Paul
Phillips for as long as I have been involved with Scala (and I assume
before that as well), and there have been and there are huge efforts
to improve it. The whole interruptible presentation compiler is one
part of that effort, as is the better incremental compilation of the
next version of Scala IDE.

But, ultimately, there are hard limits on how much it can be improved.
JVM cannot represent many concepts in Scala, which makes it necessary
for the compiler to do more work than javac has to. Traits, for
instance, means tracing class file dependency is much harder for Scala
than for Java. Methods provided through implicit conversion are much
slower to compile than methods provided by the static type. Implicit
parameter search is inherently slow, and type inference is hard.

One can improve compilation speed of one own's code by specifying
types, avoiding methods provided implicitly, reducing number of
implicits visible as much as possible, as well as implicit usage.

As for stack traces, there's nothing the compiler can do about that,
and the JVM, again, doesn't help: it doesn't have a field for column
range of code, for example. External tools will have to be developed
to deal with it, and the present effort for that is on Scala IDE.
Given the number of improvements that Scala IDE must gain, I fear the
priority for this is just not there.

--
Daniel C. Sobral

I travel to the future all the time.

Iain Cambridge

unread,
Mar 30, 2012, 10:37:39 AM3/30/12
to scala-debate, scala-l...@googlegroups.com, scala-user
Oops forgot reply-to all :(

---------- Forwarded message ----------
From: Iain Cambridge <bac...@backie.org>
Date: 30 March 2012 15:25
Subject: Re: [scala-user] Re: [scala-debate] Issues hindering Scala adoption in enterprises
To: Aleksey Nikiforov <lex...@gmail.com>


I've only been playing about with Scala for a few weeks so maybe it's due to lack of experience but here's my opinion.

1. I have to say it seems like the root cause of his speed issue is with the Play Framework itself.

2. The messages so far for me haven't been that cryptic, but I am coming from a PHP background which are pretty cryptic.

3. Also I've not found them that hard to understand.

Iain

Scott Spillmann

unread,
Mar 30, 2012, 12:26:22 PM3/30/12
to scala...@googlegroups.com
I have to agree with Ian here.  I ran through tutorials and now feel comfortable enough in Scala to be productive.  We're even starting a new enterprise level project that is web-based using Scala+Java (not Play though, too heavy). 

If things seem hard to comprehend in a pure scala way, then do it in scala using java methods and techniques and eventually you'll learn more Scala techniques.  Incidentally, this technique will also speed up compile time and make messages easier to understand at first.

-Scott

Mark_Grand

unread,
Mar 30, 2012, 2:19:36 PM3/30/12
to scala-user
I live in the Atlanta area and have attended a couple of scala meet-
ups. One thing that I have noticed about the attendees of these
gatherings it that they generally have a much stronger computer
science background than most Java programmers. I wonder if
appreciating the advantages of Scala requires a deeper knowledge of
computer science than most Java programmers have.

For me, the biggest productivity win for Scala is its support for
multiple programming paradigms. This is valuable to me because it
allows me to combine multiple paradigms such as functional programming
and pattern matching in a way that is more powerful than either of
these alone. This advantage is lost on programmers who only understand
one paradigm.

For me, the second biggest productivity win for Scala is its type
system, which catches higher level bugs at compile time than does
Java. However, taking full advantage of the type system requires a
more abstract style of coding than Java, which is not helpful for the
many programmers to are not skilled at thinking in abstractions.

All this leads me to suspect that the future of Scala may be as the
language of a highly productive few rather than as a language of the
average many. This is not a happy conclusion for me. Has anyone had
experiences that would lead to a different conclusion about the future
of Scala?
> show me thishttp://xkcd.com/303/I thought we were out of C++ compilation
> era. I am sure scala compiler has to perform a lot of magic to convert to
> optimized bytecode and go through all its 25 elaborate stages. However,
> instead of working on all other cool features added to Scala for now,
> couldn't one release be dedicated to making compiler faster. Or am I just
> asking for too much? Also:
>
> Recently Play framework moved to 2.0 with a very aggressive introduction of
> Scala in it and the response has been pretty much violent. If you have been
> following play's mailing list, you'd know what I am talking about. First
> comes the "scala is a nice language but too complex" argument which looses
> Play a lot of territory with early adopters moving to Django. Next people
> who are trying to use Scala are faced with horrible compile times in DEV
> mode. Here is a guy with similar issue of how using Play with Scala took
> all the speed away:https://groups.google.com/forum/#!msg/play-framework/AcZs8GXNWUc/Ianb...

Alex Cruise

unread,
Mar 30, 2012, 2:23:50 PM3/30/12
to Mark_Grand, scala-user
On Fri, Mar 30, 2012 at 11:19 AM, Mark_Grand <mgr...@markgrand.com> wrote:
One thing that I have noticed about the attendees of these
gatherings it that they generally have a much stronger computer
science background than most Java programmers.  I wonder if
appreciating the advantages of Scala requires a deeper knowledge of
computer science than most Java programmers have.

It helps A LOT, but I wouldn't say it's a prerequisite.  I have very little formal CS background, and learning Scala has thrown me into the deep end. And I'm extremely grateful for that. :)

-0xe1a

Aleksey Nikiforov

unread,
Mar 30, 2012, 2:27:51 PM3/30/12
to Mark_Grand, scala-user
I think what Scala needs is more useful and rock-solid libraries that significantly reduce boilerplate code or provide features unique to Scala. Once these are written we will see much stronger adoption, because companies will want to cut on development time and complexity by using these libraries.

Scala alone is not enough to open the door, because Java libraries can be used from Java, and the standard library can only cover so much ground.

Clint Gilbert

unread,
Mar 30, 2012, 2:49:30 PM3/30/12
to scala...@googlegroups.com
On 03/30/2012 10:16 AM, Aleksey Nikiforov wrote:
> 1. Using SBT resolves 95% of slow compilation issues. Giving more ram to
> compiler also helps.

I wish it were so easy. :/

I'm skeptical that SBT is a silver bullet - my experience with it hasn't
been entirely positive** - but even if it is, switching to SBT isn't an
option for many projects.

Where I work, we've been introducing Scala bit by bit into a
decent-sized multi-module Maven project. We can't - and don't want to!
- switch to SBT. This is a common scenario.

Scala compilation speed is a very big issue because it has wide-ranging
effects. For example, using the latest Scala IDE for Eclipse, I
frequently get modal popups because I'm editing several files (or just
the same file) too quickly, because builds triggered by one change
aren't finished when the next build is triggered. The Eclipse plugin
uses SBT for dependency tracking, as I understand it, and (in theory)
should be recompiling only as many files as SBT would from the command
line. :(

** This is not a knock on SBT. Dependency management and
build-wrangling for large multi-module projects is inherently complex,
and probably won't ever be pretty.

Jean-Francois Im

unread,
Mar 30, 2012, 7:09:39 PM3/30/12
to amulya rattan, scala-debate, scala-l...@googlegroups.com, scala-user
Even on the Play 1.2.x series, the scala plugin was horrendously slow.
Given the choice between Java and Scala though, I'd rather type less
and wait for the computer to compile; at least one of them leaves my
hands free to grab a sip of coffee. :)

With regards to speed, though, I think it has already been said that
it won't ever match the speed of javac. Java is a very simple language
compared to Scala and the fact that source file names map to class
names makes it trivial to find sources to compile when doing
dependency analysis; Scala on the other hand allows you to name your
files as you like and cram several classes in a file, so tools like
sbt must be aware of that when recomputing dependencies.

That being said, if speed is the utmost concern, there are other
languages that run on the JVM that share some similarities with Scala.
Kotlin by JetBrains, Xtend by the Eclipse Foundation and Ceylon by
RedHat probably all compile much faster than Scala. In fact, one of
the design goals of Kotlin is to compile at roughly the same speed as
javac while being "simpler than Scala."

There might also be an issue with the command line parameters for sbt;
I know that on my machine, sbt often fills the entire permgen when
compiling a Play 2 app and then dies there with the VM needing to be
kill -9'ed. By default, it will only run with -Xmx512M, but bumping
the permgen a bit and running with -client (on Mac, since -server is
horribly broken there) makes a big difference. YMMV. Try checking what
happens inside of sbt using VisualVM and see if there's anything
needing to be fixed.

I wouldn't complain though if the Scala compiler was faster. It should
also come with free cupcakes, courtesy of Typesafe. :)

Message has been deleted

Cédric Beust ♔

unread,
Mar 31, 2012, 2:30:22 AM3/31/12
to Mark_Grand, scala-user
On Fri, Mar 30, 2012 at 11:19 AM, Mark_Grand <mgr...@markgrand.com> wrote:
For me, the biggest productivity win for Scala is its support for
multiple programming paradigms.

This is another example where the parallel between C++ and Scala is very strong, so this discussion sounds very familiar to anyone who used to write C++ code back in the days. Stroustrup repeated over and over that C++ is *not* an OO language, it's a multi-paradigm language.

The problem with this approach is not just that such languages negatively impact people who "only understand one paradigm", to quote you, but it just adds unavoidable complexity in the language in the way the features coming from the different paradigms collide.

Finally, I'd like to caution you against the condescension tones that shine through your email:
    • "This advantage is lost on programmers who only understand one paradigm."
    • "Java, which is not helpful for the many programmers to are not skilled at thinking in abstractions."
      This kind of attitude is precisely what gave Scala a bad reputation in the first place.

      In the immortal, albeit slightly reworked, words of xkcd:

      Inline image 1

      -- 
      Cédric



      xkcd-annoying.png

      Geir Hedemark

      unread,
      Mar 31, 2012, 3:23:11 AM3/31/12
      to Clint Gilbert, scala...@googlegroups.com
      On 2012, Mar 30, at 8:49 PM, Clint Gilbert wrote:
      >> 1. Using SBT resolves 95% of slow compilation issues. Giving more ram to
      >> compiler also helps.
      > I'm skeptical that SBT is a silver bullet - my experience with it hasn't
      > been entirely positive** - but even if it is, switching to SBT isn't an
      > option for many projects.

      For compilation time, it more or less is. SBT has all kinds of other problems, but the maven scala compiler is double-plus ungood.

      Of course you can switch to SBT. You are using the same compiler. You mean you don't want to make the investment, and that is a sentiment I can understand. We stuck around with maven for more than a year before we switched to SBT.

      Geir

      Francois

      unread,
      Mar 31, 2012, 3:26:59 AM3/31/12
      to Clint Gilbert, scala...@googlegroups.com
      On 30/03/2012 20:49, Clint Gilbert wrote:
      On 03/30/2012 10:16 AM, Aleksey Nikiforov wrote:
      
      1. Using SBT resolves 95% of slow compilation issues. Giving more ram to
      compiler also helps.
      
      I wish it were so easy. :/
      
      I'm skeptical that SBT is a silver bullet - my experience with it hasn't
      been entirely positive** - but even if it is, switching to SBT isn't an
      option for many projects.
      
      Where I work, we've been introducing Scala bit by bit into a
      decent-sized multi-module Maven project.  We can't - and don't want to!
      - switch to SBT.  This is a common scenario.


      I really want to emphasis that, I can't agree more. For all of you saying "just use SBT", you are just missing the point, especially in a thread with such a subject.


      Scala compilation speed is a very big issue because it has wide-ranging
      effects.  [...]
      

      I believe it's even worse than that: hyper-quick tools allows complettly new use cases and patterns for them.

      Git is so successful also because it is insanly efficient. Think to your common Git use cases with a 2 seconds lags on each command (or better use an alias to actually try it), and you will see that they are just not workable in such a scenario.

      An other example is the Java compiler, so much quick that IDE developers can just rebuild a whole project (or most of it) in place of having to manage complexe source file dependency paths.

      So, yes, an insanly quick Scala compiler would be awesome, and could lead to a full new range of Scala use cases (scripting ?). Of course, we are on the real world with its bounds...

      And because all of that, I'm really thankful to all of Scala developers working hard on compilation efficiency and any other things allowing a feeling of speed (smaller jar, modularized user lib, etc), thanks for the hard work !

      -- 
      Francois ARMAND
      http://fanf42.blogspot.com
      http://www.normation.com
      

      martin odersky

      unread,
      Mar 31, 2012, 3:41:58 AM3/31/12
      to Francois, Clint Gilbert, scala...@googlegroups.com
      A warm scalac gives me about 1000 lines per second on moderately
      complex Scala code on a 2 year old IMac. Not great, but not
      catastrophic either, in particular if you can make use of incremental
      compilation, compiling only those files that need it. And, it looks
      like it might be very hard to change that speed. Improving compilation
      speed has been in he focus of several people for quite some time now,
      but progress has been slow.

      But 1000 lines a second is not bad at all if you are smart about it. A
      Play template is maybe 100 lines, so that should be 1/10sec, not the
      reported 13 sec. The main challenge here is (1) keep the compiler
      warm, and keep as much info as is safe loaded, and (2) make that warm
      compiler available in as many environments as possible. (1) is
      essentially solved by fsc. Simply call fsc instead of scalac from the
      command line, and you are set. Or use fsc for IntelliJ. sbt is a bit
      more conservative and therefore a bit slower, but still much faster
      than compiling from scratch with scalac. The main problem is that if
      you use other build tools, like maven, then you are stuck, and have to
      suffer really bad build times.

      We are currently working on making incremental compilation available
      to maven and other build tools. Hopefully that will be a constructive
      solution to the issue.

      Cheers

      - Martin

      Francois

      unread,
      Mar 31, 2012, 3:55:14 AM3/31/12
      to martin odersky, Clint Gilbert, scala...@googlegroups.com
      On 31/03/2012 09:41, martin odersky wrote:
      > A warm scalac gives me about 1000 lines per second on moderately
      > complex Scala code on a 2 year old IMac.

      As a reference, have you any idea of the speed of javac (so that I could
      have a feeling of the gap) ? 10 000 ? 100 000 ? Even more ?

      > [...] We are currently working on making incremental compilation available


      > to maven and other build tools. Hopefully that will be a constructive
      > solution to the issue.

      That's awesome ! Hope it would

      Tony Morris

      unread,
      Mar 31, 2012, 3:51:21 AM3/31/12
      to scala...@googlegroups.com

      I often use sbt with mountains of scala code -- a project depending on
      internal libraries depending on scalaz, etc. I rarely wait longer than 3
      or so seconds for compiler feedback. The incremental compilation helps
      immensely, so long as you have enough memory.

      --
      Tony Morris
      http://tmorris.net/


      HamsterofDeath

      unread,
      Mar 31, 2012, 4:59:10 AM3/31/12
      to scala...@googlegroups.com
      the master himself has praised intellij :D

      Alberto SOUZA

      unread,
      Mar 31, 2012, 7:07:35 AM3/31/12
      to HamsterofDeath, scala...@googlegroups.com
      I don't know...  I think that one of the biggest problem here is that each one has a different opinion about the compilation... At least for me, 3 seconds to get feedback for compilation is real good... In my case here, we have to wait much more... ,  in a lot of cases, the wait is of 2 minutes. We made a big refactoring here, deleting an amount of implicit, and even with this change, the compilation time still too hard...

      Andreas Joseph Krogh

      unread,
      Mar 31, 2012, 9:12:22 AM3/31/12
      to scala...@googlegroups.com

      That's great news! Any place we can read about/test improved build-times
      for a multi-module Maven scala-project?

      --
      Andreas Joseph Krogh<and...@officenet.no> - mob: +47 909 56 963
      Senior Software Developer / CEO - OfficeNet AS - http://www.officenet.no
      Public key: http://home.officenet.no/~andreak/public_key.asc

      Daniel Sobral

      unread,
      Mar 31, 2012, 10:01:12 AM3/31/12
      to scala-l...@googlegroups.com, scala-...@googlegroups.com, scala...@googlegroups.com
      On Sat, Mar 31, 2012 at 04:22, Jesper Nordenberg <mega...@yahoo.com> wrote:
      > I don't see any inherent reason why scalac should be much slower than javac.
      > Yes, it has to do implicit searches, type inference etc., but that doesn't
      > make up for the difference in compile time. If there are no noticeable
      > performance hotspots, there must be excessive object allocation or use of
      > inefficient algorithms in the compiler.

      Given that Odersky has written the basis to the current javac
      (http://www.scala-lang.org/node/241), I think it should be taken for
      granted that there _are_ inherent reasons for the huge performance
      gap.

      As someone who reads the e-mails and commits by Paul Phillips in his
      quest for the holy speed, I can also tell you that serious work has
      been put in improving that speed. All the same, anyone can grab
      profilers and other performance tools, see what's going on with the
      compiler, and take a shot at getting a few more seconds out of it. I
      can't think of any commit I'd rather have in Scala than one that
      begins with "Improve compile speed". :-)

      >
      > /Jesper Nordenberg

      --

      S Wade

      unread,
      Mar 31, 2012, 10:24:21 AM3/31/12
      to martin odersky, scala...@googlegroups.com
      This is great news. I'm guessing that, like me, many users work with non-sbt builds (for a variety of reasons). This will make our lives significantly better.

      thanks and keep up the good work,
      wade

      Brian Schlining

      unread,
      Mar 31, 2012, 12:36:17 PM3/31/12
      to martin odersky, Francois, Clint Gilbert, scala...@googlegroups.com


      We are currently working on making incremental compilation available
      to maven and other build tools. Hopefully that will be a constructive
      solution to the issue.


      +1 !! 


      --
      ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
      Brian Schlining
      bschl...@gmail.com

      Eric Kolotyluk

      unread,
      Mar 31, 2012, 1:32:11 PM3/31/12
      to scala...@googlegroups.com
      +1

      Nils Kilden-Pedersen

      unread,
      Mar 31, 2012, 6:00:23 PM3/31/12
      to martin odersky, scala...@googlegroups.com
      On Sat, Mar 31, 2012 at 2:41 AM, martin odersky <martin....@epfl.ch> wrote:
       (2) make that warm
      compiler available in as many environments as possible. 

      Running sequentially presumably?

      Is it an insurmountable task (realistically) to make the compiler run in parallel? If not, perhaps that could be the next step towards making it many times faster, or is that a pipe dream?

      Aleksey Nikiforov

      unread,
      Mar 31, 2012, 10:52:59 PM3/31/12
      to Nils Kilden-Pedersen, martin odersky, scala...@googlegroups.com
      Compiler has been running in parallel for many years now. Trying compiling a reasonable size project and watch your task manager.

      Nils Kilden-Pedersen

      unread,
      Apr 1, 2012, 11:15:15 AM4/1/12
      to Aleksey Nikiforov, martin odersky, scala...@googlegroups.com
      On Sat, Mar 31, 2012 at 9:52 PM, Aleksey Nikiforov <lex...@gmail.com> wrote:
      Compiler has been running in parallel for many years now. Trying compiling a reasonable size project and watch your task manager.

      I must have faulty memory then. Thanks.

      Daniel Sobral

      unread,
      Apr 1, 2012, 12:27:05 PM4/1/12
      to Jesper Nordenberg, scala-...@googlegroups.com, scala-l...@googlegroups.com, scala...@googlegroups.com
      On Sun, Apr 1, 2012 at 03:31, Jesper Nordenberg <mega...@yahoo.com> wrote:

      > Daniel Sobral skrev 2012-03-31 16:01:
      >
      >> Given that Odersky has written the basis to the current javac
      >> (http://www.scala-lang.org/node/241), I think it should be taken for
      >> granted that there _are_ inherent reasons for the huge performance
      >> gap.
      >
      >
      > Well, javac wasn't written in Scala. :)

      Nor was scalac 1.x. :-)

      Jason Zaugg

      unread,
      Apr 1, 2012, 12:43:12 PM4/1/12
      to Nils Kilden-Pedersen, Aleksey Nikiforov, martin odersky, scala...@googlegroups.com

      Your memory is fine. scalac doesn't parallelize compilation of files.
      SBT, however, can parallelize compilation of independent sub-projects.

      -jason

      Christian Szegedy

      unread,
      Apr 1, 2012, 9:42:12 PM4/1/12
      to scala-user
      On Mar 31, 7:01 am, Daniel Sobral <dcsob...@gmail.com> wrote:
      > Given that Odersky has written the basis to the current javac
      > (http://www.scala-lang.org/node/241), I think it should be taken for
      > granted that there _are_ inherent reasons for the huge performance
      > gap.
      >
      >

      OCaml is a language of similar complexity to scala and it compiles
      much
      quicker (>10K lines/sec). It may compile even faster than java.

      I can imagine that namespace separation plays the most fundamental
      role here:
      both java and scala partitions its namespace on compilation-unit
      boundaries.
      When done right, it does not seem to be inflexible: Even in java,
      classes
      are often used as a scoping mechanism allowing to put several classes
      into the
      same compilation unit. Similarly to ocaml, where each source file
      opens a new
      "module" that can contain submodules, etc.

      So it may be the case that allowing different source files
      contributing to the same
      component without any separation was a fundamentally flawed design
      decision,
      resulting in inherent or hard to solve performance problems.

      BTW, very similar problems arise for C++ which has similar performance
      issues.
      Although by the same logic C should be also slow, it relatively fast
      speed is due
      to the relatively limited information exchange between different
      compilation units.
      Still, even for C: this has an adverse impact on the speed of linking,
      which should
      be considered as the final stage of the compilation process.

      Daniel Sobral

      unread,
      Apr 1, 2012, 9:58:29 PM4/1/12
      to Christian Szegedy, scala-user
      On Sun, Apr 1, 2012 at 22:42, Christian Szegedy
      <christia...@gmail.com> wrote:
      > On Mar 31, 7:01 am, Daniel Sobral <dcsob...@gmail.com> wrote:
      >> Given that Odersky has written the basis to the current javac
      >> (http://www.scala-lang.org/node/241), I think it should be taken for
      >> granted that there _are_ inherent reasons for the huge performance
      >> gap.
      >>
      >>
      >
      > OCaml is a language of similar complexity to scala and it compiles
      > much
      > quicker (>10K lines/sec). It may compile even faster than java.

      Similar complexity? OCaml's type system isn't even turing complete!

      Christian Szegedy

      unread,
      Apr 1, 2012, 10:29:39 PM4/1/12
      to Daniel Sobral, scala-user
      This is not an argument. One could easily introduce a Turing complete type
      system which is much less complex that both Scala and Ocaml.

      In fact, I think it requires in general more features from the type system to
      be expressive enough but not Turing complete at the same time.



      Naftoli Gugenheim

      unread,
      Apr 2, 2012, 12:37:18 AM4/2/12
      to Jason Zaugg, Nils Kilden-Pedersen, Aleksey Nikiforov, martin odersky, scala...@googlegroups.com
      If I'm not mistaken, it's not only independent subprojects, but also independent trees of dependencies. That is, if you modify A.scala, and it affects B.scala, C.scala, and D.scala, where B and C are interdependent, but not directly related to D, then D will compile in parallel to B+C.


      -jason

      Jeppe Nejsum Madsen

      unread,
      Apr 2, 2012, 5:13:24 AM4/2/12
      to martin odersky, Francois, Clint Gilbert, scala...@googlegroups.com
      On Sat, Mar 31, 2012 at 9:41 AM, martin odersky <martin....@epfl.ch> wrote:
      > We are currently working on making incremental compilation available
      > to maven and other build tools. Hopefully that will be a constructive
      > solution to the issue.

      For me, the compilation speed of a single file is not that bad, it's
      the recompilation of the "solution" that is problematic. The
      turnaround time for making a simple change in a single file often
      causes a compilation of most files in the project.

      I think one of the main issues is that even with SBT, the incremental
      compilation compiles too much (some details here
      https://groups.google.com/forum/#!msg/scala-ide-user/h00tWkpw1Lk/ZFZW4XUc1QoJ)

      Add to that use of e.g. the cake pattern, suddenly everything depends
      on the cake and even minor changes causes full recompilation.

      I'm seriously thinking of adding e.g. Guice just to break this compile
      time dependency.

      /Jeppe

      Lars Hupel

      unread,
      Apr 2, 2012, 8:44:58 AM4/2/12
      to scala...@googlegroups.com
      > OCaml is a language of similar complexity to scala [...]

      No. OCaml lacks dependent types and a proper kind system. Also,
      subtyping is not ubiquitous in common OCaml programs. It may have row
      polymorphism, but we have something similar in Scala, too.

      Guillaume Yziquel

      unread,
      Apr 2, 2012, 10:25:46 AM4/2/12
      to Lars Hupel, scala...@googlegroups.com
      Le Monday 02 Apr 2012 � 14:44:58 (+0200), Lars Hupel a �crit :

      > > OCaml is a language of similar complexity to scala [...]
      >
      > No. OCaml lacks dependent types

      Well, with first-class modules, you do have some ways to encode GADTs,
      so I'd be quite cautious with these kind of assertions.

      > and a proper kind system. Also, subtyping is not ubiquitous in common
      > OCaml programs. It may have row polymorphism, but we have something
      > similar in Scala, too.

      Generally I believe Scala's type system to be richer, however, the real
      differences in type systems, aside from usability issues, are in my
      opinion:

      -1- lack of any implicits in OCaml (which comprises Scala's version of
      type classes)

      -2- less pervasive inference in Scala (although type inference
      frameworks such as ML^F may in the long run bridge the two approaches,
      as far as I understand it).

      Anyway, to me, it's comparing apples and oranges.

      --
      Guillaume Yziquel
      Crossing-Tech
      Parc Scientifique EPFL

      Daniel Sobral

      unread,
      Apr 2, 2012, 11:09:32 AM4/2/12
      to Christian Szegedy, scala-user
      On Sun, Apr 1, 2012 at 23:29, Christian Szegedy

      <christia...@gmail.com> wrote:
      > On Sun, Apr 1, 2012 at 6:58 PM, Daniel Sobral <dcso...@gmail.com> wrote:
      >>
      >> On Sun, Apr 1, 2012 at 22:42, Christian Szegedy
      >> <christia...@gmail.com> wrote:
      >> > On Mar 31, 7:01 am, Daniel Sobral <dcsob...@gmail.com> wrote:
      >> >> Given that Odersky has written the basis to the current javac
      >> >> (http://www.scala-lang.org/node/241), I think it should be taken for
      >> >> granted that there _are_ inherent reasons for the huge performance
      >> >> gap.
      >> >>
      >> >>
      >> >
      >> > OCaml is a language of similar complexity to scala and it compiles
      >> > much
      >> > quicker (>10K lines/sec). It may compile even faster than java.
      >>
      >> Similar complexity? OCaml's type system isn't even turing complete!
      >>
      >> --
      >> Daniel C. Sobral
      >>
      >> I travel to the future all the time.
      >>
      > This is not an argument. One could easily introduce a Turing complete type
      > system which is much less complex that both Scala and Ocaml.

      It isn't? I means parsing a single line can take arbitrarily long,
      which is certainly not the case for any rational non-turing complete
      type system.

      For example, parsing line 42 of https://gist.github.com/66925#L42
      involves producing the answer to the Hanoi Towers problem with four
      discs. And while that is a gross example of abuse of the type system,
      the mechanism used in it is very common, and can easily produce long
      searches to solve.

      >
      > In fact, I think it requires in general more features from the type system
      > to
      > be expressive enough but not Turing complete at the same time.

      --

      Clint Gilbert

      unread,
      Apr 2, 2012, 11:34:17 AM4/2/12
      to scala...@googlegroups.com
      This is exactly one of the reasons I didn't find SBT to be a big
      improvement. I routinely saw SBT rebuild most of my project when I
      changed one file. I admit my code wasn't the best-designed in the
      world, but it was written in a DI manner, with coupling generally
      minimized.

      So maybe I could reorganize my code to compile faster, but I don't
      want to have to know enough about the compiler to know what kinds of
      designs and source files it can compile at a reasonable pace.

      Christian Szegedy

      unread,
      Apr 3, 2012, 12:04:47 AM4/3/12
      to scala-user
      I think you are confuse two completely different issues:

      1) The run-time complexity of type checking
      2) The (informal) complexity of language features

      (BTW it is not the "parsing" of line 42 that takes long, but its type
      checking.)

      In both languages you can have exponentially slow few-liners that are
      exponentially slow to type-check in the input size.



      On Apr 2, 8:09 am, Daniel Sobral <dcsob...@gmail.com> wrote:
      > On Sun, Apr 1, 2012 at 23:29, Christian Szegedy
      >
      >
      >
      >
      >
      >
      >
      >
      >
      > <christian.szeg...@gmail.com> wrote:
      > > On Sun, Apr 1, 2012 at 6:58 PM, Daniel Sobral <dcsob...@gmail.com> wrote:
      >
      > >> On Sun, Apr 1, 2012 at 22:42, Christian Szegedy
      > >> <christian.szeg...@gmail.com> wrote:
      > >> > On Mar 31, 7:01 am, Daniel Sobral <dcsob...@gmail.com> wrote:
      > >> >> Given that Odersky has written the basis to the current javac
      > >> >> (http://www.scala-lang.org/node/241), I think it should be taken for
      > >> >> granted that there _are_ inherent reasons for the huge performance
      > >> >> gap.
      >
      > >> > OCaml is a language of similar complexity to scala and it compiles
      > >> > much
      > >> > quicker (>10K lines/sec). It may compile even faster than java.
      >
      > >> Similar complexity? OCaml's type system isn't even turing complete!
      >
      > >> --
      > >> Daniel C. Sobral
      >
      > >> I travel to the future all the time.
      >
      > > This is not an argument. One could easily introduce a Turing complete type
      > > system which is much less complex that both Scala and Ocaml.
      >
      > It isn't? I means parsing a single line can take arbitrarily long,
      > which is certainly not the case for any rational non-turing complete
      > type system.
      >
      > For example, parsing line 42 ofhttps://gist.github.com/66925#L42

      Daniel Sobral

      unread,
      Apr 3, 2012, 8:41:56 AM4/3/12
      to Christian Szegedy, scala-user
      On Tue, Apr 3, 2012 at 01:04, Christian Szegedy
      <christia...@gmail.com> wrote:
      > I think you are confuse two completely different issues:

      The issue, is, from the first e-mail, "1. Compile time is super slow". So:

      > 1) The run-time complexity of type checking
      > 2) The (informal) complexity of language features

      You said that "OCaml is a language of similar complexity to scala and
      it compiles much quicker (>10K lines/sec)." I'm showing an example of
      how compiling a single line can take arbitrarily long, a feature which
      is directly derived from the fact that Scala's type system is turing
      complete.

      >
      > (BTW it is not the "parsing" of line 42 that takes long, but its type
      > checking.)

      I misused "parsing" to mean everything from lexing to code generation.
      I thought of using "typer", but the culprit here is implicit
      resolution, which might be done at the typer phase, but is not what
      people usually think of when they think "type checking".

      > In both languages you can have exponentially slow few-liners that are
      > exponentially slow to type-check in the input size.

      So look at the program again. It's speed is not related to input size:
      I can make it slower or faster without changes to the input size.
      That's what I mean when I say "arbitrary".

      Christian Szegedy

      unread,
      Apr 3, 2012, 3:35:47 PM4/3/12
      to Daniel Sobral, scala-user
      I agree. It is clear that worst case type checking runtime is
      very bad for both languages:
      - Potentially hanging for scala.
      - Running for days on less than 60 characters for Ocaml. (or any
      other functional programming language that uses Hindley-Miller
      type inference for a basis.)

      Still I think that the practically relevant question is the running time for
      practical every day programs without too much (that is: deep) logic
      encoded in the types.

      OCaml is a good example for a language where you pay the price
      for type inference and type checking only where it affects you even
      with its quite feature rich type system.

      The real interesting question what is to blame for the speed difference:
      - scala's implicits?
      - potentially rich interaction between compilation units in scala
      - Differences in typical programming style and standard libraries.
        (It is arguable that case classes in scala come with a relatively
        heavy baggage compared to the quite lightweight implementation
        of variants in OCaml.)
      - something else inherent to the language
      - or just the current implementation is not as efficient as it could be.

      I don't know the answer, but I think that OCaml is an interesting data
      point for comparison.

      Raoul Duke

      unread,
      Apr 3, 2012, 3:41:41 PM4/3/12
      to scala-user
      >> >> For example, parsing line 42 ofhttps://gist.github.com/66925#L42
      >> >> involves producing the answer to the Hanoi Towers problem with four
      >> >> discs. And while that is a gross example of abuse of the type system,
      >> >> the mechanism used in it is very common, and can easily produce long
      >> >> searches to solve.

      hey, i think the 1980's are calling and want their c++ metaprogramming back!

      Daniel Sobral

      unread,
      Apr 3, 2012, 4:29:07 PM4/3/12
      to Raoul Duke, scala-user

      You couldn't do metaprogramming with standard C++ back in the 80s.
      You'd have to go to the 90s for that.

      Besides, metaprogramming is boring. Metaprogramming is what Forth
      programmers call "writing code". Metacompiling, now, that's fun.

      Raoul Duke

      unread,
      Apr 3, 2012, 4:37:29 PM4/3/12
      to scala-user
      On Tue, Apr 3, 2012 at 1:29 PM, Daniel Sobral <dcso...@gmail.com> wrote:
      > Besides, metaprogramming is boring. Metaprogramming is what Forth
      > programmers call "writing code". Metacompiling, now, that's fun.

      yeah, and Forth doesn't get adopted by enterprise, either.

      but for some strange reason, C++ is a shoe-in.

      what is up with that?

      Erik Osheim

      unread,
      Apr 3, 2012, 4:42:27 PM4/3/12
      to Daniel Sobral, Raoul Duke, scala-user
      On Tue, Apr 03, 2012 at 05:29:07PM -0300, Daniel Sobral wrote:
      > Besides, metaprogramming is boring. Metaprogramming is what Forth
      > programmers call "writing code". Metacompiling, now, that's fun.

      If you're implying that writing FORTH is boring then I would have to
      disagree. I can think of other adjectives, but never boring.

      -- Erik

      Daniel Sobral

      unread,
      Apr 3, 2012, 4:50:05 PM4/3/12
      to Erik Osheim, Raoul Duke, scala-user

      Geoffrey S. Knauth

      unread,
      Apr 3, 2012, 4:56:07 PM4/3/12
      to Erik Osheim, scala-user
      On Apr 3, 2012, at 16:42 , Erik Osheim wrote:

      > If [...] implying that writing FORTH is boring then I would have to


      > disagree. I can think of other adjectives, but never boring.

      Boring not FORTH is. FORTH was actually used in the enterprise, sort of—the grand enterprise to boldly go where no one has gone before.

      http://solarsystem.nasa.gov/galileo/

      "Galileo plunged into Jupiter's crushing atmosphere on Sept. 21, 2003. The spacecraft was deliberately destroyed to protect one of its own discoveries - a possible ocean beneath the icy crust of the moon Europa."

      http://www.flownet.com/gat/jpl-lisp.html

      "... in 1993 I used MCL to help generate a code patch for the Gallileo magnetometer. The magnetometer had an RCA1802 processor, 2k each of RAM and ROM, and was programmed in Forth using a development system that ran on a long-since-decommissioned Apple II. The instrument had developed a bad memory byte right in the middle of the code. The code needed to be patched to not use this bad byte. The magnetometer team had originally estimated that resurrecting the development environment and generating the code patch would take so long that they were not even going to attempt it. Using Lisp I wrote from scratch a Forth development environment for the instrument (including a simulator for the hardware) and used it to generate the patch." [Ron Garret]

      Eduardo M. Cavalcanti

      unread,
      Apr 3, 2012, 5:08:21 PM4/3/12
      to scala...@googlegroups.com
      Hi,
      I have been using Scala for 2 years now. I only can say thanks very much
      for everyone involved in its development and improvement. Scala has made
      HUGE, really, really HUGE improvements in quality, performance,
      documention, features. It has evolved and has not lost a single bit of
      consistency. It is really amazing.
      Thanks very much.

      On 30/03/2012 10:48, amulya rattan wrote:
      > Cool developers,
      >
      > I must preface this by saying I absolutely love Scala and frameworks
      > that are built on it. I wanna marry them all, move to south of France
      > and start a band named "Scalalala", dont ask why. But there are some
      > serious _serious_ __serious__ problems with it that are definitely
      > avoiding a fast adoption of Scala everywhere.
      >
      > On the face of it, comparison of Scala and Java leaves no doubt that
      > one should make a good investment in the future and that future is
      > Scala. One reads a Scala book, scribbles few lines of code on REPL and
      > goes, "This is so cool! I'll go tomorrow and introduce my team to some
      > of this awesomeness". Learning/Understanding its intricate Type system
      > feels like an acid trip, and a good one at that. One comes out a
      > better man. Implicits are yummy. Traits are simple and powerful.
      > Introduction of Typeclasses from Haskell is more than welcome. Merging
      > of OO and FP to seamlessly transition from one style to other is neat.
      > Frameworks like Scalaz can be mind-bending, requiring a lot of effort
      > and diligence, but you know if you apply yourself, you can understand
      > it. And dont even get me started about Akka, its a work of art.
      >
      > Wow, so many positives!! So why isn't everyone investing their
      > resources into it?? Here are few silly/rebuttable reasons/arguments:
      >
      > 1. Some people are just adverse to change: And you can't do anything
      > about it. They want to continue living in null land and so be it.
      > 2. Some people are just BIG java fans and don't want to use anything
      > else: To them Java, like many other languages, is like a religion and
      > they don't wanna convert. Java is turing-complete and you can do
      > everything in it, just more verbosely and less runtime-safe-ly.
      > 3. Some people make effort but fallback on "Scala is too complex" cuz
      > they read it is somewhere: Well this has come up many times before and
      > doesn't need any more argumenting. For my 2 cents, Complex is just
      > intertwining of a lot of simple things. Plus since most programmers
      > are engineers and hence smart people, one would expect them not to shy
      > away from complex but embrace it, break it down and see what sort of
      > scary organism is living down there. However, thats not how the world
      > works. And there is a flip-side to it, since some people would love to
      > understand the "hairy" concepts but are busy otherwise.
      >
      > However, here are few arguments that are legitimate and definitely
      > something that Scala team and contributors, perhaps need to
      > contemplate on:
      >
      > 1. Compile time is super slow - And yes, running a seperate server
      > especially for compilation doesn't really help a lot. It doesn't. And
      > its not a single incident. Bigger your code base is in scala, worse
      > are the times. I am afraid to pitch Scala for our products, since I am
      > sure he'll show me this http://xkcd.com/303/ I thought we were out of
      > C++ compilation era. I am sure scala compiler has to perform a lot of
      > magic to convert to optimized bytecode and go through all its 25
      > elaborate stages. However, instead of working on all other cool
      > features added to Scala for now, couldn't one release be dedicated to
      > making compiler faster. Or am I just asking for too much? Also:
      >
      > Recently Play framework moved to 2.0 with a very aggressive
      > introduction of Scala in it and the response has been pretty much
      > violent. If you have been following play's mailing list, you'd know
      > what I am talking about. First comes the "scala is a nice language but
      > too complex" argument which looses Play a lot of territory with early
      > adopters moving to Django. Next people who are trying to use Scala are
      > faced with horrible compile times in DEV mode. Here is a guy with
      > similar issue of how using Play with Scala took all the speed away:
      > https://groups.google.com/forum/#!msg/play-framework/AcZs8GXNWUc/IanbqC-c-MkJ
      > <https://groups.google.com/forum/#%21msg/play-framework/AcZs8GXNWUc/IanbqC-c-MkJ>
      >
      > Now this guy at least tried using scala, but failed not because scala
      > is complex but because Scala has very slow compilation. And he
      > presents a _very_ legitimate problem that other organizations are
      > facing or will face if they started using scala unless Scala team does
      > something about it.
      >
      > 2. Cryptic errors - Everyone has seen them so I don't need to elaborate.
      >
      > 3. Those run time stack traces just leave me hanging
      >
      > These three arguments are as legitimate as anything can be and as far
      > as I know, every Scala developer has seen them. And they are all
      > compiler-magic centric. And while Scala is all about about
      > compiler-magic, does it leave no scope for optimization or errors that
      > are more _day-to-day_ developer friendly than _rockstar_ developer
      > friendly. Not everyone who is a programmer is super-smart and can go
      > around those issues using their black-magic skills. A day-to-day
      > developer in an organization wants to get through a task as fast as
      > possible because that's what business end demands off of him/her.
      > Software engineering doesn't exist on its own, there is a big layer of
      > business on top of it and biggest business need is smallest
      > Time_To_Market.
      >
      > If Scala needs to take off as it aspires to or as it should, it needs
      > to cater to day-to-day developer. The developer who just wants a fast
      > compile time, a error explaining what he did wrong or what could go
      > wrong and a stack trace explaining what kind of muck he created. Not
      > every organization would want to have a Typesafe subscription to
      > ensure they avoid scala related issues. They expect their everyday
      > developer to take care of language-related issues.
      >
      > Now I spent my time writing this to extract what the other developers
      > in scala feel about this and to know if Scala people do consider this
      > a major issue and have a plan to solve it very soon or dont know of
      > any existing solution to it or simply dont give a muck about it.
      >
      > Expecting some responses.
      >
      > ~Amulya

      Eric Kolotyluk

      unread,
      Apr 3, 2012, 5:59:41 PM4/3/12
      to scala...@googlegroups.com
      OMG - there are other people in the world who know Forth!

      What telescope did you work at ;-)

      Cheers, Eric

      Reply all
      Reply to author
      Forward
      0 new messages