The "Scala is too complex" argument

[Nicholas Faiz]

Hi,

As someone learning Scala and enjoying doing so, I'm curious about how to evaluate the 'Scala is too complex" argument I often hear. It ranges from the idea that Scala code is too hard to debug in production, due to comparing exception messages in bytecode that has gone through type erasure, to objections about the syntax and semantics. I actually carried the naive idea that it was too complex myself, before properly considering it. Companies such as Twitter and LinkedIn seem to be using it very effectively. It's hard to weigh up those industry stories with the bias I routinely encounter amongst a lot of programmers when I mention Scala.

Has anyone encountered serious production downtime due to Scala's complexity factor? How much substance is there to this sort of objection to Scala?

Cheers,

Nicholas

[Travis Dixon]

Ultimately the sort of argument you're talking about isn't one you can really "win" as part of a conversation.  

Best bet is for both parties to give it a shot and compare first hand experience.

I've seen programmers with primarily java and/or ruby experience become productive (not 100%, but a net positive) within a couple of months, however this was with a concerted effort in training / up skilling them from people who were already quite proficient.

Ken did a talk at a YOW! night not so long ago covering a bunch of teams at REA who had good Scala experiences, even the one who didn't have much/any existing scala experience reported it was hard going but they delivered their solution.

--
You received this message because you are subscribed to the Google Groups "Melbourne Scala User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-melb+...@googlegroups.com.
To post to this group, send email to scala...@googlegroups.com.
Visit this group at http://groups.google.com/group/scala-melb.
For more options, visit https://groups.google.com/d/optout.

[Nicholas Faiz]

Right, it seems the only way to resolve it is to invest enough time and effort learning it to release a small production worthy app and build firsthand experience. It's an odd burden of proof, though; Scala seems to carry it more than a lot of other languages. I wanted to resolve it without the learn and discovery phase myself, if possible. I was hoping, for example, that perhaps a previous version of Scala had had significant problems, which had had to be fixed, and when a lot of people complained about its complexity they were really referring to a past problem. 

I almost hesitated in posting this. I'm sure most of this list is tired of the complexity issue. It is a real factor to resolve, though, when investing time to learn the language and planning to use it for an application that will take however many hours to write.

Cheers,

Nicholas

[Travis Dixon]

> I wanted to resolve it without the learn and discovery phase myself, if possible.

If I told you python* was too complex and that you shouldn't take it on as a production app, how would you resolve my complaint?  

What is the difference if you swap python with Scala there?

*not picking on python, it's a great language that I enjoy

> I was hoping, for example, that perhaps a previous version of Scala had had significant problems

Sure, Scala and its ecosystem has improved dramatically since inception.  SBT is waaaay easier to use than it used to be, Typesafe has put a lot of support into a bunch of frameworks which has given them a bigger community, things like Dispatch, Play!, Spray, Akka, Spark, Slick, Scalaz all reduce the amount of problems you need to solve on your own.

Intellij and the Eclipse based IDE are both vastly improved compared to 5 years ago, which I suppose would also ease the learning curve.

I guess my feeling is that it's all what you'd expect for a language that's been around a while and had pretty good adoption

[Tony Morris]

I have had much experience with this "argument." It is not really an argument because it very rarely follows any discourse in relation to the original claim. It comes in other varieties, such as Functional Programming is too complex. There are others. You may have seen them.

My experience has consistently been that when queried on what precisely is "complex", it is really just a lack of familiarity with new concepts. Nobody, in my experience (or on reading articles), has ever actually pointed to anything that can reasonably be described as complex about Scala. That is not to say they don't exist; just that so few people actually point to them. Tools that simplify programming, albeit unfamiliar, can hardly be described as complex.

The way to make a constructive outcome of this scenario is to hold the claim to task. Once it has been clearly established that "complex" and "unfamiliar" are not synonyms, which is the biggest hurdle (over which I am handwaving for now), you move on to providing the tools for the complainant to accurately assess what is really happening (they are learning new, simple tools). Give them the vocabulary to make accurate, critical remarks about their observations.

If all these hurdles can be overcome, you have someone who was once saying ridiculous things, who is now equipped to realise that those things are indeed ridiculous. The next complainant who confuses the adjectives "complex" and "unfamiliar" now has two people to help them overcome their problems and anxieties. And so on...

Just because someone says something ridiculous, doesn't mean that they are doomed to forever say it. Help them. I have done this many times, in production software environments where other potentially difficult factors must also be also considered. Godspeed.

On Wed, Jun 24, 2015 at 11:34 AM, Nicholas Faiz <nichol...@gmail.com> wrote:

[Toby Corkindale]

For whatever it's worth.. I think the error messages emitted by the Scala compiler have improved since the early days.

Apart from that -- A "problem" with Scala is that it's possible for people to write complex code with it. As in, unnecessarily* complex (or just hard to read) code. There's a lot of the language that you can abuse to write code that's clever, and works, but is quite opaque to anyone who hasn't spent a while studying the code.

This isn't really a problem specific to Scala, as a language, but it seems like the Scala community has been much more accepting of it. I think that trend has declined over time, though. People who were initially excited about using crazy edge cases of features have got over it, maybe?

(I mean, just because you can define your entire app in a DSL entirely based on unprintable unicode code points as types, doesn't mean you *should* :)

(Although the concept of using zero-width-non-breaking-spaces to differentiate apparently identical things in order to confuse readers does amuse me))

But if you're using Scala at your company, you can set internal standards, in much the same way you would for any other language. Require sensible structures, comments, good naming, and so on.

-Toby

* In my humble opinion. I suspect the authors of such often think it was totally necessary.

[Nicholas Faiz]

Hey The Trav,

On Thursday, June 25, 2015 at 11:08:57 AM UTC+10, The Trav wrote:

> I wanted to resolve it without the learn and discovery phase myself, if possible.

If I told you python* was too complex and that you shouldn't take it on as a production app, how would you resolve my complaint?  

What is the difference if you swap python with Scala there?

Unless you are building firsthand experience I could point to industry perceptions of Python.

I could point to Google's successful use of it, or any number of other companies or academic institutions. I could probably ask questions to pinpoint what was supposedly complex as I have programmed commercially in Python. 

The main point is, Python doesn't have to resolve this ambiguity; Scala does. 

So, there's a lurking stigma or bias about Scala which each developer has to resolve at some point for themselves. Yesterday I discovered a Melbourne company that has made a decision not to use Scala, even though their overseas counterparts do. I don't hear about such problems with other languages very often but they seem to gather under the Scala label.

Which is problematic, for me, because I am very interested in continuing to learn Scala and envision using it. 

Whether it's a language I'm picking up for fun or whether it's a language I'll use commercially in the future I am more uncertain about it due to the complexity factor. It seems like the only course of action is to build firsthand experience and *not* rely on industry perception; which differs a lot depending on who you are talking to.

 

*not picking on python, it's a great language that I enjoy

Yup :)

 

> I was hoping, for example, that perhaps a previous version of Scala had had significant problems

Sure, Scala and its ecosystem has improved dramatically since inception.  SBT is waaaay easier to use than it used to be, Typesafe has put a lot of support into a bunch of frameworks which has given them a bigger community, things like Dispatch, Play!, Spray, Akka, Spark, Slick, Scalaz all reduce the amount of problems you need to solve on your own.

Intellij and the Eclipse based IDE are both vastly improved compared to 5 years ago, which I suppose would also ease the learning curve.

I guess my feeling is that it's all what you'd expect for a language that's been around a while and had pretty good adoption

Yeh, I'm partly hoping that now might be a very good time to learn Scala. There seems to be a lot of maturity gathered around it since I considered learning it in 2011.

Cheers,

Nick

[Andrew Conway]

I love Scala. I am very familiar with it. I still think that Scala is complex, and this is a somewhat valid criticism, but it is not the whole story. When you learn a language you learn the language, the libraries, the toolset, and how to work around deficiencies in the language (e.g. "Java Patterns"). The language itself is often the simplest of these.

6502 machine language is a very simple language. There are a small number of commands, each of which does a very simple thing. You could easily learn all of them in a day. There is no standard library to learn. This is not a good thing.

Scala has most of the features of Java, and many of the features of Haskell, and some of its own stuff. This makes it very powerful, but I would say that Scala is more complex than either of these languages.

Scala often gives multiple ways to solve the same problem. E.g.

  package a.b

or
 
  package a
  package b

which mean *almost* the same thing.

Multiple ways to solve the same problem mean you can use the one that is most natural and readable. Implicit arguments allow clear, concise code (eg List.max). But both can be abused to make unreadable code. As other responders have said, don't do it.

So I would change the argument - I would say that Scala is more complex than Java, but that is not the appropriate comparison. Scala+scala patterns is arguably simpler (and a lot less error prone) than Java+java patterns.

[Travis Dixon]

> I could point to Google's successful use of it, or any number of other companies or academic institutions.

Twitter, The Guardian, Walmart... https://www.typesafe.com/resources/case-studies-and-stories

> I could probably ask questions to pinpoint what was supposedly complex as I have programmed commercially in Python. 

This is probably the closest to what I'm saying... I accidentally picked something you already know, so how about ruby?  erlang?  smalltalk?  keep swapping languages until we come across one you don't know.  Alternatively, how would you have handled it before you got your first try at python?

> Yesterday I discovered a Melbourne company that has made a decision not to use Scala, even though their overseas counterparts do.

I'm a fan of allowing different teams to select their own languages.  I see this as healthy and in no way indicative of a problem with Scala.  

> It seems like the only course of action is to build firsthand experience and *not* rely on industry perception; which differs a lot depending on who you are talking to.

Certainly that's the course of action I would recommend in most if not all decisions you make as an engineer.  Perception is for brand and marketing to worry about. 

[Bryan Murphy]

I think if you want to program like a Java Developer then Scala is significantly less complex once you learn a few basics.  The case classes, collection library, type inferencing, more powerful Futures, avoidance of Null, preferance of immutability, etc, results in more concise, reliable and readable code.

The "problem" is that Scala offers a lot more avenues to achieve even more concise, powerful and reliable code and these come at the cost of understanding more concepts (eg Functional Programming which has over 50 years of intellectual property associated with it, type level programming ) and implementation techniques (eg use of implicits, macros, dependent types).  In some cases the implementation techniques stretch the capabiliities of the language and require a few tricks or careful use to get right.   This is seen as complexity but is a natural consequence of smart people trying to further the art of programming in new and interesting ways.

Another interesting outcome is that the infrastructure code to support a DSL can be quite complex but the users of the DSL can be very productive just by seeing a few examples (and hopefully by reading the documentation of the DSL which becomes more important) so you might need a few very experienced Scala programmers to support the infrastructure code whilst less experienced Scala programmers just use it.

For a given code base and group of programmers you need to choose the amount of "complexity" you are willing to tolerate - both in the code that is written by the team and the external libraries that are used.  There is another side to the complexity story and that is satisfaction of the programmers.  One reason Scala is becoming popular is that people get frustrated by limitations of languages like Java and are looking for ways to avoid boiler plate and create more testable, reliable code.  They don't mind a bit more complexity if it enables them to become better programmers.

If you have a small team of above average programmers then you can choose a higher threshold of complexity than a large group of average programmers - and one could argue that you need to if you want them to stick around.

Java 8 is a huge validation of Scala as it introduces some (but by no means all) of the benefits that Scala has enjoyed from the very beginning.  They have undeniably made Java more complex but I haven't heard anyone portraying that as a negative.  In fact I would say it was seen as necessary to avoid stagnation.

On a personal note I have been using Scala for a number of years and code that I used to find baffling is now more approachable but there is still Scala code out there I can't easily understand.  The code I don't understand doesn't daunt me, it just means I would be less inclined to write similar code myself (in 2015 and in my day job) or incorporate it from github/maven repo if it wasn't well supported.  It also is good to know there is still things to learn and knowledge to aspire to gain.

So I would say Scala is not "too complex" but it does enable levels of sophistication which need to be used wisely.

Bryan

[Nicholas Faiz]

Thanks for the response.

The big thing I wanted to understand was the implication of the complexity. No one is talking about production systems failing. It's mostly about the different levels the language can be used at and how they relate to the level of programmers engaging with it. 

It was mostly a sound check against some of the criticisms I've heard from sysadmins who bemoan Scala, etc..

Cheers,

Nicholas

[Travis Dixon]

sysadmins have close to zero impact from scala... it just looks like another jar to them.

however i have had sysadmins complain about not getting war files for their horrible JEE Application servers...  that's just as much a problem in pain java as scala though.

--

[Christopher Hunt]

To answer your original questions directly:

> Has anyone encountered serious production downtime due to Scala's complexity factor? 

I personally have not.

> How much substance is there to this sort of objection to Scala?

When I'm having these types of conversation I wheel out my mantra:

    I want to write less code, but more code of a higher quality.

Scala gets me there.

[Bernard Leach]

The number one tip I could offer any team picking up any new language is to make sure you have access to good experts in the language.  One or team team members that have experience with a language who are willing to spend time getting the team up to speed will make a massive difference.  On the other hand, working in a void will probably result in either poor code or a feeling complete disillusionment (and probably both!).  Experts in this case don't necessarily need to be experts in the strict sense, a good grasp of the language and a willingness to solve issues goes a long way.

[Tianhao Li]

http://underscore.io/blog/posts/2015/06/25/keeping-scala-simple.html

Haha, just read this article. 

[Tony Morris]

It is better to argue in terms of costs (complexity), for a benefit.

Scala has made some unwise choices here. For example, record .copy
method incurs all the penalties of a new syntax, but does not fully
exploit the best known practical solution at the time (store-based lenses).

Another is that for-comprehensions do not have a polymorphic type. All
the penalties of a new syntax, forgoing almost the entire potential
benefit, so that we can point to some superficial syntactic aesthetics
as the only benefit. It is a net lose-lose. These scenarios can
reasonably be described as "excessively complex."

What cannot be described as complex are new tools that deal with actual
complexity and achieve the goal of reducing complexity. I almost always
see this deployed in a discussion of complexity.

If anyone is going to argue that Scala is complex, it should be in these
terms. It almost never is though. This article compounds this problem.

On 26/06/15 09:54, Tianhao Li wrote:
> http://underscore.io/blog/posts/2015/06/25/keeping-scala-simple.html
>
>
> Haha, just read this article.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Melbourne Scala User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to scala-melb+...@googlegroups.com

> <mailto:scala-melb+...@googlegroups.com>.

> To post to this group, send email to scala...@googlegroups.com

> <mailto:scala...@googlegroups.com>.

[Tony Morris]

If I could modify this, just slightly.

Make sure you have experts in programming *concepts*, not programming
languages. Programming language merely allow the exploitation of sound,
fundamental programming concepts to some extent. Scala just happens to
be measured at, "some extent that is significantly further than the others."

You want programmers who know how to exploit these concepts and deploy
them for a benefit. The programming language is merely one tool of many
that may (or may not) be a vehicle to achieve this.

Never hire a "Expert Scala programmer."