Reasons that cause migration away from Scala - feedback

[ExScala :(]

I have been enjoying Scala for many years so far. Unfortunately, I have reached the point at which I am forced to abandon it and move away to Java (8). I am not seeking help and am not trying to stop people from starting with or using Scala. Far from it - I am in specific circumstances that make some of the issues much greater than for most. I am posting this in hope that this will help shape the future of Scala in a way that will eliminate the problems I am facing today.

There are many nagging issues with Scala development, just like there are with any other language. A reader of this post should NOT take this as a sole source of information deciding whether or not to use Scala. Scala the language is great and has helped us develop a lot of functionality very quickly, with a very small team. Even starting with it is not that hard because Java developers could start writing Java-like Scala code with very little effort and then evolve from there. Not the best Scala code, but it works. I will be focusing only on the negative and only on Scala (I could list many negatives for anything else).

To get it over with with the main reason I will begin with it - binary compatibility. With Scala there is no guarantee that (binary) code compiled with one version of Scala compiler will work with the code compiled with another version. There has been a lot of focus on organizing and providing fresh versions of libraries to alleviate this problem but that only works in specific cases. In my case we exposed a large multi-component/service platform-like product (partly written in Scala) that accepts plugins also written in Scala by other companies. If a customer buys a plugin built with one version of Scala (say 2.11) and deploys it in our platform that was to be upgraded to 2.12 or later, it won't work (well, there is some chance that it may, but it is not guaranteed). For this reason we can't let plugins to be developed in Scala. This is a real problem. I wish Scala finds a way to standardize and finalize how code is generated so that binary compatibility issues never occur again - i.e. newer platforms should always be able to run old (byte)code.

Scala-Java interop is great but not great enough to have the platform still be written in Scala due to many "small" things LIKE differences in vararg calls and the fact that we can't really give ScalaDoc API reference to Java developers and cannot link between the two. This linking is absolutely required, as we do need to link from JavaDoc of "downstream" components back to ScalaDoc and vice versa for complete, readable documentation.

The IDE toolset is not great enough. Note that my experience is limited to Eclipse + Scala IDE. There were or still are various issues with how to ensure matching Scala libraries between what is in Scala IDE and what is to be used in production and various freezing, debugging and downright "not producing any bytecode but not showing any errors either" oddities. Even though Scala and Java source can be mixed, something we had to rely on, every so often the IDE would enter an odd state with tons of errors that seem to make no sense and do not help really find what the cause is. Sometimes a clean-build helped (such things should not have happened), but not always.

We even hit some Scala's own deficiencies/inconsistencies over time, such as http://stackoverflow.com/questions/5676301/scala-currying-and-default-arguments. Developers get scared of the gazillion symbolic operators that all look alike and are hard to remember. Yes, once you memorize them they are useful in writing more compact code, but that code only looks more cryptic and unreadable to everyone else. Saving a few characters here-and-there is not always the best thing, especially when it comes to identifiers. 

Finally, I was not lucky enough to have all developers in our large team(s) either respect or be willing to accept and learn Scala. This has nothing to do with Scala the language, anything different from what they are used to (Java) would face the same problem, but it was nevertheless a continuous source of nagging and complaining that I had to spend/waste time fighting. Sometimes these complaints had a point. For example, generic type system is better than Java's, but it is also more strict (for a reason). However, it continues to miss features that can make it useful for very complex cases and the strictness complicates Java interop. Worse yet, separating some sub-expressions into individual 'vals' seemed to eliminate spurious errors/bugs when these are embedded inside more complex expressions - even though they should not really. This turned learning Scala not only into learning how to use its features but also how to avoid the mines in its 'dark zone'. Could these have been my and other developers' 'user errors'. Improbably but possibly - and that does not lessen the problem. A language that only a few can use is a language that only a few can use, whatever the reason is.

P.S. We haven't moved away from Scala yet. We just decided that we have no choice. I have yet to figure out how to rewrite a ton of Scala (2.11) code into four tons of Java so that I can start using Java 8 as well. I can't move to 2.12 either as it isn't out yet and see no light at the end of that tunnel... and then there is that binary compatibility issue as well, to reappear again with Dotty :(

[Rex Kerr]

If one has extremely stringent requirements that everything be binary compatible forever, Java is about the best one can do.  It's generally advisable to pick the language(s) that solve your key problems, and there are some key problems that Java can solve better than anything.

But Java-level compatibility comes at a high cost: the language is relatively stagnant, and old pitfalls and awkwardness stay in the language and library essentially forever.

If you view Scala not as another Java but as a Java library, the current level of binary compatibility (source is pretty good even between major versions, binary breaks) is not terribly different than you expect from other major Java libraries.  For instance, remember when Jackson went from 1.x to 2.x?  Breakage all over the place.  What if some libraries require Jackson 1 and some 2?  Aaaaah!

I think it's reasonable to expect within-version binary compatibility for Scala, but if your requirements are really so high for stability, well, you have to decide which projects are going to dictate the allowed versions, and force everything else to follow along with the same dependencies.  You have this problem within 100% Java.  (Possibly somewhat ameliorated by modules in Java 9, but a lot of the time that won't work either because it doesn't magically translate data structures for you.)

On Fri, Sep 2, 2016 at 7:10 AM, ExScala :( <he...@susnjar.net> wrote:

In my case we exposed a large multi-component/service platform-like product (partly written in Scala) that accepts plugins also written in Scala by other companies. If a customer buys a plugin built with one version of Scala (say 2.11) and deploys it in our platform that was to be upgraded to 2.12 or later, it won't work (well, there is some chance that it may, but it is not guaranteed). For this reason we can't let plugins to be developed in Scala. This is a real problem.

In particular, this is a real problem, but it is a problem with *every* library.  There is an inherent tension between "I want awesome cool new stuff that isn't overwhelmingly intricate and makes my life easier" and "I want the old broken stupid stuff to work forever".  We've already got Java playing the conservative side of that game; I think for Scala to play the other side is completely reasonable.  Of course _gratuitous_ breaking changes are unwanted.  But if they solve real problems (avoid unsoundness errors, reduce boilerplate, circumvent common gotchas, etc.) that were present in the old library, there's a place for them.

You can always just not upgrade!

Also, "We are going to sell this plugin to you for $$$$ but we are too lazy to upgrade it when new versions come out" is not the kind of proposition that I would be interested in.  Any system where I'm buying plugins that aren't even maintained to work with recent versions would send me running.  Maybe in your ecosystem this is considered perfectly okay.  I think it's on the wrong side of history, though.

 

Scala-Java interop is great but not great enough to have the platform still be written in Scala due to many "small" things LIKE differences in vararg calls and the fact that we can't really give ScalaDoc API reference to Java developers and cannot link between the two. This linking is absolutely required, as we do need to link from JavaDoc of "downstream" components back to ScalaDoc and vice versa for complete, readable documentation.

That's a good point.  Anyone could write a tool that makes this happen, though.  Javadoc is perfectly happy with in-line URLs, so you "just" have to know where to point.

It's a non-negligible effort, I admit.

 

The IDE toolset is not great enough. Note that my experience is limited to Eclipse + Scala IDE. There were or still are various issues with how to ensure matching Scala libraries between what is in Scala IDE and what is to be used in production and various freezing, debugging and downright "not producing any bytecode but not showing any errors either" oddities. Even though Scala and Java source can be mixed, something we had to rely on, every so often the IDE would enter an odd state with tons of errors that seem to make no sense and do not help really find what the cause is. Sometimes a clean-build helped (such things should not have happened), but not always.

Despite IDEs being a lot better now than a couple of years ago, there's still room for improvement, I agree.

Personally I write my Scala in a way that makes IDEs practically pointless, but that's not practical for a lot of people I guess.

 

We even hit some Scala's own deficiencies/inconsistencies over time, such as http://stackoverflow.com/questions/5676301/scala-currying-and-default-arguments. Developers get scared of the gazillion symbolic operators that all look alike and are hard to remember. Yes, once you memorize them they are useful in writing more compact code, but that code only looks more cryptic and unreadable to everyone else. Saving a few characters here-and-there is not always the best thing, especially when it comes to identifiers. 

Yes, but that's a style issue.  Pretty much everyone agrees with this now, and doesn't write crazy stuff with operators even though the language itself doesn't forbid you from doing it.  It's a problem with some old libraries, and I suppose with some new teams that haven't had enough C++ experience and/or haven't read Li Haoyi's blog on naming.  Mostly the operators are sparing and really pull their weight in the standard library (in that e.g. with the barest of training, a ++ b is a lot faster to read and understand than a.concat(b)).

 

Finally, I was not lucky enough to have all developers in our large team(s) either respect or be willing to accept and learn Scala. This has nothing to do with Scala the language, anything different from what they are used to (Java) would face the same problem, but it was nevertheless a continuous source of nagging and complaining that I had to spend/waste time fighting. Sometimes these complaints had a point. For example, generic type system is better than Java's, but it is also more strict (for a reason). However, it continues to miss features that can make it useful for very complex cases and the strictness complicates Java interop.

(Not enough details here to comment.  Interop with generics and typeclasses is a pain, I agree.)

 

Worse yet, separating some sub-expressions into individual 'vals' seemed to eliminate spurious errors/bugs when these are embedded inside more complex expressions - even though they should not really.

When this happens it's generally because the type inference has too many possibilities to figure out what to do, but by creating a val you signal that "no, that's it, figure out something solid here".  Very rarely this will change a compiling expression into another compiling expression that does something different (because the break enforces the opposite choice than default priorities would suggest); more often, it's just the difference between compiling and not.

It is frustrating that type inference isn't perfect, but it's a very complex problem in a language as flexible as Scala.  It does have to be worked around sometimes, but either explicitly specifying the types or introducing intermediate results generally does the trick.

 

This turned learning Scala not only into learning how to use its features but also how to avoid the mines in its 'dark zone'. Could these have been my and other developers' 'user errors'. Improbably but possibly - and that does not lessen the problem. A language that only a few can use is a language that only a few can use, whatever the reason is.

The reason does make a difference as to how actionable this is.

"We won't learn X, because X is not Java."   Can't do much.  Wait for a better Java.  (Maybe they won't use that either?)

"We won't learn X, because we try to use X like Java and run into problems."  Can do something: teach!

"We won't learn X, because we try to use X like Java and run into problems, and we don't want to learn anything."  Can't do much.

Scala can't solve social problems of Scala not being Java.

 

P.S. We haven't moved away from Scala yet. We just decided that we have no choice. I have yet to figure out how to rewrite a ton of Scala (2.11) code into four tons of Java so that I can start using Java 8 as well. I can't move to 2.12 either as it isn't out yet and see no light at the end of that tunnel.

2.12 is really close, actually.  It will be out much sooner than you can write a ton of Scala into Java.

You may also find that you can switch to Scala 2.12 and then just not develop new stuff in Scala now that you realize that stability trumps everything else in your case, but leave the legacy code as Scala and treat 2.12 as a legacy library.

  --Rex

[Erik Bruchez]

This might be too late for you, but my understanding is that the new TASTY intermediary format for Scala will (help) solve binary compatibility issues, including between Scala 2.x and Dotty. It's a bit early to tell if it will work in practice, but in theory it sounds very, very good:

    https://docs.google.com/document/d/1Wp86JKpRxyWTqUU39H40ZdXOlacTNs20aTj7anZLQDw/edit

-Erik

On Fri, Sep 2, 2016 at 7:10 AM, ExScala :( <he...@susnjar.net> wrote:

--
You received this message because you are subscribed to the Google Groups "scala-debate" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-debate+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Naftoli Gugenheim]

On Fri, Sep 2, 2016, 10:10 AM ExScala :( <he...@susnjar.net> wrote:

I have been enjoying Scala for many years so far. Unfortunately, I have reached the point at which I am forced to abandon it and move away to Java (8). I am not seeking help and am not trying to stop people from starting with or using Scala. Far from it - I am in specific circumstances that make some of the issues much greater than for most. I am posting this in hope that this will help shape the future of Scala in a way that will eliminate the problems I am facing today.

There are many nagging issues with Scala development, just like there are with any other language. A reader of this post should NOT take this as a sole source of information deciding whether or not to use Scala. Scala the language is great and has helped us develop a lot of functionality very quickly, with a very small team. Even starting with it is not that hard because Java developers could start writing Java-like Scala code with very little effort and then evolve from there. Not the best Scala code, but it works. I will be focusing only on the negative and only on Scala (I could list many negatives for anything else).

To get it over with with the main reason I will begin with it - binary compatibility. With Scala there is no guarantee that (binary) code compiled with one version of Scala compiler will work with the code compiled with another version. There has been a lot of focus on organizing and providing fresh versions of libraries to alleviate this problem but that only works in specific cases. In my case we exposed a large multi-component/service platform-like product (partly written in Scala) that accepts plugins also written in Scala by other companies.

Perhaps the plugins could be submitted in source format, and have your plugin system compile them?

If a customer buys a plugin built with one version of Scala (say 2.11) and deploys it in our platform that was to be upgraded to 2.12 or later, it won't work (well, there is some chance that it may, but it is not guaranteed).

Couldn't you just stay on one scala version? How is that worse than Java?

For this reason we can't let plugins to be developed in Scala.

So could you have the plugins written in Java (or some scripting language) and keep your own codebase in Scala?

This is a real problem. I wish Scala finds a way to standardize and finalize how code is generated so that binary compatibility issues never occur again - i.e. newer platforms should always be able to run old (byte)code.

I

Scala-Java interop is great but not great enough to have the platform still be written in Scala due to many "small" things LIKE differences in vararg calls and the fact that we can't really give ScalaDoc API reference to Java developers and cannot link between the two. This linking is absolutely required, as we do need to link from JavaDoc of "downstream" components back to ScalaDoc and vice versa for complete, readable documentation.

The IDE toolset is not great enough. Note that my experience is limited to Eclipse + Scala IDE. There were or still are various issues with how to ensure matching Scala libraries between what is in Scala IDE and what is to be used in production and various freezing, debugging and downright "not producing any bytecode but not showing any errors either" oddities. Even though Scala and Java source can be mixed, something we had to rely on, every so often the IDE would enter an odd state with tons of errors that seem to make no sense and do not help really find what the cause is. Sometimes a clean-build helped (such things should not have happened), but not always.

We even hit some Scala's own deficiencies/inconsistencies over time, such as http://stackoverflow.com/questions/5676301/scala-currying-and-default-arguments. Developers get scared of the gazillion symbolic operators that all look alike and are hard to remember. Yes, once you memorize them they are useful in writing more compact code, but that code only looks more cryptic and unreadable to everyone else. Saving a few characters here-and-there is not always the best thing, especially when it comes to identifiers. 

Finally, I was not lucky enough to have all developers in our large team(s) either respect or be willing to accept and learn Scala. This has nothing to do with Scala the language, anything different from what they are used to (Java) would face the same problem, but it was nevertheless a continuous source of nagging and complaining that I had to spend/waste time fighting. Sometimes these complaints had a point. For example, generic type system is better than Java's, but it is also more strict (for a reason). However, it continues to miss features that can make it useful for very complex cases and the strictness complicates Java interop. Worse yet, separating some sub-expressions into individual 'vals' seemed to eliminate spurious errors/bugs when these are embedded inside more complex expressions - even though they should not really. This turned learning Scala not only into learning how to use its features but also how to avoid the mines in its 'dark zone'. Could these have been my and other developers' 'user errors'. Improbably but possibly - and that does not lessen the problem. A language that only a few can use is a language that only a few can use, whatever the reason is.

P.S. We haven't moved away from Scala yet. We just decided that we have no choice. I have yet to figure out how to rewrite a ton of Scala (2.11) code into four tons of Java so that I can start using Java 8 as well. I can't move to 2.12 either as it isn't out yet and see no light at the end of that tunnel... and then there is that binary compatibility issue as well, to reappear again with Dotty :(

--
You received this message because you are subscribed to the Google Groups "scala-debate" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-debate...@googlegroups.com.

[ExScala :(]

Re:

But Java-level compatibility comes at a high cost: the language is relatively stagnant, and old pitfalls and awkwardness stay in the language and library essentially forever.

Yes, I understand reasons for this. I am not really blaming anyone. But this is not a new problem and there have been solutions made to problems like these, better or worse, without compromising source level functionality, only performance and/or binary size.

Re:

If you view Scala not as another Java but as a Java library, the current level of binary compatibility (source is pretty good even between major versions, binary breaks) is not terribly different than you expect from other major Java libraries.  For instance, remember when Jackson went from 1.x to 2.x?  Breakage all over the place.  What if some libraries require Jackson 1 and some 2?  Aaaaah!

That is correct. No denying here. That does not change this reality.

Re:

I think it's reasonable to expect within-version binary compatibility for Scala, but if your requirements are really so high for stability, well, you have to decide which projects are going to dictate the allowed versions, and force everything else to follow along with the same dependencies.  You have this problem within 100% Java.  (Possibly somewhat ameliorated by modules in Java 9, but a lot of the time that won't work either because it doesn't magically translate data structures for you.)

Java has maintained substantial/impressive levels of backward compatibility. One can still run very old Java bytecode in newest JVMs and make calls to these from the fresh new Java 8 code.

Re:

You can always just not upgrade!

That isn't really a choice, especially at the point we are beginning to publish this feature, yet be stuck on the outgoing version of the language. The support for languages themselves is a part of the product and support for Java 8, for example, is very much wanted. We need to support that. We want to keep up-to-date and be competitive with Scala too, but we can't.

Re:

Also, "We are going to sell this plugin to you for $$$$ but we are too lazy to upgrade it when new versions come out" is not the kind of proposition that I would be interested in.  Any system where I'm buying plugins that aren't even maintained to work with recent versions would send me running.  Maybe in your ecosystem this is considered perfectly okay.  I think it's on the wrong side of history, though.

This isn't about us only. And it isn't about the other companies that are building plugins. It is also about the customers. For us it is unacceptable that we force customers to worry about problems like these and we can't help them either as we cannot ask plugin developers for their source code either.

Re:

That's a good point.  Anyone could write a tool that makes this happen, though.  Javadoc is perfectly happy with in-line URLs, so you "just" have to know where to point.
It's a non-negligible effort, I admit.

Exactly. And it would be an ongoing effort.

Re:

Scala can't solve social problems of Scala not being Java.

No disagreement here.

Re:

This might be too late for you, but my understanding is that the new TASTY intermediary format for Scala will (help) solve binary compatibility issues, including between Scala 2.x and Dotty. It's a bit early to tell if it will work in practice, but in theory it sounds very, very good:

Yes, that looks like it would help, if but it would have to be out NOW. 

Re:

Perhaps the plugins could be submitted in source format, and have your plugin system compile them?

Thought of that, but we are not really in the open source world. Very much the opposite.

Re:

Couldn't you just stay on one scala version? How is that worse than Java?

We can't. Because we don't stay with one Java version. We are pretty up to date with it, behind only because we could not upgrade Scala to match Java on time.

Re:

So could you have the plugins written in Java (or some scripting language) and keep your own codebase in Scala?

I explained that in the original post. Not feasible.

[Viktor Klang]

Also, for javadoc integration, see: https://www.lightbend.com/blog/genjavadoc-generate-true-javadoc-for-scala-pr

--
You received this message because you are subscribed to the Google Groups "scala-debate" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-debate+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[ExScala :(]

One extra bit re:

Also, "We are going to sell this plugin to you for $$$$ but we are too lazy to upgrade it when new versions come out" is not the kind of proposition that I would be interested in.  Any system where I'm buying plugins that aren't even maintained to work with recent versions would send me running.  Maybe in your ecosystem this is considered perfectly okay.  I think it's on the wrong side of history, though.

This isn't about us only. And it isn't about the other companies that are building plugins. It is also about the customers. For us it is unacceptable that we force customers to worry about problems like these and we can't help them either as we cannot ask plugin developers for their source code either. 

Our customers DO have a lot of work to do with this EVEN if both us and the plugin developer companies do their homework and supply automated solutions for this. Customer acceptance testing, certification, extremely high level of security needs demand this for every new bit of code, even if it is said to come from the same source. Don't ignore this.

[ExScala :(]

Re:

Also, for javadoc integration, see: https://www.lightbend.com/blog/genjavadoc-generate-true-javadoc-for-scala-pr

That may actually be EXTREMELY useful.

 

[Alex Cruise]

You raise interesting points, and there are serious tradeoffs to be made here. 

Going off-topic a bit... 

One thing I'd suggest, taking a longer-term perspective, is when we're trying to foster a developer ecosystem, we should try really hard not to lock people in to any particular platform/runtime. In cases like these my recommendation is that we try to come up with a small number of *out-of-process* API/SPIs that future components, written in literally any programming language, could participate in. The form that these APIs should take is a matter for thoughtful, case-by-case analysis--maybe it's stdin/stdout, maybe REST services, maybe Kafka/0MQ... Not all choices here will ever be equally convenient--we won't want to maintain a huge number of SDKs. The point is to make it *possible* to implement new modules in anything the customer/partner likes.

I realize you have a significant investment in your current system, and making major changes is inherently painful and costly. But you're already contemplating one major and costly change, why not both? ;)

-0xe1a

--

[ExScala :(]

That isn't far off-topic. 

We try very hard to expose maintainable API/SPIs. The thing is that Scala makes a lot of things so much easier/shorter (well, it did so far) that we wanted to pass that benefit on to plugin developers too. There we face the binary compatibility issue but also the Scala/JavaDoc (that seems to have a solution now, as per Viktor Klang, thanks!) and Java interop issues. Now, out-of-process (remote) APIs/SPIs are a different thing. We have those too, but they can *never* be nearly as functional and performant as we need them to be, not to mention that they complicate things a lot. As you point out, that would also be a huge investment as well.

I am now trying to think of a phased approach to eliminating Scala, which has some chance of never completing :), thanks to Viktor's post. Plugins would not have the benefit of using Scala but we may still be able to use it in the platform...

To unsubscribe from this group and stop receiving emails from it, send an email to scala-debate...@googlegroups.com.

[Simon Ochsenreither]

Thanks for your feedback!

Not exactly what you are asking for, but support for JavaDoc has been added to ScalaDoc for 2.12: https://github.com/scala/scala/pull/5246

[Simon Ochsenreither]

Just a comment on this:

Yes, I understand reasons for this. I am not really blaming anyone. But this is not a new problem and there have been solutions made to problems like these, better or worse, without compromising source level functionality, only performance and/or binary size.

 

Java has maintained substantial/impressive levels of backward compatibility. One can still run very old Java bytecode in newest JVMs and make calls to these from the fresh new Java 8 code.

This is true, but it's much easier if you not only own the language but also the runtime. Then you can just add more workarounds to the runtime to support existing code.

It's often, but not always visible how much cruft has been accumulated in the runtime to facilitate that. The recent work on Valhalla shows how much all the accumulated stuff has impact on the development of improvements to the language/runtime. It feels like 80% of the effort on Valhalla is working around things that should have stopped existing 10 years ago.

In the end we will end up with yet another feature that adds even more cruft to the language, making future improvements even harder. Scala's approach has always been a bit different: We try to pay down the debt on past mistakes to make sure that the language can evolve in the best way we can imagine. Scala is in good company here with most other languages (C#, JavaScript, Python, Ruby, Haskell, ...), but it might not be the right choice for everyone.

(Although it seems Oracle has decided to give up on compatibility a bit and started deprecating and removing things starting with Java 9...)

[ExScala :(]

Re:

This is true, but it's much easier if you not only own the language but also the runtime. Then you can just add more workarounds to the runtime to support existing code.

Yes, I am very much aware of that. I used to make my own languages and runtimes (a very long time ago), so I know how this works. However, I also know that there are ways to work around those issues, at the cosr of some performance. Think of all the languages that compiled directly to native code, for example. Or of so/dll libraries. One can establish set ways to make calls and interactions and decide to not improve them over time in binary (yep, that's the cost). Another approach is to have newer platforms expose multiple ways to invoke same things - some for old code, others for new. I know this is a huge simplification compared to what Scala has to do, but it is just to get the point across. A bit of work could be deferred all the way to the Scala-specific class loader, for example, as well.

So there is a tradeoff between stability and performance really. I do not accept the belief in things being impossible to accomplish.

[Matthew Pocock]

Seamless scala/java interop is difficult. It is very easy to develop a nice scala api that can't be implemented or used from java.

Re plugging, if you want to have things work across version boundaries, you need to sandbox your classloaders. Run each plugin within a classloader environment with the version of scala it expects. Then have the plugin container manage bridging this back into your core application.

--
You received this message because you are subscribed to the Google Groups "scala-debate" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-debate+unsubscribe@googlegroups.com.

[Steven Marcus]

I've been using Scala since 2.8, and personally find it makes me more productive.

At one time I tried to use "scala everywhere" but found, the hard way, that it complicates life too much.

Binary version incompatibility is one reason I don't see Scala as a platform for projects of any significant size, especially when more than one team is involved.

These days we've gone back to Java8/Spring as the platform.

Any particular component/microservice is free to use Scala if it makes sense in that particular context, but Scala is not allowed to "leak" out.

Simple components stay simpler without Scala, especially since Java8.

Some components benefit from the richer language features provided by Scala.

Things have got much simpler since we've adopted this approach.

Friction created and time-wasted on build tools, IDEs, interop and others has been reduced.

So the takeaway I'd like to propose is that Scala is great when you use it in the correct context.

Some projects may benefit from Scala everywhere, but now that the world is shifting towards micro-services I think you get a big-win by adopting a platform with a broader more stable base.

Even after Scala 2.12 is released I don't see this general approach changing.

We are simply hoping for better Java8 interop.

Slightly off-topic: I think the approach of re-writing the (java) world in scala terms makes no sense commercially.

And the approach of creating a Java api within a Scala api (Play, Akka) doesn't work very well either. There's always the poor-relation syndrome to deal with.

From a distance I'd say that trying to create a "scala-only" ecosystem hasn't worked out the way anyone had hoped.

A big opportunity was lost by de-emphasising Java interop and long-term stability/compatibility.

 

My 2c. Thanks for reading.

 

[Rex Kerr]

On Sun, Sep 4, 2016 at 5:16 PM, Steven Marcus <steven...@gmail.com> wrote:

I've been using Scala since 2.8, and personally find it makes me more productive.

[...]
 

And the approach of creating a Java api within a Scala api (Play, Akka) doesn't work very well either. There's always the poor-relation syndrome to deal with.

Is there any other choice but for Java to be a poor-relation, if Scala is good at making you more productive?

 

A big opportunity was lost by de-emphasising Java interop and long-term stability/compatibility.

We can argue the stability point, but do you have any measures in mind that would substantially improve Java interop without giving up on a substantial part of "makes me more productive"?  What was the opportunity that was missed?

For example, some of the things that Scala has that really help my productivity are implicits (especially typeclasses), extension methods, and declaration-site variance.  All of these pose problems for Java interop because Java has a worse or absent way for doing the same thing.

Also, Java 8 adds features that are like those that Scala has had for years, but which were not implemented using Scala's battle-tested approach.  What is the opportunity here to get both interop and compatibility?

I understand the point that a single platform makes things easier, but this doesn't point the way to any particular opportunity except "just use Java".

On the other hand, if you have concrete realizable suggestions, they could still perhaps be implemented.

  --Rex

[Steven Marcus]

I don't disagree that Scala starting on top of Java8 would be radically different than the Scala we have that was started on Java5/6.

And that Scala still offers a richer language for problem solving and expressive implementations.

My slant on this is purely from a pragmatic commercial development perspective: use a broad based stable platform but feel free to adopt the best implementation tool in an isolated context.

So in a multi-project / multi-team situation, you have your broad-based stable platform as the LCD: Java, and possibly Spring, interfaces and classes. The impl of any component could be Scala, using all the language features that makes sense in a localised context.

You can combine jars from multiple projects carefully, with risk of binary versioning issues arising every 18 months or so.

But, these days, assembling a monolith is not in favour. So associated binary versioning issues are reduced by going with a micro-services architecture -- a way to realise more isolated contexts.

So I don't think I'm saying "just use Java"?

As far as "lost opportunity": My recollection is that that the broader Java/Spring commercial dev community was not averse to Scala in the beginning. But that community did learn to avoid Scala after running into the issues that started off this thread. I think it's fair to say that Scala became seen as an impediment to agility and getting things done. This was confirmed by the tendency of the re-write the world in scala approach. I can't help but wonder how the commercial Java development world would look today if (then) Typesafe had fully embraced Spring and made Scala-support a priority in that eco-system. But perhaps all is well in the "scala-only" world and that I am too distant from it to voice an informed opinion. As I mentioned this part of my original reply is off-topic and is an expression of my long-term experience in trying to use Scala in mainstream commercial software development. 

 

[Rex Kerr]

On Sun, Sep 4, 2016 at 6:39 PM, Steven Marcus <steven...@gmail.com> wrote:

I don't disagree that Scala starting on top of Java8 would be radically different than the Scala we have that was started on Java5/6.

I don't think that's what I was saying.  Java is moving towards Scala.  The implementation details would have been different--but then they are going to be somewhat different in 2.12.

I was making a point about binary compatibility: Java is doing its own thing, and Scala has to change itself to maintain compatibility.  Playing both the interop *and* compatibility games at the same time, while maintaining substantially more features than Java (which are not explicitly supported by the JVM), is a really difficult task.

That doesn't mean that the pain of incompatibility is any less, but it does mean that there may be less opportunity to do things differently than one would hope.

 

But, these days, assembling a monolith is not in favour. So associated binary versioning issues are reduced by going with a micro-services architecture -- a way to realise more isolated contexts.

Agreed.

 

So I don't think I'm saying "just use Java"?

I didn't mean you were in general, just in the case where you need a common platform.

 

As far as "lost opportunity": My recollection is that that the broader Java/Spring commercial dev community was not averse to Scala in the beginning. But that community did learn to avoid Scala after running into the issues that started off this thread.

They are real issues.  To the extent that they can be ameliorated, it would be good to do so.  But I'm still struggling to see any _actual_ opportunity rather than a wish.

 

I can't help but wonder how the commercial Java development world would look today if (then) Typesafe had fully embraced Spring and made Scala-support a priority in that eco-system.

Dunno, but doesn't this fall afoul of the Java-gets-a-second-class-API problem?  That is, if Spring got a really first-class Scala API, then there'd be the same tension that there is with other libraries that support both, and which you highlighted as a problem?

Anyway, I think you hit the best solution earlier: microservices.  And there's still plenty of opportunity for that in whichever language.

  --Rex

[ExScala :(]

Re:

My 2c. Thanks for reading.

You're welcome :)

Re:

Slightly off-topic: I think the approach of re-writing the (java) world in scala terms makes no sense commercially.

And the approach of creating a Java api within a Scala api (Play, Akka) doesn't work very well either. There's always the poor-relation syndrome to deal with.

From a distance I'd say that trying to create a "scala-only" ecosystem hasn't worked out the way anyone had hoped.

A big opportunity was lost by de-emphasising Java interop and long-term stability/compatibility.

That has been mine/our experience too.

Re:

Is there any other choice but for Java to be a poor-relation, if Scala is good at making you more productive?

Well, Scala did make us more productive in certain/many respects, that is until we hit the wall. Now we are paying the toll. Right now I would say that *if* you can entirely isolate your Scala code from the rest of the world, and keep it Scala only, never have external code attempt to interop with it (Java *or* Scala), then Scala can make you very much so more productive. But if you start poking that protective bubble as we ended up doing you'll face trouble.

Re:

We can argue the stability point, but do you have any measures in mind that would substantially improve Java interop without giving up on a substantial part of "makes me more productive"?  What was the opportunity that was missed?

You don't need to lose any part of "makes me more productive" at all, yet you can improve interop. There can be another price to pay, such as language complexity and (slight) performance loss, but nothing else. Simple things would be sufficient:

1. Stabilize the bytecode, so that there is binary compatibility towards Java (we're talking about Java interop here, I'd want this with Scala too, but that is a separate topic).
2. Document the above, promote to specification.
3. Match as many Java concepts as possible so that we can use Scala to define and implement APIs and contracts for Java users. Things like vararg methods, enums, relaxed generics, ...

The opportunity that was missed, from my perspective, is to solidify Scala as *the* language to safely use on JVMs instead of Java.

Re:

Also, Java 8 adds features that are like those that Scala has had for years, but which were not implemented using Scala's battle-tested approach.  What is the opportunity here to get both interop and compatibility?

For example including *both* approaches, automatically picking the best available at either/both compile time and/or run time.

Re: anything to do with Spring...

That is a separate topic. I tend to stay as far way as possible from that atrociousness. It is one big antipattern with a huge following. Scala actually reduces the little glimmer of need for things like Spring.

Re: microservices.

Agreed but only to a point. It works up until when it no longer does. It is also not a solution at all, it is simply a way of saying "solve it and maintain it on your own in a different way" instead of relying on the tools (such as Scala). 

[Steven Marcus]

Just to clarify, your reply referenced both my reply and Rex's reply.

Re: anything to do with Spring...
That is a separate topic. I tend to stay as far way as possible from that atrociousness. It is one big antipattern with a huge following. Scala actually reduces the little glimmer of need for things like Spring.

You may, and I do, find Spring over-abstracted and obtuse in many areas, but it delivers great value to main stream commercial software development. Pivotal have succeeded in adapting, extending and re-positioning the framework and associated projects to ride the current micro-services wave. If I'm building a microservice that I want to keep running for more than a year, perhaps with staff changes during its lifetime, I don't know of a better solution.

Re: microservices.
Agreed but only to a point. It works up until when it no longer does. It is also not a solution at all, it is simply a way of saying "solve it and maintain it on your own in a different way" instead of relying on the tools (such as Scala).

Microservices place a greater burden on the deployment environment in exchange for simplifications gained from a smaller isolated context. Similar to the way that the JEE app server provides a complicated deployment environment on which to build your value-add. Microservices are a practical expression of the fact that small teams work better. But a small dysfunctional team will not necessarily produce a working microservice. But the failure should be smaller :)

[ExScala :(]

Microservices place a greater burden on the deployment environment in exchange for simplifications gained from a smaller isolated context. Similar to the way that the JEE app server provides a complicated deployment environment on which to build your value-add. Microservices are a practical expression of the fact that small teams work better. But a small dysfunctional team will not necessarily produce a working microservice. But the failure should be smaller :)

Yes, but that is the easy bit. I wasn't talking about that. The problem is in the glue between them. And "glue" is another word and/or a way for "interop" and "compatibility"...

[Naftoli Gugenheim]

Kind of off topic, but is there any major hurdle to using remote Akka actors with different deployables using different Scala versions (with the messages as simple case classes in a module that's cross-compiled)? So you have a pure Scala app (could you call it microservices?) in multiple Scala versions, with the interop implemented as actor messages.

On Mon, Sep 5, 2016, 12:31 AM ExScala :( <he...@susnjar.net> wrote:

Microservices place a greater burden on the deployment environment in exchange for simplifications gained from a smaller isolated context. Similar to the way that the JEE app server provides a complicated deployment environment on which to build your value-add. Microservices are a practical expression of the fact that small teams work better. But a small dysfunctional team will not necessarily produce a working microservice. But the failure should be smaller :)

Yes, but that is the easy bit. I wasn't talking about that. The problem is in the glue between them. And "glue" is another word and/or a way for "interop" and "compatibility"...

--

You received this message because you are subscribed to the Google Groups "scala-debate" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-debate...@googlegroups.com.

[Jasper-M]

1. Match as many Java concepts as possible so that we can use Scala to define and implement APIs and contracts for Java users. Things like vararg methods, enums, relaxed generics, ... 

There is an annotation that generates a java-varargs method: http://www.scala-lang.org/api/2.11.8/#scala.annotation.varargs

The enums thing is being worked on as well: https://github.com/scala/scala/pull/5352

Re: anything to do with Spring...

That is a separate topic. I tend to stay as far way as possible from that atrociousness. It is one big antipattern with a huge following. Scala actually reduces the little glimmer of need for things like Spring.

I can only agree here...

[Justin du coeur]

On Sun, Sep 4, 2016 at 8:16 PM, Steven Marcus <steven...@gmail.com> wrote:

I've been using Scala since 2.8, and personally find it makes me more productive.

At one time I tried to use "scala everywhere" but found, the hard way, that it complicates life too much.

I think this depends enormously on the structure and stage of the company, and where Scala fits into it.  My bootstrapped startup is only really *possible* because of a Scala-everywhere approach -- the improved productivity makes it possible to build something *insanely* ambitious with extremely limited resources.  I'm quite sure that the more traditional stack would have sunk the company before I started.

Indeed, for the first two years I was using JavaScript for my front end, and that was a disaster, dragging the rest of the project down.  Switching to Scala.js (and, importantly, Autowire) was a life-saver: the front end stopped being a bottleneck, and instead became relatively easy and productive, precisely between everything -- front end, back end, and the APIs between them -- were end-to-end strongly-typed.

There will be a cost to this, and I bought into it very consciously: it means that we're going to have to teach Scala to engineers that we hire.  But I believe that's the right choice in the long run -- it'll be a significant training expense and slower productivity at first, but I am pretty sure it will amortize over the first year of any engineer's employment, and be a net positive thereafter.  (And it probably means that we can't hire mediocre programmers, but that's fine: in my experience, no startup can afford weak programmers anyway.)

Of course, this is the viewpoint of a still-minuscule company, and I can understand that this isn't the right approach for every firm -- in particular, it's going to be challenging for any single group in a large company to go hardcore Scala if they have to work with other groups that are committed to Java.  (Or have to be providing a Java platform externally, as the original poster apparently does.)  But if you're in a position to commit the company to Scala, and go into that with your eyes open about the implications, I still think it's an excellent path to follow...

[Robert Kohlenberger]

Have thought of this myself.  The implementation has to be carefully thought through.  For example, the case class messages might themselves contain code as functions or methods.  This code would need to execute on the target Actor's vm.  If security is a concern, some means of validating or sandboxing the code might be needed.  If some Actor-based services are implemented in another language (say Java) some standard means of translating types and packing/unpacking case classes might be needed.  Personally, I would love to see some sort of simple, standard API like this that everyone could write to.  "Spring without the atrociousness."  Could Tyrpesafe get behind something like this?

[Seth Tisue]

On Friday, September 2, 2016 at 7:10:16 AM UTC-7, ExScala :( wrote:

I can't move to 2.12 either as it isn't out yet and see no light at the end of that tunnel

fwiw, Scala 2.12.0-RC1 has been built and will be formally announced soon; see pre-announcement at https://groups.google.com/d/msg/scala-internals/JslA9u20mow/H2t2XMi-EwAJ

Seth Tisue / Scala team / Lightbend, Inc.

[Naftoli Gugenheim]

On Mon, Sep 5, 2016 at 5:47 PM Robert Kohlenberger <kohl...@comcast.net> wrote:

Have thought of this myself.  The implementation has to be carefully thought through.  For example, the case class messages might themselves contain code as functions or methods.

They should definitely not have arguments of a function type. Functions aren't serializable.

If the case class has a method, the method is not transported over the wire. The receiving VM has whatever methods the class it deserializes as has.

 

This code would need to execute on the target Actor's vm.  If security is a concern, some means of validating or sandboxing the code might be needed.

Again, assuming you're not actually serializing code, the receiver either has it or doesn't.

None of this is new or specific to cross-version or cross-language interop.

 

If some Actor-based services are implemented in another language (say Java) some standard means of translating types

They can use the same type from both languages

 

and packing/unpacking case classes

That's regardless

 

might be needed.  Personally, I would love to see some sort of simple, standard API like this

What would this API abstract over? I mean it exists, it's called actors. What else do you want to achieve?

 

that everyone could write to.  "Spring without the atrociousness."

What is that a description of?

 

 Could Tyrpesafe get behind something like this?

Behind Akka? I think they already are... :)

Also don't forget Lagom.

 

On Sunday, September 4, 2016 at 10:41:35 PM UTC-7, nafg wrote:

Kind of off topic, but is there any major hurdle to using remote Akka actors with different deployables using different Scala versions (with the messages as simple case classes in a module that's cross-compiled)? So you have a pure Scala app (could you call it microservices?) in multiple Scala versions, with the interop implemented as actor messages.

--