[erlang-questions] The future of Erlang and BEAM

[Radek]

Hello Group,

it's my first post here, although I've been reading many interesting posts here for quite a long time.
Anyway, I posted a question on StackOverflow today with the same title (here). As I've written there, I don't want to start any flame-war, particularly because I'm actually an Erlang fan-boy :) but I'd like to know your opinion on the subject. Here's the original question:

-------
Some time ago I got seriously interested in Erlang (coming from C++/PHP/Java world) - and I've seen it has been successfuly used in the industry, by Ericsson, Facebook, Goldman Sachs, etc. So, I thought it would be a great platform to build high demanding apps, with low-latency profile, with a lot cleaner and nicer language than, for example, Java (for me).

But after "wow effect" has gone, I discovered that there are many high performance Java libraries that seem to resolve many problems that Erlang is theoretically best suited for (real-time, low-latency applications, concurrency, fault-tolerance, etc.). Moreover, it seems that there are things that, despite Erlang profile, are just not possible to achieve on BEAM (like LMAX Disruptor concurrent framework).

So the question arises: is Erlang still the best platform to build such demanding applications ? Wouldn't it be better if we stick to one, very mature (J)VM and try to make it even better than trying to achieve something similar with less resources available (size of OTP team vs. JVM team, supporters, etc) ? And is it possible at all to achieve this kind of performance and adoption with BEAM ?

And just to make things clear: I don't want to init a flame war here. I am just curious becouse I really like Erlang and I think it's a great platform and I'd like to invest time and effort to build real-life projects on it. But I'd just like to know what others might say about that and - if I'wrong - maybe someone could correct me.
-------

I hope we can discuss it since I think it would be valuable not only for me.

Greetings,
Radek

[Max Bourinov]

Hi Radek,

Before answering your question, could you please clarify to the list what is so special in "LMAX Disruptor"? Why it is impossible to implement it in Erlang? Does it really makes sense to use this technology? I asked those questions because I read about "LMAX Disruptor" while ago, but I didn't find anything special in it. Moreover for me is seems as an obvious thing in a shiny package.

About Java vs Erlang: After many years in Java world (banking, investing, transportation) I started to work with Erlang (including all guys in our company). Now all our software is written in Erlang/C. It handles huge payload, very easy to scale, in case of problems it heals itself automatically. It is very easy to develop and maintain software in Erlang. This is a true dream technology. We are really happy now. I don't see any Java use cases in our company in the future.

p.s. Of course Erlang is not designed for UI.

---

So, thanks again for Ericson company and all those great people that developed Erlang (I hope you read this mail. I am really proud to know you) and all of those people that contribute to Erlang/OTP. Thank you!

Best regards,

Max

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Ulf Wiger]

Radek,

My take on the LMAX Disruptor is that it would probably be hard to get the same raw throughput in Erlang. OTOH, for many products, this is not the clincher, and Erlang may still be the better option over time, since it tends to make it easier to make use of multicore or improvements in cluster technology in order to get the desired performance improvements.

In my own experience, this has been demonstrated many times. I have witnessed many different C++-based applications temporarily run circles around their Erlang-based counterparts, but eventually running into severe problems, either because they hit hard limits that stopped them from achieving the required performance on available hardware [1], or because changing requirements added a complexity that they were not designed to manage [2]. Another notable occasion was the shift to multicore, where most C++ and Java developers ended up facing a magnificent challenge, while we could ship Erlang-based systems on the new architectures with only minimal changes.

The thing that was missing from your list of things Erlang is that it shines for coordination problems. Erlang is sometimes called an *orchestration language*. This is where most of the competing solutions are often woefully inadequate. The key is where you get inputs from different sources, and must be able to handle events arriving in unexpected order. This is typical for control systems, and actually where the biggest complexity lies.

LMAX Disruptor, if I recall correctly, is essentially a processing pipeline, where each job is independent and processed to completion. Thus, there is no coordination complexity to speak of, and the main challenges are managing access to shared resources and achieving high processor utilization.

Granted, I have spent my whole career working with problems where Erlang is a near-perfect fit. YMMV depending on domain. :)

BR,

Ulf W

[1] In one case, the performance gains came in no small part from the fact that the C++ app was using static memory management. This worked fine, until the required capacity required more RAM than could be installed onto the available processor board. Rewriting the code to use dynamic memory management was deemed too risky and expensive, given the schedule.

[2] See e.g. my presentation, Death by Accidental Complexity, http://www.infoq.com/presentations/Death-by-Accidental-Complexity

[Radek]

# sending once more to show on mailing list

Hi Max,

Thank you for such a quick answer, it's nice to know there is someone to talk to

In posted question I indeed didn't exactly clarifiy what (probably) is so special about these (and others) libraries, so let me do it now.

1. LMAX Disruptor architecture is desribed in Martin Fowler's article here (long) - http://martinfowler.com/articles/lmax.html?t=1319912579 .
I got interested in it when I found it's being advertised as a "High Performance Inter-Thread Messaging Library" by which you can run huge number of concurrent tasks with minimal latency between each "thread" (nanoseconds order of magnitude). Of course, it's not "that simple" but still, the numbers are really amazing.

2. Java Chronicle library (https://github.com/peter-lawrey/Java-Chronicle) which describes itself as a "ultra low latency, high throughput, persisted, messaging and event driven in memory database". With their numbers (16 nano-seconds latency and throughputs of 5-20 million messages/record updates per second) I think oi could be easily used as a platform to build very low-latency message-queue or similar kind of middleware.

3. The well-known (it was mentioned on Erlang Factory) library Akka (http://akka.io/) which copies actor model, distribution and fault-tolerance from Erlang and again, looking at their numbers, tends to be faster than Erlang in terms of message passing latency (due to shared heap).

4. Kilim library (http://www.malhar.net/sriram/kilim/) which is somehow similar to akka, but just for message passing. Anyway, similar strength of zero-copy messaging.

And probably more, but that's what I remember the most.

One other thing is, that could be controversial, arising of Clojure (mainly it, but also other FP languages on JVM) which show that we could be faster just because of the Hot-Spot JIT compiler on JVM.

Of course, I could be all wrong, just because I'm rather a beginner (still) in terms of functional programming, Erlang, etc. But I wanted to know your opinion on the subject and maybe show me errors in the way I see this problem.

Greetings,
Radek

[Radek]

# the same as above, sorry for inconvenience

Hi Ulf,

thank you very much for answer.

I understand that Erlang has been created in purpose of managing coordination complexity and it does it's job well.
But, maybe I'm a bit of devil's advocate, let's say that we (by which I mean mostly OTP team because of their knowledge of Erlang intrinsics but others too) could follow similar direction that Clojure's creator Rick Hickey took. Which means, to build Erlang on JVM, using tools that we already have (like earlier mentioned Java libraries, etc) and optimize it for using with JIT.

Of course, it's not obvious that (despite being big effort) it would bring desired performance (although, it seems it would) but some advantages for sure, like ubiquity of JVM, wide gremium of supporters/maintainers, ease of deployment (even single .jar maybe), access to huge variety of other languages and libraries, and even some minor things like proper module managament, etc.

As I said before, I'm rather novice in Erlang world (still digging through Erlang and OTP in Action) so all above may be wrong/incomplete/etc. I had this idea some time ago so I finally took decision to write about my concerns here.

Greetings,
Radek

[Gleb Peregud]

On Sat, Feb 11, 2012 at 13:20, Radek <popros...@gmail.com> wrote:
> But, maybe I'm a bit of devil's advocate, let's say that we (by which I mean
> mostly OTP team because of their knowledge of Erlang intrinsics but others
> too) could follow similar direction that Clojure's creator Rick Hickey took.
> Which means, to build Erlang on JVM, using tools that we already have (like
> earlier mentioned Java libraries, etc) and optimize it for using with JIT.

http://erjang.org/ - actually there is such solution already :) Not
sure how mature it is, but afaik it implements most of stdlib and bifs

[Radek]

Hi Gleb,

Yes, I've seen it already. But I'm not sure if it's exactly what I meant.
I rather thought about making something like a framework which would
emphasize concurrency, fault-tolerance, etc. and implement Erlang as a
DSL just to operate on it.

Kresten (author of Erjang) did a great job but took (as I understand it;
I may be wrong) little different option and "translated" C code to Java,
made .beam files possible to be loaded on JVN thus making it capable of
running Erlang code.

By making a special framework and Erlang-as-DSL to operate on it, we
could then use normal Erlang tools without many downsides of them (for
example, we could always rely on messages because of no copy-on-send
penalty, use more data structures apart from just lists and tuples, and
so on).

Of course, I'd like to repeat that I'm not an Erlang-master nor skilled
in language design etc so my ideas can be just wrong.

Greetings,
Radek

W dniu 2012-02-11 13:37, Gleb Peregud pisze:

[Miles Fidelman]

Radek wrote:
>
> But after "wow effect" has gone, I discovered that there are many high
> performance Java libraries that seem to resolve many problems that
> Erlang is theoretically best suited for (real-time, low-latency
> applications, concurrency, fault-tolerance, etc.). Moreover, it seems
> that there are things that, despite Erlang profile, are just not
> possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
>
> So the question arises: is Erlang still the best platform to build
> such demanding applications ? Wouldn't it be better if we stick to
> one, very mature (J)VM and try to make it even better than trying to
> achieve something similar with less resources available (size of OTP
> team vs. JVM team, supporters, etc) ? And is it possible at all to
> achieve this kind of performance and adoption with BEAM ?
>

Two observations, from someone who has just started a migration to
Erlang - in service of a project with massive concurrency at its core:

1. Erlang and Java start from very different programming perspectives.
As far as I can tell, Erlang is still the only environment designed,
from day one, to support massive concurrency (processes as fundamental
vs. objects as fundamental).

2. Erlang is about 10 years older than Java, and used for building
production systems from its inception. There's a lot more real-world
experience packed into Erlang and OTP - particularly where massive
concurrency is concerned.

Seems to me that Erlang is far more mature than Java for the kinds of
problems and environments that are now emerging.

Now one might argue that there's a good academic exercise in writing yet
another new language, that learns from the lessons of Erlang, and maybe
cleans up some of the more obscure syntax, but such an exercise would
take a long time before yielding any kind of mature platform.

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Attila Rajmund Nohl]

2012/2/11, Radek <popros...@gmail.com>:
[...]

> But after "wow effect" has gone, I discovered that there are many high
> performance Java libraries that seem to resolve many problems that
> Erlang is theoretically best suited for (real-time, low-latency
> applications, concurrency, fault-tolerance, etc.). Moreover, it seems
> that there are things that, despite Erlang profile, are just not
> possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
>
> So the question arises: is Erlang still the best platform to build such
> demanding applications ? Wouldn't it be better if we stick to one, very
> mature (J)VM and try to make it even better than trying to achieve
> something similar with less resources available (size of OTP team vs.
> JVM team, supporters, etc) ? And is it possible at all to achieve this
> kind of performance and adoption with BEAM ?

The feature I really like in Erlang is it's debuggablity. I can
connect to the live system, I can get an Erlang shell on the VM and
can inspect the system real time. I can also load code (not only to
the live system), which makes development really fast.

[Radek]

Hi Miles,

Well, although I didn't mean to write a new language (I meant writing
Erlang-for-JVM, but rather as a DSL), I agree that Erlang IS very mature
and stable. And of course, it started as a industry-level language, so
it's designed to operate as such.
I was just wondering if and maybe we could benefit from being hosted on
JVM which is, although not THAT industrial-level, also capable of being
used in such conditions and is most popular VM on the world (so far).
The obvious advantage of using Erlang in such conditions would be (apart
from above) just not using verbose Java, which is big plus on it's own
:) (in my opinion).

So, if we could have both of the worlds, i.e. industry-level of OTP and
JVM ubiquity, that would be something huge I think.

Greetings,
Radek

W dniu 2012-02-11 14:41, Miles Fidelman pisze:

_______________________________________________

[Radek]

Hi Attila,

that's undoubtedly one of the very handy features of Erlang/OTP, which
has roots in it's real-time non-stop usage.
But of course I meant the bigger picture here; live debugging is handy
but can be (quite) easily reproduced on the JVM also (using OSGi framework).

Of course, Erlang has this from many years now which counts as a big
advantage, that's true.

Greetings,
Radek

W dniu 2012-02-11 14:58, Attila Rajmund Nohl pisze:

[Miles Fidelman]

Radek wrote:
> Hi Miles,
>
> Well, although I didn't mean to write a new language (I meant writing
> Erlang-for-JVM, but rather as a DSL), I agree that Erlang IS very
> mature and stable. And of course, it started as a industry-level
> language, so it's designed to operate as such.
> I was just wondering if and maybe we could benefit from being hosted
> on JVM which is, although not THAT industrial-level, also capable of
> being used in such conditions and is most popular VM on the world (so
> far). The obvious advantage of using Erlang in such conditions would
> be (apart from above) just not using verbose Java, which is big plus
> on it's own :) (in my opinion).
>
> So, if we could have both of the worlds, i.e. industry-level of OTP
> and JVM ubiquity, that would be something huge I think.
>

But why would I want to run massively concurrent software on a JVM?
What makes Erlang different are its underpinnings. Massive concurrency
on top of a JVM is just broken.

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Radek]

Well, I think it depends. It's true that using default concurrency
libraries is quite difficult and error prone but as I said before, thare
are others which mimics Erlang approach (and do it well, just looking at
numbers).

But, I'm not sure if it's a good idea to run anything massively
concurrent on JVM; that's one of reasons I've started this topic.

Greetings,
Radek

W dniu 2012-02-11 15:44, Miles Fidelman pisze:

> Radek wrote:
>> Hi Miles,
>>
>> Well, although I didn't mean to write a new language (I meant writing
>> Erlang-for-JVM, but rather as a DSL), I agree that Erlang IS very
>> mature and stable. And of course, it started as a industry-level
>> language, so it's designed to operate as such.
>> I was just wondering if and maybe we could benefit from being hosted
>> on JVM which is, although not THAT industrial-level, also capable of
>> being used in such conditions and is most popular VM on the world (so
>> far). The obvious advantage of using Erlang in such conditions would
>> be (apart from above) just not using verbose Java, which is big plus
>> on it's own :) (in my opinion).
>>
>> So, if we could have both of the worlds, i.e. industry-level of OTP
>> and JVM ubiquity, that would be something huge I think.
>>
>
> But why would I want to run massively concurrent software on a JVM?
> What makes Erlang different are its underpinnings. Massive
> concurrency on top of a JVM is just broken.
>
>
>

_______________________________________________

[Miles Fidelman]

Radek wrote:
> Well, I think it depends. It's true that using default concurrency
> libraries is quite difficult and error prone but as I said before,
> thare are others which mimics Erlang approach (and do it well, just
> looking at numbers).
>
> But, I'm not sure if it's a good idea to run anything massively
> concurrent on JVM; that's one of reasons I've started this topic.

The only way to run something massively concurrent on a JVM is to write
a collection of libraries that would look suspiciously like BEAM. So
why bother?

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Max Bourinov]

+1

[Tony Rogvall]

I do agree.

But on the other hand, there is nothing stopping people to implement the Erlang platform, and compete.

So stop talking about it and start hacking. Let the best implementation win ;)

/Tony

"Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"

[Radek]

Well, that's a good point. But it also shows that there is place for
improvement. Because, it would be really come in handy when one could
write correct, highly-concurrent, callback-free code (Erlang), using his
favourite libraries/frameworks (Java) on a platform that his company
uses (JVM).

Like I said, I like Erlang. This is the reason why I left Java etc, just
because it's not nice to write something that has to be concurrent and
(mainly at least) correct, not to mention development time (which is
usually shorter in Java). But I sometimes think that we could make one
step more and, as I would call it, join efforts (with JVM community, so
to speak).

Greetings,
Radek

W dniu 2012-02-11 17:09, Miles Fidelman pisze:

> Radek wrote:
>> Well, I think it depends. It's true that using default concurrency
>> libraries is quite difficult and error prone but as I said before,
>> thare are others which mimics Erlang approach (and do it well, just
>> looking at numbers).
>>
>> But, I'm not sure if it's a good idea to run anything massively
>> concurrent on JVM; that's one of reasons I've started this topic.
>
> The only way to run something massively concurrent on a JVM is to
> write a collection of libraries that would look suspiciously like
> BEAM. So why bother?
>
>

_______________________________________________

[Joe Armstrong]

I'll try and give a short answer to your question.

To start with you are comparing chalk with cheese - Erlang was
NOT designed to be a fast messaging passing language. It was designed
for building fault-tolerant applications - it wasn't designed
for speed it was designed for safety.

A lot of the message passing platforms you read about, and mention in
the same breath as Erlang were designed for speed
and not safety - so it's hardly surprising that is the absence of
errors these other platforms are faster than Erlang.

Where Erlang should excel is when things go wrong - when the
code is buggy and crashes, when the system is overloaded, when you
stress things in unexpected ways.

The Erlang view of the world is very different to the view of
the world used in most other languages.

Take a web server, for example. In the conventional view of the
world if there might be one web server per machine, the
web server might spawn an OS process for each incoming task
(if it wants safety) of it might handle all sessions in the same task
if it wants speed but no safety. It (the web server) uses OS processes
for reasons of safety - to isolate the user sessions from
each other - but this is very slow.

What happens if there is a rare
error in the web-server code itself - one that is very rarely
encountered. Does this affect *all* user sessions or only the
session where the error occurred? - My guess is that all sessions
will collapse if the web-server itself collapses.

How is a web-server in Erlang written? On the face of things it
looks like any other web sever - but an Erlang system that appears
from the outside to be handling (say) 2000 user sessions,
is NOT one web server with 2000 sessions - it is 2000 erlang
process (all independent) and each one is running ONE web server.

So in the Erlang case we have 2000 web servers, each running
one user session - in the conventional case we have one web
server running 2000 sessions.

Back to my question - what happens if there is a rare error
in the web server code itself which is triggered in one session?
In the Erlang case only one web server dies and all the rest
survive. In the case where all 2000 sessions are handled by the
faster server all sessions die.

So you choose - fast or safe? - If you want the fastest possible
solution Erlang is not the answer - BUT it will be unsafe.

Adding checks and recovery mechanism COSTS something.
There are no free lunches.

I often see messaging passing systems mentioned that are said
to have "Erlang like semantics" and are said to be faster.

But these are unsafe systems and Erlang like semantics means
*a lot more* than fast message passing.

Erlang is good at what it was designed to be good at - which was
building fault-tolerant soft real-time distributed systems with
dynamic code upgrade - I know of nothing that has remotely
similar semantics.

There are *many* systems with superficial similarities - but non
that I would trust to build fault-tolerant systems with.

Cheers

/Joe

[Matthew Evans]

Yes, I could write an essay on this one..

The other thing to remember about all these Java (and C++) libraries is that there are, frankly, so many of them. Each time you look at one it's a whole different set of semantics that need learning. With Erlang you have a single set of concurrency principles that need learning that are core to the language, and who's behavior is well understood. It makes for code that is easy to reason about, especially when things go wrong. I also worry that many of these Java libraries are only going to be supported until something shinier, newer or better comes along. I should add here that I have recently started learning Java, and quite like the language, especially compared to the effects that C++ has had on my hairline. ;-)

Having said that there is a valid criticism of Erlang in that it is often perceived to be too slow. As Joe said, you can't have all the features of resilience and reliability without some cost, but when trying to "sell" Erlang it's often the first argument against Erlang that I hear. The pointy haired managers look at things like the Alioth shootout and see Erlang perform poorly for most tests. Now, most Erlang users will correctly say "well, Erlang isn't designed for those types of highly serial tasks, use C, C++ or Java for that instead, and use Erlang where it makes sense". But this now means that a company needs to embrace 2 (perhaps more) languages to do a job. This isn't necessarily a bad thing, but can make for pain when hiring, training and retaining staff.

My point is that I don't think Erlang will ever be as fast as Java and certainly never as fast as C/C++, but I would like to see a greater focus on performance, perhaps get it into the top 10 languages for performance. I'm happy to see that work is been done WRT to JIT in the Erlang VM, and LLVM+HiPE, so maybe future releases will address those concerns.

I've also often wondered if it makes sense to implement some of the Kernel and STDLIB modules as BIFs instead of in Erlang. Would that improve things?

Matt

________________________________
> Date: Sat, 11 Feb 2012 13:20:54 +0100
> From: popros...@gmail.com
> To: u...@feuerlabs.com
> CC: erlang-q...@erlang.org
> Subject: Re: [erlang-questions] The future of Erlang and BEAM

[Miles Fidelman]

Joe,

I'll add one more thing - correct me if I'm wrong here (you are, after
all the expert :-)....

Joe Armstrong wrote:
> To start with you are comparing chalk with cheese - Erlang was
> NOT designed to be a fast messaging passing language. It was designed
> for building fault-tolerant applications - it wasn't designed
> for speed it was designed for safety.
>
>

Erlang also is really nice for building systems where you want to THINK
and organize your code in terms of concurrent processes. Simulation is
the application that comes to mind (based on my own experience).

If you want to simulate 10,000 tanks, on a simulated battlefield - my
first instinct is "each tank is a process." Turns out, that most (all?)
conventional simulators model each tank as an object, with a small
number of threads touching each object 20-40 times per second (a state
update thread, a display thread, etc.) Leads to really horrible
spaghetti code and systems that are really hard to change.

I found Erlang when I joined a simulation software house, after a
background in networking (where we spawn processes all the time).
Programmers kept telling me that "we can't spawn 10,000 processes, the
overhead will kill us" - which is true with C++ (what we were using) or
Java. I did some digging and found Erlang as a counter-example.

I guess what I draw from this is that, while Erlang might not be
blindingly fast for some applications, if I want to write code that
spawns 1000s of processes, it's a lot faster than anything else around.

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Radek]

Dear Group,

first of all, I am pretty amazed that my question started such a nice
discussion with interesting posts.
What amazes me even more, that (with full&great respect to the others)
even such known people like Joe Armstrong or Ulf Wiger found some time
to answer me directly. I think it's truly amazing :)

Thank you very much, all of you, for such interesting answers. Now I
think I understand what is the real difference between Erlang & rest of
the world. I really haven't thought about safety and reliability in the
first place. Now it seems to be logical for me that "raw performance"
isn't an only factor which I should consider.

Thank all of you very much, once more. Now I think I'm Erlang fab-boy
even more :)

Greetings,
Radek

W dniu 2012-02-11 19:48, Miles Fidelman pisze:

_______________________________________________

[Fred Hebert]

The best question to ask after this is "What's slow?", "What do you consider to be fast enough?", "What are your requirements?". You often (but not always) find out that they do not really know what they want, so they aim for the fastest overall thinking 'surely, I won't make a mistake there.' Do you need a 5 nanosecond message sending requirement when you're going to treat the message for 20 milliseconds anyway?

In some cases they do need something too fast for Erlang, or think Erlang is optimized for areas it isn't. Then being honest will be the best argument to prove you're giving sane recommendations and are not a fanboy. But before knowing what they really need, it is pretty hard to counter any argument.

There's no use fighting someone whose task to accomplish requires too much work or time in Erlang, but sometimes apprehensions come from not knowing the technology or the problem enough; that's where we can help.

[Tim Watson]

I'd really second this. I work with both Erlang and Java (alongside a few other bits and peices) on a daily basis and slowness isn't something I've really run into major difficulties with Erlang at all. If you take a look at that rather well documented case of the chat at Github who implemented an Erlang git daemon, but then found it to be too slow so abandoned it. A more experiences Erlang programmer came along and refactored the code for massive performance improvements.

It's true that sometimes you need raw throughput that Erlang can't provide and need to break out to C/C++ with a port/nif or whatever, but that's not so often the case. Also I prefer to 'pick the right tool for the job' and not every system has to be monolithic running on only one VM.

Cheers,

Tim

[Richard Carlsson]

On 2012-02-11 14:41, Miles Fidelman wrote:
> 2. Erlang is about 10 years older than Java,

Not really; the first prototype implementation of Erlang was made in
1987; it got its first users at Ericsson 1988-89, and was shown to the
public in 1990. Java was implemented 1991-1992 (under the name Oak);
they tried to market it to makers of embedded systems and such, but
no-one gave a damn, so they changed focus to Internet and multimedia and
got Java included in Netscape Navigator in 1995. So Erlang is younger by
only about 4 years.

/Richard

[Paul Davis]

On Sat, Feb 11, 2012 at 2:49 PM, Tim Watson <watson....@gmail.com> wrote:
> I'd really second this. I work with both Erlang and Java (alongside a few
> other bits and peices) on a daily basis and slowness isn't something I've
> really run into major difficulties with Erlang at all. If you take a look at
> that rather well documented case of the chat at Github who implemented an
> Erlang git daemon, but then found it to be too slow so abandoned it. A more
> experiences Erlang programmer came along and refactored the code for massive
> performance improvements.
>

Here's a link to the articles about refactoring egitd by Andrew
Thompson. I'd highly recommend them to anyone that's not read the
series.

http://andrew.hijacked.us/article/324/optimizing-egitd-introduction

[Richard Carlsson]

On 2012-02-11 22:24, Richard Carlsson wrote:
> younger by only about 4 years.

Older, I meant. Sorry.

[Jesper Louis Andersen]

On 2/11/12 11:34 AM, Radek wrote:

So the question arises: is Erlang still the best platform to build such demanding applications ? Wouldn't it be better if we stick to one, very mature (J)VM and try to make it even better than trying to achieve something similar with less resources available (size of OTP team vs. JVM team, supporters, etc) ? And is it possible at all to achieve this kind of performance and adoption with BEAM ?

Performance is but one of many aspects of writing a program. Usually performance is possible, but bounded by human time spent. That is, if you have limited time, say 3 days, to implement something then usually the solutions are very different performance wise. One of the distinct advantages of Erlang is how fast you can build up prototypes and make them run. This means you have ample time to sit back and tune your system for better performance. You have more time to change algorithms and tune for faster execution.

Another strength of Erlang is that programs can evolve. Adding another process with a certain responsibility or altering the internal implementation of a process is easy: the messages passed forces an API between your components. This yields a system where you can gradually build it up over time.

Erlang is a rarely seen triumvirate: fault-tolerance, distribution+message passing, and functional sequential programming. You often see other systems implementing one or two of those, but I have not yet seen all three implemented in another language. This places Erlang in a niche, and I think many people who do take time to learn the details enjoy the ideology of Erlang as a fabric for program construction.

-- 
Jesper Louis Andersen
  Erlang Solutions Ltd., Copenhagen, DK

[Radek]

I'm still a novice, but I can clearly say that Erlang is a nice language to play with; There is short period of time needed to convert the working prototype to full solution, since it's sometimes just the case of clearing/adding some API (because you already use gen_server, gen_fsm, ...) etc.

Moreover, I think that many times it's just simpler to use Erlang compared to Java/C++/... because you don't have such difference from model (when it's said "sending message from object A to object B) to implementation. In Erlang you actually send a message and don't have to bother to properly implement Observer pattern and all that stuff. So it's a huge advantage of both Erlang style on concurrency & functional programming.

I was just wondering if making it on JVM wouln't be a good idea, but I realized I didn't put some thing under consideration (like safety and truly fault-tolerance). And, it seems it's easier to make Erlang faster by researching for better algorithm/solution or sacrificing some performance for overall better (in terms of speed/reliability ratio) quality of product than trying to make things to work the same way on platform(-s) that wasn't designed for it in the first place.

Greetings,
Radek

W dniu 2012-02-11 23:06, Jesper Louis Andersen pisze:

[Michael Turner]

What probably matters more than age-measured-from-inception is the
sponsoring corporation's agenda. Java was open-sourced by Sun in part
because open-sourcing it furthered the goal of getting Java adopted as
a de facto industry standard; Erlang was open-sourced (years after
Java was) to escape getting strangled in its crib by Ericsson. By this
measure, Erlang might be considerably "younger" than Java, and
moreover suffering from a kind of "arrested development" -- because,
as useful as Erlang might be for Ericsson, it's not really considered
a *strategic* technology, whose spread furthers Ericsson's core
interests.

-michael turner

[Deryk Barker]

As a mere amateur in the erlang world - I've used it a few times in my
network programming course, probably bending the rules slightly to do so...

But I've been programming for a while (first program FORTRAN on an IBM
7090 while still at school in 1968, second program in BASIC on a PDP-8
while taking a course from Maurice Wilkes at Cambridge in 1970) and
there's one aspect in all of this thread that has either not been
mentioned or at least not made much of:

the fact that erlang is a functional language.

For years FP got a bad press, from the early days when the accusation
was inefficiency ("LISP programmers know the value of everything and the
cost of nothing") to the more recent criticisms that FP is very nice for
academics but won't work in the *real* world.

Erlang clearly blows these ideas out of the water and combines all of
the attributes everybody else has mentioned with those of developing in
a functional language, with all of its associated benefits, especially
of correctness.

Besides, I think java is a terribly *ugly* language...:-)

deryk
--

Deryk Barker, Computer Science Dept.
Camosun College, Victoria, BC, Canada

[Tim Watson]

Java is ugly and bloated to boot. The JVM though, is actually a pretty fantastic piece of engineering and if you dig into the guts of it, there's a lot that VM teams across the board can learn, especially around self tuning and diagnostic/instrumentation support.

I think the point about Michael made about mind-share is a very good one. Also the Java platform, for all its bloated complexity, is more mature in some respects. What I mean by that is, for developing typical business applications, Java (and god forgive me for saying it, the other big unmentionable corporate platform too) offers a number of things that are very beneficial and currently lacking from Erlang's ecosystem. For example, JDBC offers reliable and high performance connectivity to almost every database on the market. There are standardised APIs for working with things like web servers and this makes a big difference to the productivity of *ordinary programmers* who are the target audience for these kinds of development platforms. 

Having said that, I'd pick Erlang and fiddle with API inconsistencies and the near suicide inducing pain of ODBC on unix any day over Java.

Going back to what the OP was asking: One of the reasons I don't think that we'll see a competitive implementation of Erlang on the JVM any time soon is that the JVM was built, first and foremost, to host an imperative language. Now the appearance of Clojure does prove that you *can* implement a functional language on the JVM, but Erlang is not just a functional langauge/platform, it is also (as Joe has put it) "Concurrency Orientated", which is about the lightweight Process being a fundamental and first class concept. Java does not have micro/green threads and native threads have limitations. The JVM does not, at its core, support the notion of immutability that we have in Erlang, despite good support for copy-on-write semantics in some (limited) circumstances. 

The scala crowd can build 'actors' and 'message passing' into their libraries, but these concepts are not part of the JVM. Developers of Scala/Clojure code *can* make their variables, types (and so on) immutable, can implement a syntax for passing around chunks of code in order to imitate higher order functions, but these concepts are not (as of today) part of the JVM. The core concepts for the JVM (and the CLR) are Objects with 'mutable state', native (os level) threads and in Java's case, primitive types that are distinct from objects. You can't get away from those underlying concepts in reality, so 'actors' are Objects and 'message passing' is done by sharing (object) state on a heap and so on.  

One of the problems people *seem* to be trying to get away from by moving to the JVM is that of raw throughput. We might see a shared heap appear in BEAM some time in the future, for certain special cases, and maybe other optimisations for performance will occur. But as Joe and others have pointed out, Erlang was never built with this in mind. And yet I'm not convinced that an Erlang implementation running on the JVM will really offer great performance benefits, because in order to be *truly like* Erlang, that implementation would *have* to recreate the BEAM emulator running on the JVM. Languages like Scala (and maybe Clojure, I don't actually know) compile down to Java byte code, which means that they're really just syntactic sugar with some runtime library support around it. Perhaps the innovator behind Erjang will pipe in and correct me, but my guess is that you couldn't do that with Erlang without loosing so much of what the langauge/platform is all about. Therefore you're now running a complex runtime environment (i.e., the BEAM emulator) inside another complex runtime environment (the JVM), and fun though they are, the implementations of Jython and JRuby demonstrate that there is a great cost (in terms of performance) when you do this.

I'm happy to be proved wrong, but for all the good engineering that resides in the JVM, I suspect it's probably not the right platform for a competitive Erlang/OTP implementation. 

[Toby Thain]

On 12/02/12 10:17 PM, Deryk Barker wrote:
> As a mere amateur in the erlang world - I've used it a few times in my
> network programming course, probably bending the rules slightly to do so...
>
> But I've been programming for a while (first program FORTRAN on an IBM
> 7090 while still at school in 1968, second program in BASIC on a PDP-8
> while taking a course from Maurice Wilkes at Cambridge in 1970) and
> there's one aspect in all of this thread that has either not been
> mentioned or at least not made much of:
>
> the fact that erlang is a functional language.
>
> For years FP got a bad press, from the early days when the accusation
> was inefficiency ("LISP programmers know the value of everything and the
> cost of nothing") to the more recent criticisms that FP is very nice for
> academics but won't work in the *real* world.
>
> Erlang clearly blows these ideas out of the water and combines all of

All modern functional languages blow those preconceptions out of the
water, for those who care to investigate.

--Toby

> the attributes everybody else has mentioned with those of developing in
> a functional language, with all of its associated benefits, especially
> of correctness.
>
> Besides, I think java is a terribly *ugly* language...:-)
>
> deryk

_______________________________________________

[Miles Fidelman]

Somehow, this conversation reminds me of Virding's First Rule of
Programming - as posted to this list
on 1/11/2008 :-)

-------- reposted message -----

After reading the blogs about how good Erlang's concurrency model is and how
we just just made a super implementation of it in XXX I have been led to
formulate Virding's First Rule of Programming:

Any sufficiently complicated concurrent program in another language contains
an ad hoc informally-specified bug-ridden slow implementation of half of
Erlang.

This is, of course, a mild travesty of Greenspun (*) but I think it is
fundamental enough to be my first rule, not the tenth.

Robert

(*) "Greenspun's Tenth Rule of Programming: any sufficiently complicated C
or Fortran program contains an ad hoc informally-specified bug-ridden slow
implementation of half of Common Lisp."

--------

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Matthew Evans]

A well written email Tim,

I would also add that with development packages such as Maven, building complex Java applications is a breeze. Now personally I think Maven is a royal pain in the ass (XML hell anyone), the ability to bring in dependancies in such a simple way is awesome for product packaging. I've even used it myself - since most of my team are Java developers - for Erlang deployments (see http://erlang-plugin.sourceforge.net/).

I think that what needs to be asked long term is where do we see Erlang been used? Will it just be a niche language, used in certain specific situations? Or do we see it becoming more of a general purpose language?

Of course, it's not ever going to beat Java, C++ or other OOP/imperative languages in terms of number of users, although it would be nice if it would. For one it's off to a bad start with it been functional language. However, unlike Haskell it's not a pure-functional language. Your code can contain little bits of imperative functionality; functions can have side effects for example. In other words it's not too hard to learn.

I guess I can only speak from my own experience trying to get Erlang accepted into two businesses. Normally there is an existing code base, or a set of users who have been coding in language X all their career. These people are always very reluctant to embrace something new. Many of these people will think that it's going to threaten their future in the company. Many will actively lie and present massaged data to prove their point (I've seen this a few times - for example comparing a simple NGINX reverse proxy that is doing nothing to a full-blown Yaws web-server for example).

Managers also have the same worry, but also need to look at what it means to retain staff and employ new staff.

Erlang, unfortunately, often times faces an up-hill battle. 

Something like Erjang can help here. I personally think it's non-ideal for most Erlang deployments, but can be used to get a foot in the door. Deploy Erjang on something that they are familiar with (JVM), have it "natively" talk to Java, and then introduce BEAM later.

I also agree with other people that Erlang's performance is in most cases more than "good enough" for what it's being used for. But when you have a manager who is already reluctant to use it, and employees who are even less willing to use it, they are going to be even less accepting when they *think* it's too slow (even though that might not actually be the case). Anything that can be done by the community to help here is invaluable.

I would add that in my experience many issues of poor-performance are often a case of poor code. People not adhering to OTP principles, people not getting to grips with concurrent programming by doing things like implementing a single Erlang process that serializes requests for the whole system. Or in other cases people not using what's available in the language (regex on binary data using binary_to_list and and then the re module, instead of the binary module). In those cases it won't matter a squat what Ericsson do to improve BEAM. But perhaps we need to make an effort to hammer home good programming practice and be more consistent with core libraries (especially with lists vs binaries).

I guess I'm trying to say that everyone is correct, performance is usually more than good enough. But there is a perception among many non-Erlang developers that that is not the case.

Matt

---

Date: Mon, 13 Feb 2012 12:04:34 +0000
From: watson....@gmail.com
To: dba...@camosun.bc.ca

CC: erlang-q...@erlang.org
Subject: Re: [erlang-questions] The future of Erlang and BEAM

[Tim Watson]

On 13 February 2012 17:53, Matthew Evans <mattev...@hotmail.com> wrote:

A well written email Tim,

I would also add that with development packages such as Maven, building complex Java applications is a breeze. Now personally I think Maven is a royal pain in the ass (XML hell anyone), the ability to bring in dependancies in such a simple way is awesome for product packaging. I've even used it myself - since most of my team are Java developers - for Erlang deployments (see http://erlang-plugin.sourceforge.net/).

Yes indeed. Rebar does a reasonable job with dependencies, but having to build them from source each time isn't ok from the perspective of a large enterprise team with dozens of components making up each project. As we've gradually started breaking up our libraries/applications to factor out common code, this problem has grown. The team have adopted the maven erlang plugin for exactly this reason, although personally whilst I don't mind using maven/ivy for java projects, I'd much rather stick to rebar for Erlang ones.

A binary artefact repository is the thing that seems to be missing, and having it only available as a hosted thing (like Erlware or CEAN 1.0/2.0) doesn't cut it for most shops with large development teams and infrastructure to manage (we have a combination of in-house and out-sourced engineers which I've heard (anecdotally) is around > 0.5 million technologists of which about a quarter are developers - so I guess we qualify as 'large'). 

Personally I think a good solution to this is to use the maven repository structure to deploy artefacts into and fetch them from. That way you can use a free (or commercial version) repository such as Sonatype Nexus for such things internally and have public facing mirrors also. There are complications (with having to use naming conventions to distinguish between different target versions of beam and different target os/architectures where drivers are present) but these aren't insurmountable. I'm quietly working on a dependency manager (like ivy, not a package manager like cean/agner) to do just this, as a rebar plugin.     

 

I think that what needs to be asked long term is where do we see Erlang been used? Will it just be a niche language, used in certain specific situations? Or do we see it becoming more of a general purpose language?

I hope the latter, although it's not fit for every domain and personally I have no problem with multiple technologies mixed together, as long as the architectural boundaries are cleanly defined (e.g., via messaging or whatever integration approach fits the domain best).

 

Of course, it's not ever going to beat Java, C++ or other OOP/imperative languages in terms of number of users, although it would be nice if it would. For one it's off to a bad start with it been functional language. However, unlike Haskell it's not a pure-functional language. Your code can contain little bits of imperative functionality; functions can have side effects for example. In other words it's not too hard to learn.

Coming from an Python + OCaml background, I found Java and .NET quite awkward to learn! ;)

 

I guess I can only speak from my own experience trying to get Erlang accepted into two businesses. Normally there is an existing code base, or a set of users who have been coding in language X all their career. These people are always very reluctant to embrace something new. Many of these people will think that it's going to threaten their future in the company. Many will actively lie and present massaged data to prove their point (I've seen this a few times - for example comparing a simple NGINX reverse proxy that is doing nothing to a full-blown Yaws web-server for example).

Managers also have the same worry, but also need to look at what it means to retain staff and employ new staff.

Erlang, unfortunately, often times faces an up-hill battle. 

I've been through this too. I work in telecoms and unsurprisingly there is a reasonable amount of Erlang based infrastructure, but absolutely nothing existed in the (BSS) business applications space, where Java and .NET dominate. That's gradually changing, but only because my team took the time to build prototypes in our spare time and demonstrate the quality, robustness, *performance* and maintenance benefits these rewrites/alternatives delivered. We still run into a lot of road blocks, and I think it'd be the same if we were pushing Ruby or Python as well.

 

Something like Erjang can help here. I personally think it's non-ideal for most Erlang deployments, but can be used to get a foot in the door. Deploy Erjang on something that they are familiar with (JVM), have it "natively" talk to Java, and then introduce BEAM later.

IMO the bottom line is that a lot of people are loved-up with Java because of the 'enterprise features' which is another way of saying 'all the stuff that makes their life as a developer simple/easy.' Those of us who've used Erlang in anger know that in reality, Java is far more complex than it needs to be and Erlang is part of a group of much more productive languages/platforms. But many more people would be willing to try if there were

- standardised interfaces/APIs (a la Servlets, JDBC)

- better dependency management tools

- awesome build tools*

* IMO rebar is already *the* awesome build tool and I'm using it for all my open source stuff. I think the other two points could (and maybe should??) come from the community, but there's a lot of effort involved. Also Erlang doesn't have a neat mechanism for defining an 'interface' that everyone can follow. There's custom behaviours, which are great, but no simple way to swap in and out of implementations cleanly and easily. Fixing that would be *massively* beneficial to the platform IMHO, but I digress. 

 

I also agree with other people that Erlang's performance is in most cases more than "good enough" for what it's being used for. But when you have a manager who is already reluctant to use it, and employees who are even less willing to use it, they are going to be even less accepting when they *think* it's too slow (even though that might not actually be the case). Anything that can be done by the community to help here is invaluable.

I would add that in my experience many issues of poor-performance are often a case of poor code. People not adhering to OTP principles, people not getting to grips with concurrent programming by doing things like implementing a single Erlang process that serializes requests for the whole system. Or in other cases people not using what's available in the language (regex on binary data using binary_to_list and and then the re module, instead of the binary module). In those cases it won't matter a squat what Ericsson do to improve BEAM. But perhaps we need to make an effort to hammer home good programming practice and be more consistent with core libraries (especially with lists vs binaries).

+1. Most performance difficulties are down to poorly written code - often code that is not 'dirty' but simply not idiomatically correct. We have a web based application (RESTful web service) written on Misultin that out performs the Java application it replaced both in terms of concurrent connections and its raw (round trip) response time (which includes accessing a postgres database).

Incidentally, the key to our success was *not* copying the request lifecycle that the java application had. Asynchronous communication with the database is handled carefully so that each row returned from the network driver is almost immediately made available to the response generating code, which in turn streams its output back to the client immediately. Not waiting until the whole response has been built up saved 3-4x memory use and increased the 'speed' of the application by a similar factor. This is a much more Erlang-ish approach, and that really seems to pay off.

 

I guess I'm trying to say that everyone is correct, performance is usually more than good enough. But there is a perception among many non-Erlang developers that that is not the case.

Benchmarks will fix this. Problem is, they need to be based on the *full request lifecycle* - e.g., for a bog standard web application/service.

Perhaps we need a proof of concept application that does all the various stuff you'd expect to see happening on the server i.e., mapping the request to a handler, validation of input data/parameters, performing some application domain logic, talking to a database, serialising the response back to the client. The Java and .NET crowds have a "Pet Store" application that demonstrates this, attempting to point out good practises along the way. Perhaps if we, as a community, were to produce the same thing, it would not only benefit new users but also provide an opportunity to show that Erlang can be a competitive solution for building such applications. I for one would be willing to participate, although I'd be *much* more inclined to do so if there was a standard API for interacting with web servers. There are a couple of open source efforts, but they're still at about 70 - 80% completeness from what I can tell, and personally I won't back anything that is based on parameterised modules until the OTP team decides once and for all if they're staying or going.

[Miles Fidelman]

Matthew Evans wrote:
>
> I think that what needs to be asked long term is where do we see
> Erlang been used? Will it just be a niche language, used in certain
> specific situations? Or do we see it becoming more of a general
> purpose language?
>

I can answer that at least from the point of view of what I'm looking at
it for:

- highly distributed applications and services (e.g., a network of
Jabber servers, RabbitMQ)

- a platform for applications with high concurrency (e.g., CouchDB)

- modeling and simulation / massively multiplayer games (i.e, anything
that is conceptually a massively concurrent application)

- high-availability service environments (i.e., carrier-class telecom
apps - where Erlang has its roots)

Beyond that, personally, does it need to be general purpose? (For that
matter, what does it mean to be general purpose these days?)

Miles Fidleman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[David Mercer]

Perhaps it wasn’t Erlang's original intent, but I find one of the amazing
things about Erlang that I had an application that ran on a single box.
However, it needed to be running 24×7, which, of course, is impossible on a
single box, since sometimes that box has to be rebooted or taken down for
maintenance. With the addition of just a few config files – no code changes
– I was able to have it running at a single place on my network, but with
failover sessions waiting in the background for the occasions when the main
process goes down. This is a fun demo, too, when people ask why I chose
Erlang for the application; I can just pull up a few remote sessions to the
servers and kill the main, and they can see the failover take over right
away.

> -----Original Message-----
> From: erlang-quest...@erlang.org [mailto:erlang-questions-
> bou...@erlang.org] On Behalf Of Joe Armstrong
> Sent: Saturday, February 11, 2012 11:02 AM
> To: Radek
> Cc: erlang-q...@erlang.org
> Subject: Re: [erlang-questions] The future of Erlang and BEAM
>

[james]

I think you are missing the point with these libraries. For me the key
point with Erlang is the per-process (ie per actor) GC. A Java system
will GC across the whole VM and the response jitter that results is
sometimes very bad. Running Java so it really does keep processing with
simultaneous GC slugs the performance, increasingly so as the core count
goes up. Its also hard to see how a JVM can really work well with a
NUMA system, while Erlang does at least have a chance at making some
optimisations.