it's my first post here, although I've been reading many interesting posts here for quite a long time. Anyway, I posted a question on StackOverflow today with the same title (here <http://stackoverflow.com/questions/9239313/the-future-of-erlang-and-beam>). As I've written there, I don't want to start any flame-war, particularly because I'm actually an Erlang fan-boy :) but I'd like to know your opinion on the subject. Here's the original question:
------- Some time ago I got seriously interested in Erlang (coming from C++/PHP/Java world) - and I've seen it has been successfuly used in the industry, by Ericsson, Facebook, Goldman Sachs, etc. So, I thought it would be a great platform to build high demanding apps, with low-latency profile, with a lot cleaner and nicer language than, for example, Java (for me).
But after "wow effect" has gone, I discovered that there are many high performance Java libraries that seem to resolve many problems that Erlang is theoretically best suited for (real-time, low-latency applications, concurrency, fault-tolerance, etc.). Moreover, it seems that there are things that, despite Erlang profile, are just not possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
So the question arises: is Erlang still the best platform to build such demanding applications ? Wouldn't it be better if we stick to one, very mature (J)VM and try to make it even better than trying to achieve something similar with less resources available (size of OTP team vs. JVM team, supporters, etc) ? And is it possible at all to achieve this kind of performance and adoption with BEAM ?
And just to make things clear: I don't want to init a flame war here. I am just curious becouse I really like Erlang and I think it's a great platform and I'd like to invest time and effort to build real-life projects on it. But I'd just like to know what others might say about that and - if I'wrong - maybe someone could correct me. -------
I hope we can discuss it since I think it would be valuable not only for me.
Before answering your question, could you please clarify to the list what is so special in "LMAX Disruptor"? Why it is impossible to implement it in Erlang? Does it really makes sense to use this technology? I asked those questions because I read about "LMAX Disruptor" while ago, but I didn't find anything special in it. Moreover for me is seems as an obvious thing in a shiny package.
About Java vs Erlang: After many years in Java world (banking, investing, transportation) I started to work with Erlang (including all guys in our company). Now all our software is written in Erlang/C. It handles huge payload, very easy to scale, in case of problems it heals itself automatically. It is very easy to develop and maintain software in Erlang. This is a true dream technology. We are really happy now. I don't see any Java use cases in our company in the future.
p.s. Of course Erlang is not designed for UI.
---
So, thanks again for Ericson company and all those great people that developed Erlang (I hope you read this mail. I am really proud to know you) and all of those people that contribute to Erlang/OTP. Thank you!
On Sat, Feb 11, 2012 at 2:34 PM, Radek <poprostura...@gmail.com> wrote: > Hello Group,
> it's my first post here, although I've been reading many interesting posts > here for quite a long time. > Anyway, I posted a question on StackOverflow today with the same title ( > here<http://stackoverflow.com/questions/9239313/the-future-of-erlang-and-beam>). > As I've written there, I don't want to start any flame-war, particularly > because I'm actually an Erlang fan-boy :) but I'd like to know your opinion > on the subject. Here's the original question:
> ------- > Some time ago I got seriously interested in Erlang (coming from > C++/PHP/Java world) - and I've seen it has been successfuly used in the > industry, by Ericsson, Facebook, Goldman Sachs, etc. So, I thought it would > be a great platform to build high demanding apps, with low-latency profile, > with a lot cleaner and nicer language than, for example, Java (for me).
> But after "wow effect" has gone, I discovered that there are many high > performance Java libraries that seem to resolve many problems that Erlang > is theoretically best suited for (real-time, low-latency applications, > concurrency, fault-tolerance, etc.). Moreover, it seems that there are > things that, despite Erlang profile, are just not possible to achieve on > BEAM (like LMAX Disruptor concurrent framework).
> So the question arises: is Erlang still the best platform to build such > demanding applications ? Wouldn't it be better if we stick to one, very > mature (J)VM and try to make it even better than trying to achieve > something similar with less resources available (size of OTP team vs. JVM > team, supporters, etc) ? And is it possible at all to achieve this kind of > performance and adoption with BEAM ?
> And just to make things clear: I don't want to init a flame war here. I am > just curious becouse I really like Erlang and I think it's a great platform > and I'd like to invest time and effort to build real-life projects on it. > But I'd just like to know what others might say about that and - if I'wrong > - maybe someone could correct me. > -------
> I hope we can discuss it since I think it would be valuable not only for > me.
My take on the LMAX Disruptor is that it would probably be hard to get the same raw throughput in Erlang. OTOH, for many products, this is not the clincher, and Erlang may still be the better option over time, since it tends to make it easier to make use of multicore or improvements in cluster technology in order to get the desired performance improvements.
In my own experience, this has been demonstrated many times. I have witnessed many different C++-based applications temporarily run circles around their Erlang-based counterparts, but eventually running into severe problems, either because they hit hard limits that stopped them from achieving the required performance on available hardware [1], or because changing requirements added a complexity that they were not designed to manage [2]. Another notable occasion was the shift to multicore, where most C++ and Java developers ended up facing a magnificent challenge, while we could ship Erlang-based systems on the new architectures with only minimal changes.
The thing that was missing from your list of things Erlang is that it shines for coordination problems. Erlang is sometimes called an *orchestration language*. This is where most of the competing solutions are often woefully inadequate. The key is where you get inputs from different sources, and must be able to handle events arriving in unexpected order. This is typical for control systems, and actually where the biggest complexity lies.
LMAX Disruptor, if I recall correctly, is essentially a processing pipeline, where each job is independent and processed to completion. Thus, there is no coordination complexity to speak of, and the main challenges are managing access to shared resources and achieving high processor utilization.
Granted, I have spent my whole career working with problems where Erlang is a near-perfect fit. YMMV depending on domain. :)
BR, Ulf W
[1] In one case, the performance gains came in no small part from the fact that the C++ app was using static memory management. This worked fine, until the required capacity required more RAM than could be installed onto the available processor board. Rewriting the code to use dynamic memory management was deemed too risky and expensive, given the schedule.
> Before answering your question, could you please clarify to the list what is so special in "LMAX Disruptor"? Why it is impossible to implement it in Erlang? Does it really makes sense to use this technology? I asked those questions because I read about "LMAX Disruptor" while ago, but I didn't find anything special in it. Moreover for me is seems as an obvious thing in a shiny package.
> About Java vs Erlang: After many years in Java world (banking, investing, transportation) I started to work with Erlang (including all guys in our company). Now all our software is written in Erlang/C. It handles huge payload, very easy to scale, in case of problems it heals itself automatically. It is very easy to develop and maintain software in Erlang. This is a true dream technology. We are really happy now. I don't see any Java use cases in our company in the future.
> p.s. Of course Erlang is not designed for UI.
> ---
> So, thanks again for Ericson company and all those great people that developed Erlang (I hope you read this mail. I am really proud to know you) and all of those people that contribute to Erlang/OTP. Thank you!
> Best regards, > Max
> On Sat, Feb 11, 2012 at 2:34 PM, Radek <poprostura...@gmail.com> wrote: > Hello Group,
> it's my first post here, although I've been reading many interesting posts here for quite a long time. > Anyway, I posted a question on StackOverflow today with the same title (here). As I've written there, I don't want to start any flame-war, particularly because I'm actually an Erlang fan-boy :) but I'd like to know your opinion on the subject. Here's the original question:
> ------- > Some time ago I got seriously interested in Erlang (coming from C++/PHP/Java world) - and I've seen it has been successfuly used in the industry, by Ericsson, Facebook, Goldman Sachs, etc. So, I thought it would be a great platform to build high demanding apps, with low-latency profile, with a lot cleaner and nicer language than, for example, Java (for me).
> But after "wow effect" has gone, I discovered that there are many high performance Java libraries that seem to resolve many problems that Erlang is theoretically best suited for (real-time, low-latency applications, concurrency, fault-tolerance, etc.). Moreover, it seems that there are things that, despite Erlang profile, are just not possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
> So the question arises: is Erlang still the best platform to build such demanding applications ? Wouldn't it be better if we stick to one, very mature (J)VM and try to make it even better than trying to achieve something similar with less resources available (size of OTP team vs. JVM team, supporters, etc) ? And is it possible at all to achieve this kind of performance and adoption with BEAM ?
> And just to make things clear: I don't want to init a flame war here. I am just curious becouse I really like Erlang and I think it's a great platform and I'd like to invest time and effort to build real-life projects on it. But I'd just like to know what others might say about that and - if I'wrong - maybe someone could correct me. > -------
> I hope we can discuss it since I think it would be valuable not only for me.
Thank you for such a quick answer, it's nice to know there is someone to talk to
In posted question I indeed didn't exactly clarifiy what (probably) is so special about these (and others) libraries, so let me do it now.
1. LMAX Disruptor architecture is desribed in Martin Fowler's article here (long) - http://martinfowler.com/articles/lmax.html?t=1319912579 . I got interested in it when I found it's being advertised as a "High Performance Inter-Thread Messaging Library" by which you can run huge number of concurrent tasks with minimal latency between each "thread" (nanoseconds order of magnitude). Of course, it's not "that simple" but still, the numbers are really amazing.
2. Java Chronicle library (https://github.com/peter-lawrey/Java-Chronicle) which describes itself as a "ultra low latency, high throughput, persisted, messaging and event driven in memory database". With their numbers (16 nano-seconds latency and throughputs of 5-20 million messages/record updates per second) I think oi could be easily used as a platform to build very low-latency message-queue or similar kind of middleware.
3. The well-known (it was mentioned on Erlang Factory) library Akka (http://akka.io/) which copies actor model, distribution and fault-tolerance from Erlang and again, looking at their numbers, tends to be faster than Erlang in terms of message passing latency (due to shared heap).
4. Kilim library (http://www.malhar.net/sriram/kilim/) which is somehow similar to akka, but just for message passing. Anyway, similar strength of zero-copy messaging.
And probably more, but that's what I remember the most.
One other thing is, that could be controversial, arising of Clojure (mainly it, but also other FP languages on JVM) which show that we could be faster just because of the Hot-Spot JIT compiler on JVM.
Of course, I could be all wrong, just because I'm rather a beginner (still) in terms of functional programming, Erlang, etc. But I wanted to know your opinion on the subject and maybe show me errors in the way I see this problem.
I understand that Erlang has been created in purpose of managing coordination complexity and it does it's job well. But, maybe I'm a bit of devil's advocate, let's say that we (by which I mean mostly OTP team because of their knowledge of Erlang intrinsics but others too) could follow similar direction that Clojure's creator Rick Hickey took. Which means, to build Erlang on JVM, using tools that we already have (like earlier mentioned Java libraries, etc) and optimize it for using with JIT.
Of course, it's not obvious that (despite being big effort) it would bring desired performance (although, it seems it would) but some advantages for sure, like ubiquity of JVM, wide gremium of supporters/maintainers, ease of deployment (even single .jar maybe), access to huge variety of other languages and libraries, and even some minor things like proper module managament, etc.
As I said before, I'm rather novice in Erlang world (still digging through Erlang and OTP in Action) so all above may be wrong/incomplete/etc. I had this idea some time ago so I finally took decision to write about my concerns here.
On Sat, Feb 11, 2012 at 13:20, Radek <poprostura...@gmail.com> wrote: > But, maybe I'm a bit of devil's advocate, let's say that we (by which I mean > mostly OTP team because of their knowledge of Erlang intrinsics but others > too) could follow similar direction that Clojure's creator Rick Hickey took. > Which means, to build Erlang on JVM, using tools that we already have (like > earlier mentioned Java libraries, etc) and optimize it for using with JIT.
http://erjang.org/ - actually there is such solution already :) Not sure how mature it is, but afaik it implements most of stdlib and bifs _______________________________________________ erlang-questions mailing list erlang-questi...@erlang.org http://erlang.org/mailman/listinfo/erlang-questions
Yes, I've seen it already. But I'm not sure if it's exactly what I meant. I rather thought about making something like a framework which would emphasize concurrency, fault-tolerance, etc. and implement Erlang as a DSL just to operate on it.
Kresten (author of Erjang) did a great job but took (as I understand it; I may be wrong) little different option and "translated" C code to Java, made .beam files possible to be loaded on JVN thus making it capable of running Erlang code.
By making a special framework and Erlang-as-DSL to operate on it, we could then use normal Erlang tools without many downsides of them (for example, we could always rely on messages because of no copy-on-send penalty, use more data structures apart from just lists and tuples, and so on).
Of course, I'd like to repeat that I'm not an Erlang-master nor skilled in language design etc so my ideas can be just wrong.
Greetings, Radek
W dniu 2012-02-11 13:37, Gleb Peregud pisze:
> On Sat, Feb 11, 2012 at 13:20, Radek<poprostura...@gmail.com> wrote: >> But, maybe I'm a bit of devil's advocate, let's say that we (by which I mean >> mostly OTP team because of their knowledge of Erlang intrinsics but others >> too) could follow similar direction that Clojure's creator Rick Hickey took. >> Which means, to build Erlang on JVM, using tools that we already have (like >> earlier mentioned Java libraries, etc) and optimize it for using with JIT. > http://erjang.org/ - actually there is such solution already :) Not > sure how mature it is, but afaik it implements most of stdlib and bifs
> But after "wow effect" has gone, I discovered that there are many high > performance Java libraries that seem to resolve many problems that > Erlang is theoretically best suited for (real-time, low-latency > applications, concurrency, fault-tolerance, etc.). Moreover, it seems > that there are things that, despite Erlang profile, are just not > possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
> So the question arises: is Erlang still the best platform to build > such demanding applications ? Wouldn't it be better if we stick to > one, very mature (J)VM and try to make it even better than trying to > achieve something similar with less resources available (size of OTP > team vs. JVM team, supporters, etc) ? And is it possible at all to > achieve this kind of performance and adoption with BEAM ?
Two observations, from someone who has just started a migration to Erlang - in service of a project with massive concurrency at its core:
1. Erlang and Java start from very different programming perspectives. As far as I can tell, Erlang is still the only environment designed, from day one, to support massive concurrency (processes as fundamental vs. objects as fundamental).
2. Erlang is about 10 years older than Java, and used for building production systems from its inception. There's a lot more real-world experience packed into Erlang and OTP - particularly where massive concurrency is concerned.
Seems to me that Erlang is far more mature than Java for the kinds of problems and environments that are now emerging.
Now one might argue that there's a good academic exercise in writing yet another new language, that learns from the lessons of Erlang, and maybe cleans up some of the more obscure syntax, but such an exercise would take a long time before yielding any kind of mature platform.
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
> But after "wow effect" has gone, I discovered that there are many high > performance Java libraries that seem to resolve many problems that > Erlang is theoretically best suited for (real-time, low-latency > applications, concurrency, fault-tolerance, etc.). Moreover, it seems > that there are things that, despite Erlang profile, are just not > possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
> So the question arises: is Erlang still the best platform to build such > demanding applications ? Wouldn't it be better if we stick to one, very > mature (J)VM and try to make it even better than trying to achieve > something similar with less resources available (size of OTP team vs. > JVM team, supporters, etc) ? And is it possible at all to achieve this > kind of performance and adoption with BEAM ?
The feature I really like in Erlang is it's debuggablity. I can connect to the live system, I can get an Erlang shell on the VM and can inspect the system real time. I can also load code (not only to the live system), which makes development really fast. _______________________________________________ erlang-questions mailing list erlang-questi...@erlang.org http://erlang.org/mailman/listinfo/erlang-questions
Well, although I didn't mean to write a new language (I meant writing Erlang-for-JVM, but rather as a DSL), I agree that Erlang IS very mature and stable. And of course, it started as a industry-level language, so it's designed to operate as such. I was just wondering if and maybe we could benefit from being hosted on JVM which is, although not THAT industrial-level, also capable of being used in such conditions and is most popular VM on the world (so far). The obvious advantage of using Erlang in such conditions would be (apart from above) just not using verbose Java, which is big plus on it's own :) (in my opinion).
So, if we could have both of the worlds, i.e. industry-level of OTP and JVM ubiquity, that would be something huge I think.
>> But after "wow effect" has gone, I discovered that there are many >> high performance Java libraries that seem to resolve many problems >> that Erlang is theoretically best suited for (real-time, low-latency >> applications, concurrency, fault-tolerance, etc.). Moreover, it seems >> that there are things that, despite Erlang profile, are just not >> possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
>> So the question arises: is Erlang still the best platform to build >> such demanding applications ? Wouldn't it be better if we stick to >> one, very mature (J)VM and try to make it even better than trying to >> achieve something similar with less resources available (size of OTP >> team vs. JVM team, supporters, etc) ? And is it possible at all to >> achieve this kind of performance and adoption with BEAM ?
> Two observations, from someone who has just started a migration to > Erlang - in service of a project with massive concurrency at its core:
> 1. Erlang and Java start from very different programming > perspectives. As far as I can tell, Erlang is still the only > environment designed, from day one, to support massive concurrency > (processes as fundamental vs. objects as fundamental).
> 2. Erlang is about 10 years older than Java, and used for building > production systems from its inception. There's a lot more real-world > experience packed into Erlang and OTP - particularly where massive > concurrency is concerned.
> Seems to me that Erlang is far more mature than Java for the kinds of > problems and environments that are now emerging.
> Now one might argue that there's a good academic exercise in writing > yet another new language, that learns from the lessons of Erlang, and > maybe cleans up some of the more obscure syntax, but such an exercise > would take a long time before yielding any kind of mature platform.
that's undoubtedly one of the very handy features of Erlang/OTP, which has roots in it's real-time non-stop usage. But of course I meant the bigger picture here; live debugging is handy but can be (quite) easily reproduced on the JVM also (using OSGi framework).
Of course, Erlang has this from many years now which counts as a big advantage, that's true.
Greetings, Radek
W dniu 2012-02-11 14:58, Attila Rajmund Nohl pisze:
> 2012/2/11, Radek<poprostura...@gmail.com>: > [...] >> But after "wow effect" has gone, I discovered that there are many high >> performance Java libraries that seem to resolve many problems that >> Erlang is theoretically best suited for (real-time, low-latency >> applications, concurrency, fault-tolerance, etc.). Moreover, it seems >> that there are things that, despite Erlang profile, are just not >> possible to achieve on BEAM (like LMAX Disruptor concurrent framework).
>> So the question arises: is Erlang still the best platform to build such >> demanding applications ? Wouldn't it be better if we stick to one, very >> mature (J)VM and try to make it even better than trying to achieve >> something similar with less resources available (size of OTP team vs. >> JVM team, supporters, etc) ? And is it possible at all to achieve this >> kind of performance and adoption with BEAM ? > The feature I really like in Erlang is it's debuggablity. I can > connect to the live system, I can get an Erlang shell on the VM and > can inspect the system real time. I can also load code (not only to > the live system), which makes development really fast.
> Well, although I didn't mean to write a new language (I meant writing > Erlang-for-JVM, but rather as a DSL), I agree that Erlang IS very > mature and stable. And of course, it started as a industry-level > language, so it's designed to operate as such. > I was just wondering if and maybe we could benefit from being hosted > on JVM which is, although not THAT industrial-level, also capable of > being used in such conditions and is most popular VM on the world (so > far). The obvious advantage of using Erlang in such conditions would > be (apart from above) just not using verbose Java, which is big plus > on it's own :) (in my opinion).
> So, if we could have both of the worlds, i.e. industry-level of OTP > and JVM ubiquity, that would be something huge I think.
But why would I want to run massively concurrent software on a JVM? What makes Erlang different are its underpinnings. Massive concurrency on top of a JVM is just broken.
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
Well, I think it depends. It's true that using default concurrency libraries is quite difficult and error prone but as I said before, thare are others which mimics Erlang approach (and do it well, just looking at numbers).
But, I'm not sure if it's a good idea to run anything massively concurrent on JVM; that's one of reasons I've started this topic.
>> Well, although I didn't mean to write a new language (I meant writing >> Erlang-for-JVM, but rather as a DSL), I agree that Erlang IS very >> mature and stable. And of course, it started as a industry-level >> language, so it's designed to operate as such. >> I was just wondering if and maybe we could benefit from being hosted >> on JVM which is, although not THAT industrial-level, also capable of >> being used in such conditions and is most popular VM on the world (so >> far). The obvious advantage of using Erlang in such conditions would >> be (apart from above) just not using verbose Java, which is big plus >> on it's own :) (in my opinion).
>> So, if we could have both of the worlds, i.e. industry-level of OTP >> and JVM ubiquity, that would be something huge I think.
> But why would I want to run massively concurrent software on a JVM? > What makes Erlang different are its underpinnings. Massive > concurrency on top of a JVM is just broken.
Radek wrote: > Well, I think it depends. It's true that using default concurrency > libraries is quite difficult and error prone but as I said before, > thare are others which mimics Erlang approach (and do it well, just > looking at numbers).
> But, I'm not sure if it's a good idea to run anything massively > concurrent on JVM; that's one of reasons I've started this topic.
The only way to run something massively concurrent on a JVM is to write a collection of libraries that would look suspiciously like BEAM. So why bother?
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
>> Well, I think it depends. It's true that using default concurrency >> libraries is quite difficult and error prone but as I said before, thare >> are others which mimics Erlang approach (and do it well, just looking at >> numbers).
>> But, I'm not sure if it's a good idea to run anything massively >> concurrent on JVM; that's one of reasons I've started this topic.
> The only way to run something massively concurrent on a JVM is to write a > collection of libraries that would look suspiciously like BEAM. So why > bother?
> -- > In theory, there is no difference between theory and practice. > In practice, there is. .... Yogi Berra
But on the other hand, there is nothing stopping people to implement the Erlang platform, and compete. So stop talking about it and start hacking. Let the best implementation win ;)
> On Sat, Feb 11, 2012 at 8:09 PM, Miles Fidelman <mfidel...@meetinghouse.net> wrote: > Radek wrote: > Well, I think it depends. It's true that using default concurrency libraries is quite difficult and error prone but as I said before, thare are others which mimics Erlang approach (and do it well, just looking at numbers).
> But, I'm not sure if it's a good idea to run anything massively concurrent on JVM; that's one of reasons I've started this topic.
> The only way to run something massively concurrent on a JVM is to write a collection of libraries that would look suspiciously like BEAM. So why bother?
> -- > In theory, there is no difference between theory and practice. > In practice, there is. .... Yogi Berra
"Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"
Well, that's a good point. But it also shows that there is place for improvement. Because, it would be really come in handy when one could write correct, highly-concurrent, callback-free code (Erlang), using his favourite libraries/frameworks (Java) on a platform that his company uses (JVM).
Like I said, I like Erlang. This is the reason why I left Java etc, just because it's not nice to write something that has to be concurrent and (mainly at least) correct, not to mention development time (which is usually shorter in Java). But I sometimes think that we could make one step more and, as I would call it, join efforts (with JVM community, so to speak).
Greetings, Radek
W dniu 2012-02-11 17:09, Miles Fidelman pisze:
> Radek wrote: >> Well, I think it depends. It's true that using default concurrency >> libraries is quite difficult and error prone but as I said before, >> thare are others which mimics Erlang approach (and do it well, just >> looking at numbers).
>> But, I'm not sure if it's a good idea to run anything massively >> concurrent on JVM; that's one of reasons I've started this topic.
> The only way to run something massively concurrent on a JVM is to > write a collection of libraries that would look suspiciously like > BEAM. So why bother?
I'll try and give a short answer to your question.
To start with you are comparing chalk with cheese - Erlang was NOT designed to be a fast messaging passing language. It was designed for building fault-tolerant applications - it wasn't designed for speed it was designed for safety.
A lot of the message passing platforms you read about, and mention in the same breath as Erlang were designed for speed and not safety - so it's hardly surprising that is the absence of errors these other platforms are faster than Erlang.
Where Erlang should excel is when things go wrong - when the code is buggy and crashes, when the system is overloaded, when you stress things in unexpected ways.
The Erlang view of the world is very different to the view of the world used in most other languages.
Take a web server, for example. In the conventional view of the world if there might be one web server per machine, the web server might spawn an OS process for each incoming task (if it wants safety) of it might handle all sessions in the same task if it wants speed but no safety. It (the web server) uses OS processes for reasons of safety - to isolate the user sessions from each other - but this is very slow.
What happens if there is a rare error in the web-server code itself - one that is very rarely encountered. Does this affect *all* user sessions or only the session where the error occurred? - My guess is that all sessions will collapse if the web-server itself collapses.
How is a web-server in Erlang written? On the face of things it looks like any other web sever - but an Erlang system that appears from the outside to be handling (say) 2000 user sessions, is NOT one web server with 2000 sessions - it is 2000 erlang process (all independent) and each one is running ONE web server.
So in the Erlang case we have 2000 web servers, each running one user session - in the conventional case we have one web server running 2000 sessions.
Back to my question - what happens if there is a rare error in the web server code itself which is triggered in one session? In the Erlang case only one web server dies and all the rest survive. In the case where all 2000 sessions are handled by the faster server all sessions die.
So you choose - fast or safe? - If you want the fastest possible solution Erlang is not the answer - BUT it will be unsafe.
Adding checks and recovery mechanism COSTS something. There are no free lunches.
I often see messaging passing systems mentioned that are said to have "Erlang like semantics" and are said to be faster.
But these are unsafe systems and Erlang like semantics means *a lot more* than fast message passing.
Erlang is good at what it was designed to be good at - which was building fault-tolerant soft real-time distributed systems with dynamic code upgrade - I know of nothing that has remotely similar semantics.
There are *many* systems with superficial similarities - but non that I would trust to build fault-tolerant systems with.
On Sat, Feb 11, 2012 at 11:34 AM, Radek <poprostura...@gmail.com> wrote: > Hello Group,
> it's my first post here, although I've been reading many interesting posts > here for quite a long time. > Anyway, I posted a question on StackOverflow today with the same title > (here). As I've written there, I don't want to start any flame-war, > particularly because I'm actually an Erlang fan-boy :) but I'd like to know > your opinion on the subject. Here's the original question:
> ------- > Some time ago I got seriously interested in Erlang (coming from C++/PHP/Java > world) - and I've seen it has been successfuly used in the industry, by > Ericsson, Facebook, Goldman Sachs, etc. So, I thought it would be a great > platform to build high demanding apps, with low-latency profile, with a lot > cleaner and nicer language than, for example, Java (for me).
> But after "wow effect" has gone, I discovered that there are many high > performance Java libraries that seem to resolve many problems that Erlang is > theoretically best suited for (real-time, low-latency applications, > concurrency, fault-tolerance, etc.). Moreover, it seems that there are > things that, despite Erlang profile, are just not possible to achieve on > BEAM (like LMAX Disruptor concurrent framework).
> So the question arises: is Erlang still the best platform to build such > demanding applications ? Wouldn't it be better if we stick to one, very > mature (J)VM and try to make it even better than trying to achieve something > similar with less resources available (size of OTP team vs. JVM team, > supporters, etc) ? And is it possible at all to achieve this kind of > performance and adoption with BEAM ?
> And just to make things clear: I don't want to init a flame war here. I am > just curious becouse I really like Erlang and I think it's a great platform > and I'd like to invest time and effort to build real-life projects on it. > But I'd just like to know what others might say about that and - if I'wrong > - maybe someone could correct me. > -------
> I hope we can discuss it since I think it would be valuable not only for me.
Yes, I could write an essay on this one.. The other thing to remember about all these Java (and C++) libraries is that there are, frankly, so many of them. Each time you look at one it's a whole different set of semantics that need learning. With Erlang you have a single set of concurrency principles that need learning that are core to the language, and who's behavior is well understood. It makes for code that is easy to reason about, especially when things go wrong. I also worry that many of these Java libraries are only going to be supported until something shinier, newer or better comes along. I should add here that I have recently started learning Java, and quite like the language, especially compared to the effects that C++ has had on my hairline. ;-) Having said that there is a valid criticism of Erlang in that it is often perceived to be too slow. As Joe said, you can't have all the features of resilience and reliability without some cost, but when trying to "sell" Erlang it's often the first argument against Erlang that I hear. The pointy haired managers look at things like the Alioth shootout and see Erlang perform poorly for most tests. Now, most Erlang users will correctly say "well, Erlang isn't designed for those types of highly serial tasks, use C, C++ or Java for that instead, and use Erlang where it makes sense". But this now means that a company needs to embrace 2 (perhaps more) languages to do a job. This isn't necessarily a bad thing, but can make for pain when hiring, training and retaining staff.
My point is that I don't think Erlang will ever be as fast as Java and certainly never as fast as C/C++, but I would like to see a greater focus on performance, perhaps get it into the top 10 languages for performance. I'm happy to see that work is been done WRT to JIT in the Erlang VM, and LLVM+HiPE, so maybe future releases will address those concerns. I've also often wondered if it makes sense to implement some of the Kernel and STDLIB modules as BIFs instead of in Erlang. Would that improve things? Matt________________________________
> Date: Sat, 11 Feb 2012 13:20:54 +0100 > From: poprostura...@gmail.com > To: u...@feuerlabs.com > CC: erlang-questi...@erlang.org > Subject: Re: [erlang-questions] The future of Erlang and BEAM
> # the same as above, sorry for inconvenience
> Hi Ulf,
> thank you very much for answer.
> I understand that Erlang has been created in purpose of managing > coordination complexity and it does it's job well. > But, maybe I'm a bit of devil's advocate, let's say that we (by which I > mean mostly OTP team because of their knowledge of Erlang intrinsics > but others too) could follow similar direction that Clojure's creator > Rick Hickey took. Which means, to build Erlang on JVM, using tools that > we already have (like earlier mentioned Java libraries, etc) and > optimize it for using with JIT.
> Of course, it's not obvious that (despite being big effort) it would > bring desired performance (although, it seems it would) but some > advantages for sure, like ubiquity of JVM, wide gremium of > supporters/maintainers, ease of deployment (even single .jar maybe), > access to huge variety of other languages and libraries, and even some > minor things like proper module managament, etc.
> As I said before, I'm rather novice in Erlang world (still digging > through Erlang and OTP in Action) so all above may be > wrong/incomplete/etc. I had this idea some time ago so I finally took > decision to write about my concerns here.
I'll add one more thing - correct me if I'm wrong here (you are, after all the expert :-)....
Joe Armstrong wrote: > To start with you are comparing chalk with cheese - Erlang was > NOT designed to be a fast messaging passing language. It was designed > for building fault-tolerant applications - it wasn't designed > for speed it was designed for safety.
Erlang also is really nice for building systems where you want to THINK and organize your code in terms of concurrent processes. Simulation is the application that comes to mind (based on my own experience).
If you want to simulate 10,000 tanks, on a simulated battlefield - my first instinct is "each tank is a process." Turns out, that most (all?) conventional simulators model each tank as an object, with a small number of threads touching each object 20-40 times per second (a state update thread, a display thread, etc.) Leads to really horrible spaghetti code and systems that are really hard to change.
I found Erlang when I joined a simulation software house, after a background in networking (where we spawn processes all the time). Programmers kept telling me that "we can't spawn 10,000 processes, the overhead will kill us" - which is true with C++ (what we were using) or Java. I did some digging and found Erlang as a counter-example.
I guess what I draw from this is that, while Erlang might not be blindingly fast for some applications, if I want to write code that spawns 1000s of processes, it's a lot faster than anything else around.
Miles Fidelman
-- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
first of all, I am pretty amazed that my question started such a nice discussion with interesting posts. What amazes me even more, that (with full&great respect to the others) even such known people like Joe Armstrong or Ulf Wiger found some time to answer me directly. I think it's truly amazing :)
Thank you very much, all of you, for such interesting answers. Now I think I understand what is the real difference between Erlang & rest of the world. I really haven't thought about safety and reliability in the first place. Now it seems to be logical for me that "raw performance" isn't an only factor which I should consider.
Thank all of you very much, once more. Now I think I'm Erlang fab-boy even more :)
> I'll add one more thing - correct me if I'm wrong here (you are, after > all the expert :-)....
> Joe Armstrong wrote: >> To start with you are comparing chalk with cheese - Erlang was >> NOT designed to be a fast messaging passing language. It was designed >> for building fault-tolerant applications - it wasn't designed >> for speed it was designed for safety.
> Erlang also is really nice for building systems where you want to > THINK and organize your code in terms of concurrent processes. > Simulation is the application that comes to mind (based on my own > experience).
> If you want to simulate 10,000 tanks, on a simulated battlefield - my > first instinct is "each tank is a process." Turns out, that most > (all?) conventional simulators model each tank as an object, with a > small number of threads touching each object 20-40 times per second (a > state update thread, a display thread, etc.) Leads to really horrible > spaghetti code and systems that are really hard to change.
> I found Erlang when I joined a simulation software house, after a > background in networking (where we spawn processes all the time). > Programmers kept telling me that "we can't spawn 10,000 processes, the > overhead will kill us" - which is true with C++ (what we were using) > or Java. I did some digging and found Erlang as a counter-example.
> I guess what I draw from this is that, while Erlang might not be > blindingly fast for some applications, if I want to write code that > spawns 1000s of processes, it's a lot faster than anything else around.
The best question to ask after this is "What's slow?", "What do you consider to be fast enough?", "What are your requirements?". You often (but not always) find out that they do not really know what they want, so they aim for the fastest overall thinking 'surely, I won't make a mistake there.' Do you need a 5 nanosecond message sending requirement when you're going to treat the message for 20 milliseconds anyway?
In some cases they do need something too fast for Erlang, or think Erlang is optimized for areas it isn't. Then being honest will be the best argument to prove you're giving sane recommendations and are not a fanboy. But before knowing what they really need, it is pretty hard to counter any argument.
There's no use fighting someone whose task to accomplish requires too much work or time in Erlang, but sometimes apprehensions come from not knowing the technology or the problem enough; that's where we can help.
On Sat, Feb 11, 2012 at 1:37 PM, Matthew Evans <mattevans...@hotmail.com>wrote:
> Having said that there is a valid criticism of Erlang in that it is often > perceived to be too slow. As Joe said, you can't have all the features of > resilience and reliability without some cost, but when trying to "sell" > Erlang it's often the first argument against Erlang that I hear. The pointy > haired managers look at things like the Alioth shootout and see Erlang > perform poorly for most tests. Now, most Erlang users will correctly say > "well, Erlang isn't designed for those types of highly serial tasks, use C, > C++ or Java for that instead, and use Erlang where it makes sense". But > this now means that a company needs to embrace 2 (perhaps more) languages > to do a job. This isn't necessarily a bad thing, but can make for pain when > hiring, training and retaining staff.
> My point is that I don't think Erlang will ever be as fast as Java and > certainly never as fast as C/C++, but I would like to see a greater focus > on performance, perhaps get it into the top 10 languages for performance. > I'm happy to see that work is been done WRT to JIT in the Erlang VM, and > LLVM+HiPE, so maybe future releases will address those concerns.
> I've also often wondered if it makes sense to implement some of the Kernel > and STDLIB modules as BIFs instead of in Erlang. Would that improve things?
> Matt > ________________________________ > > Date: Sat, 11 Feb 2012 13:20:54 +0100 > > From: poprostura...@gmail.com > > To: u...@feuerlabs.com > > CC: erlang-questi...@erlang.org > > Subject: Re: [erlang-questions] The future of Erlang and BEAM
> > # the same as above, sorry for inconvenience
> > Hi Ulf,
> > thank you very much for answer.
> > I understand that Erlang has been created in purpose of managing > > coordination complexity and it does it's job well. > > But, maybe I'm a bit of devil's advocate, let's say that we (by which I > > mean mostly OTP team because of their knowledge of Erlang intrinsics > > but others too) could follow similar direction that Clojure's creator > > Rick Hickey took. Which means, to build Erlang on JVM, using tools that > > we already have (like earlier mentioned Java libraries, etc) and > > optimize it for using with JIT.
> > Of course, it's not obvious that (despite being big effort) it would > > bring desired performance (although, it seems it would) but some > > advantages for sure, like ubiquity of JVM, wide gremium of > > supporters/maintainers, ease of deployment (even single .jar maybe), > > access to huge variety of other languages and libraries, and even some > > minor things like proper module managament, etc.
> > As I said before, I'm rather novice in Erlang world (still digging > > through Erlang and OTP in Action) so all above may be > > wrong/incomplete/etc. I had this idea some time ago so I finally took > > decision to write about my concerns here.
I'd really second this. I work with both Erlang and Java (alongside a few other bits and peices) on a daily basis and slowness isn't something I've really run into major difficulties with Erlang at all. If you take a look at that rather well documented case of the chat at Github who implemented an Erlang git daemon, but then found it to be too slow so abandoned it. A more experiences Erlang programmer came along and refactored the code for massive performance improvements.
It's true that sometimes you need raw throughput that Erlang can't provide and need to break out to C/C++ with a port/nif or whatever, but that's not so often the case. Also I prefer to 'pick the right tool for the job' and not every system has to be monolithic running on only one VM.
Cheers,
Tim
On 11 February 2012 20:33, Fred Hebert <monon...@gmail.com> wrote:
> The best question to ask after this is "What's slow?", "What do you > consider to be fast enough?", "What are your requirements?". You often (but > not always) find out that they do not really know what they want, so they > aim for the fastest overall thinking 'surely, I won't make a mistake > there.' Do you need a 5 nanosecond message sending requirement when you're > going to treat the message for 20 milliseconds anyway?
> In some cases they do need something too fast for Erlang, or think Erlang > is optimized for areas it isn't. Then being honest will be the best > argument to prove you're giving sane recommendations and are not a fanboy. > But before knowing what they really need, it is pretty hard to counter any > argument.
> There's no use fighting someone whose task to accomplish requires too much > work or time in Erlang, but sometimes apprehensions come from not knowing > the technology or the problem enough; that's where we can help.
> On Sat, Feb 11, 2012 at 1:37 PM, Matthew Evans <mattevans...@hotmail.com>wrote:
>> Having said that there is a valid criticism of Erlang in that it is often >> perceived to be too slow. As Joe said, you can't have all the features of >> resilience and reliability without some cost, but when trying to "sell" >> Erlang it's often the first argument against Erlang that I hear. The pointy >> haired managers look at things like the Alioth shootout and see Erlang >> perform poorly for most tests. Now, most Erlang users will correctly say >> "well, Erlang isn't designed for those types of highly serial tasks, use C, >> C++ or Java for that instead, and use Erlang where it makes sense". But >> this now means that a company needs to embrace 2 (perhaps more) languages >> to do a job. This isn't necessarily a bad thing, but can make for pain when >> hiring, training and retaining staff.
>> My point is that I don't think Erlang will ever be as fast as Java and >> certainly never as fast as C/C++, but I would like to see a greater focus >> on performance, perhaps get it into the top 10 languages for performance. >> I'm happy to see that work is been done WRT to JIT in the Erlang VM, and >> LLVM+HiPE, so maybe future releases will address those concerns.
>> I've also often wondered if it makes sense to implement some of the >> Kernel and STDLIB modules as BIFs instead of in Erlang. Would that improve >> things?
>> Matt >> ________________________________ >> > Date: Sat, 11 Feb 2012 13:20:54 +0100 >> > From: poprostura...@gmail.com >> > To: u...@feuerlabs.com >> > CC: erlang-questi...@erlang.org >> > Subject: Re: [erlang-questions] The future of Erlang and BEAM
>> > # the same as above, sorry for inconvenience
>> > Hi Ulf,
>> > thank you very much for answer.
>> > I understand that Erlang has been created in purpose of managing >> > coordination complexity and it does it's job well. >> > But, maybe I'm a bit of devil's advocate, let's say that we (by which I >> > mean mostly OTP team because of their knowledge of Erlang intrinsics >> > but others too) could follow similar direction that Clojure's creator >> > Rick Hickey took. Which means, to build Erlang on JVM, using tools that >> > we already have (like earlier mentioned Java libraries, etc) and >> > optimize it for using with JIT.
>> > Of course, it's not obvious that (despite being big effort) it would >> > bring desired performance (although, it seems it would) but some >> > advantages for sure, like ubiquity of JVM, wide gremium of >> > supporters/maintainers, ease of deployment (even single .jar maybe), >> > access to huge variety of other languages and libraries, and even some >> > minor things like proper module managament, etc.
>> > As I said before, I'm rather novice in Erlang world (still digging >> > through Erlang and OTP in Action) so all above may be >> > wrong/incomplete/etc. I had this idea some time ago so I finally took >> > decision to write about my concerns here.
Not really; the first prototype implementation of Erlang was made in 1987; it got its first users at Ericsson 1988-89, and was shown to the public in 1990. Java was implemented 1991-1992 (under the name Oak); they tried to market it to makers of embedded systems and such, but no-one gave a damn, so they changed focus to Internet and multimedia and got Java included in Netscape Navigator in 1995. So Erlang is younger by only about 4 years.
On Sat, Feb 11, 2012 at 2:49 PM, Tim Watson <watson.timo...@gmail.com> wrote: > I'd really second this. I work with both Erlang and Java (alongside a few > other bits and peices) on a daily basis and slowness isn't something I've > really run into major difficulties with Erlang at all. If you take a look at > that rather well documented case of the chat at Github who implemented an > Erlang git daemon, but then found it to be too slow so abandoned it. A more > experiences Erlang programmer came along and refactored the code for massive > performance improvements.
Here's a link to the articles about refactoring egitd by Andrew Thompson. I'd highly recommend them to anyone that's not read the series.
> It's true that sometimes you need raw throughput that Erlang can't provide > and need to break out to C/C++ with a port/nif or whatever, but that's not > so often the case. Also I prefer to 'pick the right tool for the job' and > not every system has to be monolithic running on only one VM.
> Cheers,
> Tim
> On 11 February 2012 20:33, Fred Hebert <monon...@gmail.com> wrote:
>> The best question to ask after this is "What's slow?", "What do you >> consider to be fast enough?", "What are your requirements?". You often (but >> not always) find out that they do not really know what they want, so they >> aim for the fastest overall thinking 'surely, I won't make a mistake there.' >> Do you need a 5 nanosecond message sending requirement when you're going to >> treat the message for 20 milliseconds anyway?
>> In some cases they do need something too fast for Erlang, or think Erlang >> is optimized for areas it isn't. Then being honest will be the best argument >> to prove you're giving sane recommendations and are not a fanboy. But before >> knowing what they really need, it is pretty hard to counter any argument.
>> There's no use fighting someone whose task to accomplish requires too much >> work or time in Erlang, but sometimes apprehensions come from not knowing >> the technology or the problem enough; that's where we can help.
>> On Sat, Feb 11, 2012 at 1:37 PM, Matthew Evans <mattevans...@hotmail.com> >> wrote:
>>> Having said that there is a valid criticism of Erlang in that it is often >>> perceived to be too slow. As Joe said, you can't have all the features of >>> resilience and reliability without some cost, but when trying to "sell" >>> Erlang it's often the first argument against Erlang that I hear. The pointy >>> haired managers look at things like the Alioth shootout and see Erlang >>> perform poorly for most tests. Now, most Erlang users will correctly say >>> "well, Erlang isn't designed for those types of highly serial tasks, use C, >>> C++ or Java for that instead, and use Erlang where it makes sense". But this >>> now means that a company needs to embrace 2 (perhaps more) languages to do a >>> job. This isn't necessarily a bad thing, but can make for pain when hiring, >>> training and retaining staff.
>>> My point is that I don't think Erlang will ever be as fast as Java and >>> certainly never as fast as C/C++, but I would like to see a greater focus on >>> performance, perhaps get it into the top 10 languages for performance. I'm >>> happy to see that work is been done WRT to JIT in the Erlang VM, and >>> LLVM+HiPE, so maybe future releases will address those concerns.
>>> I've also often wondered if it makes sense to implement some of the >>> Kernel and STDLIB modules as BIFs instead of in Erlang. Would that improve >>> things?
>>> Matt >>> ________________________________ >>> > Date: Sat, 11 Feb 2012 13:20:54 +0100 >>> > From: poprostura...@gmail.com >>> > To: u...@feuerlabs.com >>> > CC: erlang-questi...@erlang.org >>> > Subject: Re: [erlang-questions] The future of Erlang and BEAM
>>> > # the same as above, sorry for inconvenience
>>> > Hi Ulf,
>>> > thank you very much for answer.
>>> > I understand that Erlang has been created in purpose of managing >>> > coordination complexity and it does it's job well. >>> > But, maybe I'm a bit of devil's advocate, let's say that we (by which I >>> > mean mostly OTP team because of their knowledge of Erlang intrinsics >>> > but others too) could follow similar direction that Clojure's creator >>> > Rick Hickey took. Which means, to build Erlang on JVM, using tools that >>> > we already have (like earlier mentioned Java libraries, etc) and >>> > optimize it for using with JIT.
>>> > Of course, it's not obvious that (despite being big effort) it would >>> > bring desired performance (although, it seems it would) but some >>> > advantages for sure, like ubiquity of JVM, wide gremium of >>> > supporters/maintainers, ease of deployment (even single .jar maybe), >>> > access to huge variety of other languages and libraries, and even some >>> > minor things like proper module managament, etc.
>>> > As I said before, I'm rather novice in Erlang world (still digging >>> > through Erlang and OTP in Action) so all above may be >>> > wrong/incomplete/etc. I had this idea some time ago so I finally took >>> > decision to write about my concerns here.
