[erlang-questions] About Erlang system performance

[hume npx]

Hi, all:

  I'am new to erlang, after investigate some benchmark such as at http://shootout.alioth.debian.org/u64q/which-programs-are-fastest.php, I found that erlang compiler is not so good at speed? the benchmark shows that erlang Hipe is 13x slowdown compared to C++, as compared to Haskell GHC (3.3x slowdown), go 2.98x slowdown or even javascript v8 is about 1x faster than erlang, I investigated the erLLVM project which reported similar results to Hipe, you know performance is so important nowadays, what caused the hard to improve performace of erlang or just there are not people working on it? Erlang is attractive to me after several days studying, but with great performance will be more attractive and competitive to some languages such as go etc.

[Schneider]

Erlang is a language, which produced byte-code which get' interpreted
by the Erlang engine. C++ instead generates mashing-code which has on
the one hand a much better performance, on the other hand,
some disadvantages like incompatibility between different OSs.

greatz Johannes

> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Matti Oinas]

Correct me if I'm wrong but doesn't HIPE compile to native code?

--
Matti

[Schneider]

you're right. HIPE generates compiled code.
my mistake.
take a look here for more info's about HIPE:
http://user.it.uu.se/~kostis/Papers/erlang03.pdf

greatz Johannes

[Pierpaolo Bernardi]

On Fri, Nov 9, 2012 at 9:42 AM, hume npx <hum...@gmail.com> wrote:
> Erlang is attractive to me after several days studying, but
> with great performance will be more attractive and competitive to some
> languages such as go etc.

Are there any complex systems implemented in go, so we can be sure we
are not comparing apples to oranges?

[Valentin Micic]

Unrelated, but would be interesting to find out -- are there any commercial-grade (let alone carrier-grade) systems that take advantage of HiPE?


V/

[Valentin Micic]

Interestingly enough, this was a kind of conclusion we've arrived at as well -- the good performance of Beam, combined with "perceived instability", acted as a serious deterrent for me.

But Max, wouldn't you like to know if there are any brave souls out there that were prepared to go beyond perceptions and successfully tried it in "real-life" situation?

V/

On 09 Nov 2012, at 3:34 PM, Max Bourinov wrote:

Unrelated, but: we were considering to give a try to HiPE but so far there is no need to do it for us because Beam outperforms everything else in our problem domain.

Best regards,

Max

[Patrik Nyblom]

On 11/09/2012 09:42 AM, hume npx wrote:

Hi, all:

  I'am new to erlang, after investigate some benchmark such as at http://shootout.alioth.debian.org/u64q/which-programs-are-fastest.php, I found that erlang compiler is not so good at speed? the benchmark shows that erlang Hipe is 13x slowdown compared to C++, as compared to Haskell GHC (3.3x slowdown), go 2.98x slowdown or even javascript v8 is about 1x faster than erlang, I investigated the erLLVM project which reported similar results to Hipe, you know performance is so important nowadays, what caused the hard to improve performace of erlang or just there are not people working on it? Erlang is attractive to me after several days studying, but with great performance will be more attractive and competitive to some languages such as go etc.

The thing is - Erlang is a concurrent programming language, so it's optimized for concurrency and the problems therein. HiPE optimizes the sequential part of the system, but cannot do anything about schedulers etc, so it does not affect the concurrency characteristics (which it shouldn't either). HiPE does a really good job, but for sequential mini-benchmarks, HiPE or no HiPE will never be as fast as C++. We do not optimize for that either. Comparing HiPE to i.e. GCH, OCaml or any other common native compiler is really unfair. The other compilers make binaries directly on top of the OS, with no support for massive concurrency in any VM. Not to say that aren't really good compilers, but they solve a different problem.

Building large systems with Erlang will show the whole picture, not running micro benchmarks. If your business is to sell game-of-life, rannkuch-redux or mandelbrot programs, you will get severely disappointed if trying to make the fastest program in Erlang. If you on the other hand is writing a large server, you will be surprised about what level of concurrency, what uptime and what hardware utilization you will achieve.

So - we are really working on optimization, but not on optimizing benchmarks, but on making it possible to implement highly available, scalable, parallell and concurrent systems.

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

Cheers,
Patrik

[Ian]

On 09/11/2012 08:42, hume npx wrote:

Hi, all:

  I'am new to erlang, after investigate some benchmark such as at http://shootout.alioth.debian.org/u64q/which-programs-are-fastest.php, I found that erlang compiler is not so good at speed? the benchmark shows that erlang Hipe is 13x slowdown compared to C++, as compared to Haskell GHC (3.3x slowdown), go 2.98x slowdown or even javascript v8 is about 1x faster than erlang, I investigated the erLLVM project which reported similar results to Hipe, you know performance is so important nowadays, what caused the hard to improve performace of erlang or just there are not people working on it? Erlang is attractive to me after several days studying, but with great performance will be more attractive and competitive to some languages such as go etc.

Hi,

I'm no expert in Erlang, but I would caution against using small benchmarks to make significant decisions. They usually measure the wrong things.

Most significant systems are speed limited due to I/O and networking: the CPU usage can be surprisingly low. Erlang handles async and simultaneous I/O and networking extremely well - by design.

From my experience, and the research of others, anything from 20% to 60% of code in multi-threaded applications is spent checking that other threads have not changed things when they shouldn't, getting and freeing locks and similar overhead. Erlang by its nature, avoids this.

Erlang has a "let it crash" approach. You code the "happy path" and get restarted after error conditions. This means that complicated error conditions do not need to be understood (= less development), detected (less code) or recovered from (much less code).

Multi-threaded processes use operating system threads, which have very very heavy creation and clean up costs, and heavy scheduling costs. Erlang, is between 100 and 10000 times faster at this.

While both traditional and Erlang sytems will wait for I/O and networking, traditional systems under load will wait for worker threads to be free or created. Erlang can proceed directly.

Traditional systems tend to crash under heavy load, while Erlang slows down, and recovers when load eases.

Erlang's garbage collection is superbly designed for soft real-time processing, while most multi-threaded systems either suffer from periodic pauses, or they have considerable code to test and check everything is deallocated when it should be (and not before). Erlang automatic and ridiculously easy to build by comparison.

So, Erlang has so much less overhead, and so much better threading, no locks to contend with (other than the internal ones) and a much lighter scheduling algorithm, and it has less to do.

If you use it for number crunching, it will be slow. For more typical applications Erlang will be built faster, be more reliable, and be smaller and easier to modify, but it might not be quite as fast.  For complicated event-driven and soft-real time processing, Erlang will win on all measures.

So fake up some meaningful test of your proposed load, before making a decision.

Regards

Ian

[Jesper Louis Andersen]

On Nov 9, 2012, at 9:42 AM, hume npx <hum...@gmail.com> wrote:

> Hi, all:
> I'am new to erlang, after investigate some benchmark such as at http://shootout.alioth.debian.org/u64q/which-programs-are-fastest.php, I found that erlang compiler is not so good at speed? the benchmark shows that erlang Hipe is 13x slowdown compared to C++, as compared to Haskell GHC (3.3x slowdown), go 2.98x slowdown or even javascript v8 is about 1x faster than erlang, I investigated the erLLVM project which reported similar results to Hipe, you know performance is so important nowadays, what caused the hard to improve performace of erlang or just there are not people working on it? Erlang is attractive to me after several days studying, but with great performance will be more attractive and competitive to some languages such as go etc.

In addition to other excellent responses, I would like to add the concept of latency on top. Most of the languages you compare to are focused more on throughput than on latency in the sense of concurrency and handling of concurrency. For certain problems, like heavy number crunching, this is *exactly* what you want.

On the other hand, if you are worried how quickly you respond back, then you are probably more served by picking Erlang as a tool.

One could say that Erlang is not really optimized for the Shootout benchmarks as much as for other kinds of problems.

Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen

[Bengt Kleberg]

Greetings,

It was(*) also true that the Shootout test where optimized to not suite
Erlang.


bengt
(*) 2005, things may have changed.

[krad]

Thanks for all of your patient explainations, I understand the main points, erlang is just what I want for my system's requests processing, while there are some other cpu-intensive jobs to run background, it seems that this would be the job of C++ and things can be integrated with nif or node or port interface.

[Fred Hebert]

We've used HiPE in parts of our system that required tight fast loops. Whenever we need something to go fast due to some algorithmic complexity, and that we can manage to isolate it as a single unit that doesn't need to call to non-HiPE code, we will try to convert it.

Basically, I've been told that HiPE and Beam have some overhead when it comes to switching from one to the other. When we have a bit of code that is a hotspot, I'll try to inline whatever calls are made to other modules (mostly lists), compile only that module with HiPE, and let it do its thing. Sometimes benchmarks go in HiPE's favour, sometimes not.

In a particular instance of our stack, it gave us nearly 2x speedup on loop & search operations that frequently happen more than 50,000 times per second. Definitely worth it, and it's been running in production for months, just for that single module.

[Max Lapshin]

What exactly gives performance problems, when calling from HiPE to beam and back?
Marshalling arguments?

But NIF doesn't require marshalling data. Maybe it is possible to achieve something like in LLVM, when function is transparently compiled from bytecode to some native.

Maybe there were some discussions on this mailing list about it?

[Michael Truog]

I think the accepted answer is that if HiPE was normally used in production, it would be the default compilation mode for the Erlang VM (and OTP). Since HiPE is not the default, and its internals are not documented, perhaps it is just to be regarded as an experimental feature that may change in the future.

[Richard Carlsson]

For the sort of computationally heavy code that you'll find in the
shootout benchmarks, you should be comparing Erlang to other dynamically
typed languages like Python and Ruby. One of the main reasons that
Haskell, C++, and the other statically typed languages in the list are
so much faster at this kind of code is that they don't have any (or at
least very little) run-time overhead for checking what type of value is
stored in a variable. Erlang is not aimed at crunching numbers, but it
does a pretty good job of it compared to other dynamically typed
languages - even without HiPE.

/Richard

[Raoul Duke]

>> not people working on it? Erlang is attractive to me after several days
>> studying, but with great performance will be more attractive and
>> competitive to some languages such as go etc.

also -- i'd like to point out that:

* Haskell even comes with channels out of the box which can be used to
get Erlangish (i do stress the "ish" part :-) systems
* Jvm languages can use Akka, or JActor, or Groovy Actors (which also
have dataflow, which is potentially even better than actors).
* Clojure has agents, if you don't need to worry about turning it into
a distributed system.

etc.

there are lots of options these days. (which is both a good and a bad
thing. remember how many miles on the tires erlang has, that's a good
thing.)

[Fred Hebert]

From "The HiPE/x86 Erlang Compiler: System Description and Performance Evaluation" (page 4):

HiPE extends the standard Erlang/OTP runtime system to permit Erlang processes to execute both interpreted code and native machine code.3

Each process has two stacks, one for interpreted code and one for native code. This simplifies garbage collection and exception handling, since each stack contains only frames of a single type.

Control flow between interpreted and native code, e.g. at function calls and returns, is handled by a mode-switch interface. The implementation uses linker-generated proxy code stubs and software trap return addresses to trigger the appropriate mode-switches when invoked. Two important properties of the mode-switch interface are that it preserves tail-recursion (i.e., no sequence of consecutive mode-switching tailcalls grows either stack by more than a constant), and that it imposes no runtime overhead on same-mode calls and returns (i.e., from native to native or from BEAM to BEAM).

To me this is an indication of a cost happening when switching modes (otherwise why specify there is no overhead on same-mode calls and returns), and if I recall correctly both Kostis Sagonas and Jesper Louis Andersen mentioned the same behavior to me at conferences.

So the problem is that if I'm stuck in a tight loop and constantly calling non-HiPE functions and then returning to HiPE, the cost of switching could dwarf HiPE improvements within that loop. Sticking to HiPE only for that part of the code would help me benefit more from the native code.

I couldn't go in deeper detail than this, because I'm not knowledgeable enough to go dive into it and figure things out, though. If more details are needed, either Kostis or Jesper  (or anyone else who knows) could add to the conversation, if they have the time for it, that is.

[Tim Watson]

RabbitMQ compiles a subset of modules using HiPE, where this is deemed
appropriate, and has a pretty huge commercial user base. For details
of what gets hipe compiled see:

- https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit.erl#L190
- https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit.erl#L255

And an extract from the bug report referenced in the first link states
"according to the hipe docs, performance suffers when the hipe-ed code
does little besides 'receive', or makes calls to non-hipe code"...

But for some code it makes a big positive difference.

Cheers,
Tim

[Joe Armstrong]

Performance depends upon the problem you want to solve.

If X is N times faster than Y for solving problem P it does not mean that

X is N times faster than Y for solving any other problem than P.

If P involves non-stop operation, code change on the fly, massive parallelism, massive concurrency, good overload protection, soft real-time garbage collection, ability to scale, faut-tolerance then Erlang is suitable.

Micro-benchmarks don't measure this kind of stuff. 

Nobody implements, for example, the entire 3GPP standard for fun in languages X and Y and compares the results - this would take thousands of man years.

The only place where comparative benchmarking is done is where the benchmarks are small and easy to write.

It's rather like chemistry, there is a reason why chemical plants don't have

200 ft high test tubes - lab experiments (done with test tubes) don't scale.

This is like software - micro benchmarks tell you little about macro behavior.

If a web server behaves well for 1 user it does not mean it will behave well for 100M users, unfortunately you can't easily do the experiment for 100M users. 

Now there are good reasons for using Erlang and there are good reasons for not using Erlang - it would do people a dis-service to suggest that Erlang is good for

everything - which it is not.

Erlang was designed to be good at fault-tolerance, scalability, code-change on-the fly

(these kind of things) - but there are no benchmarks that measure this.

How do you measure fault-tolerance? Write the system in 10 different languages and run for five years and tell me the results - it's not going to happen.

Rule of thumb: Type A - small memory intensive computations - local memory - "maths" - use C.

Type B - Fault-tolerance, code change on the fly, scalable, non-stop operation, massive concurrency, distribution, soft-real time response, networking .. ues Erlang

Don't apply type A benchmarks to type B problems.

In type B there are no benchmarks - here the problem is usually getting the

stuff to work at-all - this is why benchmarks are skewed towards type A problems.

Beging good at type A problems does not mean you will be good at type B systems.

(The there is the hybrid approach - use Erlang to co-ordinate and distribute type

A tasks)

Cheers

/Joe

On Fri, Nov 9, 2012 at 9:42 AM, hume npx <hum...@gmail.com> wrote:

Hi, all:

  I'am new to erlang, after investigate some benchmark such as at http://shootout.alioth.debian.org/u64q/which-programs-are-fastest.php, I found that erlang compiler is not so good at speed? the benchmark shows that erlang Hipe is 13x slowdown compared to C++, as compared to Haskell GHC (3.3x slowdown), go 2.98x slowdown or even javascript v8 is about 1x faster than erlang, I investigated the erLLVM project which reported similar results to Hipe, you know performance is so important nowadays, what caused the hard to improve performace of erlang or just there are not people working on it? Erlang is attractive to me after several days studying, but with great performance will be more attractive and competitive to some languages such as go etc.

[Isaac Gouy]

On Friday, November 9, 2012 5:05:33 AM UTC-8, Pierpaolo Bernardi wrote:

On Fri, Nov 9, 2012 at 9:42 AM, hume npx <hum...@gmail.com> wrote:
> Erlang is attractive to me after several days studying, but
> with great performance will be more attractive and competitive to some
> languages such as go etc.

Are there any complex systems implemented in go, so we can be sure we
are not comparing apples to oranges?

Interesting programming languages are more different than apples to oranges :-)

http://shootout.alioth.debian.org/dont-jump-to-conclusions.php#comparison
 

[Richard O'Keefe]

>
> Most significant systems are speed limited due to I/O and networking: the CPU usage can be surprisingly low. Erlang handles async and simultaneous I/O and networking extremely well - by design.

I recently had a program where each run took 3 hours.
Of that, about 2 minutes was CPU time (in C).
The thing that needed to be optimised was disc seeks,
not CPU instructions.

Systems and benchmarks can be very different.