[erlang-questions] suitability of erlang

[Rustom Mody]

Hello

We are considering Erlang for a project involving significant data-processing under severe time-crunch

I am trying to understand the sentence (from http://www.erlang.org/faq/introduction.html#id49576 )
----------
The most common class of 'less suitable' problems is characterised by performance being a prime requirement and constant-factors having a large effect on performance.
----------

How do I understand the '... and constant-factors...'

Thanks

Rusi

[Ulf Wiger]

How do I understand the '... and constant-factors…'

For example, if the problem involves a significant sequential

component that needs to be very fast, and cannot be made 

fast by distributing the work across multiple work units.

In some cases, even if the problem can scale, it may be the case

that Erlang is simply too slow to ever reach parity with very fast

sequential solutions. One such example is the ant colony 

optimization experiment [1], where the Erlang solution did scale

well, but started off being ca 130x slower on one core than the 

MLTon Standard ML solution. The fact that the Erlang solution 

got 1.9x speedup with two cores then becomes rather academic.

[1] http://eric_rollins.home.mindspring.com/erlangAnt.html

This is of course only speaking in general terms. In many cases,

small parts of performance-critical parts can be factored out and

written in e.g. C, with the rest of the application written in Erlang.

BR,

Ulf W

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.

http://feuerlabs.com

[Matthias Lang]

On Wednesday, October 10, Rustom Mody wrote:

> We are considering Erlang for a project involving significant
> data-processing under severe time-crunch
>
> I am trying to understand the sentence (from
> http://www.erlang.org/faq/introduction.html#id49576 )
> ----------
> The most common class of 'less suitable' problems is characterised by

> performance being a prime requirement *and* constant-factors having a large

> effect on performance.
> ----------

> How do I understand the '... and constant-factors...'

I agree, that statement doesn't make a whole heap of sense. Looking at
it now, I'm no longer sure what I was thinking then. Maybe it was
"If it matters that your pure-Erlang system runs, say, 4x slower than
a competitor's C system, then Erlang alone isn't a good choice". I'll
think about it a bit and try and come up with something which makes
more sense and isn't a novel.

Back to your implied question:

If you haven't worked with Erlang before, then picking a small part of
what you want to do and implementing it is going to give you a much
better feel for performance than replies on a mailing list. E.g. you
have no idea whether I have a clue or whether I'm just making stuff up
because I like seeing myself on a mailing list.

Many (most? all?) systems which use Erlang and which do "significant
data-processing" use C to improve performance on the parts where
performance is important. E.g. if you're converting JPEG images to
PNG, then Erlang is going to be several times slower than C.

There's a classic blog post about this:

http://www.dadgum.com/james/performance.html

in some ways, the post says the opposite of what I wrote. The post was
also written before the native-code compiler became more widely used.

----

Maybe also of interest:

http://www.codinghorror.com/blog/2008/06/exploring-wide-finder.html

----

Another thing: you write "under severe time-crunch", but that could
mean "we need to ship a product in a very short time" or it could mean
"the data has to be processed in a very short time".

Matt
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Rustom Mody]

Well mainly the latter
But the former is also true. (When is it ever not :-))
My concern is that there is some desire to try out erlang which may be fine... However I am wondering whether the difference between concurrency and parallelism has not been appreciated.
http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/
is a haskell implementers view of the terminology
I wonder where erlang fits in on this

Rusi

[Ulf Wiger]

Simon Marlow is a very smart guy, and knows what he's talking about.

Erlang doesn't really target the domain of parallelism, but to some

extent, you can roll your own. Your mileage will depend in part 

on how heavy your jobs are, and there are some other interesting 

differences between the Haskell approach and Erlang's approach.

- While you can do something similar to Erlang-style concurrency

  in Haskell, to a certain extent this has been a research area.

  Nowadays, you can play with Cloud Haskell [1], which closely 

  mimicks Erlang's semantics (well, actually a proposed semantics

  for 'future Erlang'). Not least exception handling has long been

  a tricky issue to figure out, due to Haskell's strict type system.

- Haskell's main focus so far (I believe) has been what Simon calls

  deterministic parallelism. There is quite some promise that 

  Haskell's type system will keep you sane when trying this style

  of programming, whereas in Erlang, you tend to end up treating

  it as an explicit coordination problem - quite doable, and better

  than in most other languages, but more error-prone than it 

  could be in Haskell.

- Erlang's origin is telecoms, where systems are supposed to 

  serve and evolve for many years in the field. This makes it 

  important to support code evolution - given telecom uptime 

  requirements, notably code evolution in a _running system_.

  Erlang's support for in-service code change is difficult to 

  reconcile with a strict static type system, and I believe this 

  feature is also absent from Cloud Haskell (at least I don't

  see it mentioned). If you're building server systems, this is 

  a feature to die for.

- Haskell is a superb research language, and it has seen some 

  success in commercial settings, but primarily in areas where it

  has been possible to maintain a very high skill level across the

  board in the design teams (I'm speculating here; I haven't 

  actually verified it). Erlang, in contrast, has been picked up by

  companies across the globe and used in commercial products

  pretty much after self-study and experimentation. You might 

  say (although (the) Simon(s) might flog me for this), that Erlang

  is quite easy to get into, whereas Haskell becomes easy once

  you've finished your PhD in Functional Programming. :)

- So basically, some of the smartest programmers in the world

  choose Haskell as their weapon, and much of the cutting-edge

  programming language research today is done in Haskell.

  Erlang is a great craftsman's tool for building robust distributed

  systems, with many of the niceties of Haskell, but much less of

  the academic rigour. When deep in the trenches, this is not 

  always a bad thing. ;-)

BR,

Ulf W

[1] http://hackage.haskell.org/package/distributed-process

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Joe Armstrong]

On Wed, Oct 10, 2012 at 6:06 PM, Rustom Mody <rusto...@gmail.com> wrote:
> On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang <matt...@corelatus.se>
> wrote:
>>
>> On Wednesday, October 10, Rustom Mody wrote:
>
>
>>
>> Another thing: you write "under severe time-crunch", but that could
>> mean "we need to ship a product in a very short time" or it could mean
>> "the data has to be processed in a very short time".
>>
>> Matt
>
>
> Well mainly the latter
> But the former is also true. (When is it ever not :-))
> My concern is that there is some desire to try out erlang which may be
> fine... However I am wondering whether the difference between concurrency
> and parallelism has not been appreciated.

Very true - I have given several lectures on this ... many people use the
words concurrent and parallel as synonyms - but there is a real difference.

In a nutshell - parallel has to do with real hardware than can do
several things "at the same time"
whereas concurrency is a software structuring abstraction which
pretends that different
parts of the software can run at the same time (if suitable hardware
were available)

Parallel has to do with hardware - on a single cpu you can write your programs
using pmap (instead of map) and parBegin and parEnd and it won't make
any difference
since the machine is still only doing one instruction at a time.

It's only because machine are fast that we think they do several
things in parallel -
in the single CPU + time sharing case if we slowed the clock down we would see
that no parallelism is involved - the CPU does one thing a time - even
when executing
concurrent programs :-)

There are only a few real sources of true parallelism - multi-cores,
what happens in the pipeline, and how data is fetched/stored

Multicores do give you true parallelism, but you must be careful to
about the granularity
of computation - ie it should not be more work to shift the work to a
remote CPU than do it
locally. Pipeline messing is old for compiler writers. Data fetching
and storing can be done in
parallel, this needs a bit of thought.

Parallel programs are inherently non portable - in the sense that they
specifically depend
upon details of the hardware.

Concurrent programs are program written using a concurrency
abstraction. In Erlang
processes are the unti of concurrency - processes are pretty coarse-grain things
ie they take a wee bit of effort to get going, so they actually map
well onto multi-cores,
even if you can detect parallelism in your program (easy in a pure FP,
just evaluate
all the arguments to a function in parallel) - it's pretty difficult
(ie an open research problem)
to map this onto the available hardware.

Even if you know how long computations take, and you know what
resources you have
to solve the computations, then mapping the computations onto the
resources involves
solving the knapsack problem which is NP hard.

It you write program in Erlang you at least have a head start over
sequential code
since the programmer has at least said (using processes) which
activities should be run in parallel.
Optimally scheduling these is NP hard - but non-optimal first on
demand, best effort scheduling
works pretty well for non-pathological code.

My 10 c.

/Joe

> http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/
> is a haskell implementers view of the terminology
> I wonder where erlang fits in on this
>
> Rusi
>
>

[Rustom Mody]

Thanks Joe for a detailed answer.

I have one more question that sits somewhere between parallelism and concurrency.
In the 'normal' world (ie C++, Java, Python) there is a general recommendation that converting a threaded solution to an event-driven solution usually speeds up the program and removes hard-to-find bugs.
And at the heart of an event-driven system is usually an FSM.
Now I am particularly interested in the combined (regular-exp+FSM) approach that ragel provides: http://www.complang.org/ragel/

In particular, here is the use of ragel to write an http server: http://www.zedshaw.com/essays/ragel_state_charts.html

Now since ragel is more or less language agnostic -- it has backends for generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make sense?

Rusi

[Ulf Wiger]

On 11 Oct 2012, at 16:17, Rustom Mody wrote:

I have one more question that sits somewhere between parallelism and concurrency.
In the 'normal' world (ie C++, Java, Python) there is a general recommendation that converting a threaded solution to an event-driven solution usually speeds up the program and removes hard-to-find bugs.
And at the heart of an event-driven system is usually an FSM.
Now I am particularly interested in the combined (regular-exp+FSM) approach that ragel provides:http://www.complang.org/ragel/

In particular, here is the use of ragel to write an http server: http://www.zedshaw.com/essays/ragel_state_charts.html

Now since ragel is more or less language agnostic -- it has backends for generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make sense?

As far as I can tell, Ragel is mainly for parser FSMs. As such, it seems

great, but I doubt whether it would add value to Erlang as a way to 

specify concurrent (message-passing) state machines.

The thing it lacks that Erlang has, is selective message reception.

This allows Erlang to ignore a message in its inbox, and keep waiting

for the desired message. In coordination problems, this is a great 

way to avoid complexity explosion.

I gave a talk about that a while ago:

http://www.infoq.com/presentations/Death-by-Accidental-Complexity

Basically, you _could_ generate Erlang FSM code from Ragel code,

I'm sure, but I think this would mainly be useful for porting existing

Ragel models over to Erlang. Ultimately, I think it would still be better

to do the 'porting' by hand, to give you a chance to think through the 

problem in a more Erlang-ish style.

The FSMs that are easy to write in Ragel, also ought to be easy to

write in Erlang.

If I missed some clever device in Ragel that handles this, please 

correct me.

BR,

Ulf W

[Raoul Duke]

On Thu, Oct 11, 2012 at 7:17 AM, Rustom Mody <rusto...@gmail.com> wrote:
> In the 'normal' world (ie C++, Java, Python) there is a general
> recommendation that converting a threaded solution to an event-driven
> solution usually speeds up the program and removes hard-to-find bugs.

please note that you are talking about problems that exist in those
paradigms. there are other paradigms that do not have the exact same
problems (and of course then have their own problems :-). i suspect
The Erlang Way is different enough that it is probably a mistake to
just take something for the usual crowd of programming language
paradigms, and dump it onto erlang.

sincerely.

[Joe Armstrong]

I have no idea - you haven't told me what problem you want to solve -
you've told me the
technique you wish to use to solve the problem.

This is a bit "back to front" - I'd like to *start* with the the
problem then figure out how to solve
it - not *start* with the idea that you want FSMs and events and so on
and not tell me what the
problem is.

I can't possibly recommend any technology without knowing what the problem is.

Regarding FSM in erlang - these are easy:

state1() ->
receive
event1 -> state2();
event2 -> state3()
end.

state2() ->
receive
event3 -> state4()
..
end.

whatever - not really worth getting grey hair over

Event driven programming is with Erlang spectacles on - not a wonderful idea.

Events are a poor substitue for the lack of concurrency and non-blocking reads.
If I had a language with a) only one thread of execution and b)
blocking I/O then you
know what I'd do - I'd invent event based programming.

Let me explain what I think happened ...

Once upon a time there was a sequential processor, that could run only
one sequential program.

The program did the following

a: start -> read -> compute -> write -> compute -> read ->
compute ... -> stop

ie it interleaved computation with reads and writes.

If the reads blocked - who cared - it's only doing one thing - a
consequence of this
architecture was "the program should not crash" - since there is only
one thread if it crashes
you have a BIG problem - so we invent "defensive programming"

Now we want to do TWO or more things at the same time:

a: start -> read -> compute -> read -> compute -> read ->
compute ... -> stop
b: start -> read -> read -> write -> compute -> read ...

Now - horrors - read stops the system - it blocks, and writes are
problematic we two things
might write the same object at the same time.

Enter - the big global write lock and event based programming

We transform a: into

a: start -> read and on_read_complete(a1)
a1: compute -> read and on_read_complete(a2)
a2: compute -> read and on_read_complete(a3)

etc. and the same for b - the result is the software is a big mess

look at Javascript

$('#result').load('ajax/test.html', function() {
alert('Load was performed.');
});

This is typical jquery javascript code

load("blaa", fun)

means read fun and when the read completes call fun.

Now look what happened:

Simple code like

a: start -> read -> computer -> write -> read ->

got broken into a mess of callbacks and you need a TOOL to help you - enter
FSM modeling the callback style a -> ... read on-read_complete(a1) becomes
a total mess and needs a graphic tool to solve the problem that should never
occurred in the first place.

If read is NON blocking (ie does not block other processes) then you
don't need callbacks.

What about crashing?

When you have ONE thread crashing is a BIG DEAL

But now we have millions of thread - who cares if one crashes - nobody
- "let it crash"

But wait a moment - fault tolerance then? - "let some other guy fix the error"

Who? some other process. Which process? - enter the link - "the linked
processess"

So now we can chuck the idea of defensive programming - don't do it.
Bye bye to 80% of your code that did error checking :-)

Erlang encourages a different mind-set - you can have hundeds and of
thousands of processes
each running sequential easy to write code

a: start -> read -> compute -> read -> read -> write

and not a FSM or event framework on the horizon

Try it - since (again) I don't know what your problem is I can't
really make more than generic observation - but from the Erlang point
of view events and so on are a very "70's" way of programming.

Cheers

/Joe

[Daniel Luna]

On 11 October 2012 12:38, Raoul Duke <rao...@gmail.com> wrote:
> On Thu, Oct 11, 2012 at 7:17 AM, Rustom Mody <rusto...@gmail.com> wrote:
>> In the 'normal' world (ie C++, Java, Python) there is a general
>> recommendation that converting a threaded solution to an event-driven
>> solution usually speeds up the program and removes hard-to-find bugs.
>
> please note that you are talking about problems that exist in those
> paradigms. there are other paradigms that do not have the exact same
> problems (and of course then have their own problems :-). i suspect
> The Erlang Way is different enough that it is probably a mistake to
> just take something for the usual crowd of programming language
> paradigms, and dump it onto erlang.

Let me pitch in an agreement here and on top of this mention that
there are several reasons the "general recommendation" doesn't even
make sense for Erlang:

1. there are no threads in Erlang
2. the standard programming patterns in Erlang (gen_server etc) are
already very event driven
3. the paradigm for Erlang in general is different from C++ or Java

Cheers,

Daniel

[Rustom Mody]

On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong <erl...@gmail.com> wrote:

I have no idea - you haven't told me what problem you want to solve -
you've told me the
technique you wish to use to solve the problem.

This is a bit "back to front" - I'd like to *start* with the the
problem then figure out how to solve
it - not *start* with the idea that you want FSMs and events and so on
and not tell me what the
problem is.

Legacy system in C -- Sits on top of an event-oriented middleware.
Since the C system is at core FSM-oriented, it is quite unstructured (from the normal programming POV)

Ideally one should throw out the mess and start fresh with an appropriate technology, perhaps Erlang.
Realistically, the stakeholders always prefer evolution to revolution.
If not for anything else, just for unraveling the old-code semantics some re-engineering is inevitably required

[Joe Armstrong]

On Thu, Oct 11, 2012 at 8:47 PM, Rustom Mody <rusto...@gmail.com> wrote:
> On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong <erl...@gmail.com> wrote:
>>
>> I have no idea - you haven't told me what problem you want to solve -
>> you've told me the
>> technique you wish to use to solve the problem.
>>
>> This is a bit "back to front" - I'd like to *start* with the the
>> problem then figure out how to solve
>> it - not *start* with the idea that you want FSMs and events and so on
>> and not tell me what the
>> problem is.
>
>
> Legacy system in C -- Sits on top of an event-oriented middleware.
> Since the C system is at core FSM-oriented, it is quite unstructured (from
> the normal programming POV)

Ohhhhh - Ummm - tricky - no easy thing to do here.

New technologies have their best chance a) immediately after a disaster
or b) at the start of a new project.

Cheers

/Joe

>
> Ideally one should throw out the mess and start fresh with an appropriate
> technology, perhaps Erlang.
> Realistically, the stakeholders always prefer evolution to revolution.
> If not for anything else, just for unraveling the old-code semantics some
> re-engineering is inevitably required

[Richard O'Keefe]

From my perspective, the event loop approach has always been
a bug-ridden nightmare. The idea of turning a multi-thread
program into an event loop one in order to *remove* bugs is,
wow, magno bizarro. I mean, it's like trying to cope with
mice in the house by releasing a lot of large hungry rats.
Yes, they'll eat up the mice, but then you have worse problems.
People who recommend rats (I mean event loops) are thinking
about threads in languages that don't really understand them.
It's hard to imagine a Concurrent Pascal, Occam, or Ada
programmer thinking their program would be improved by turning
it inside out and back to front. But then, Concurrent Pascal,
Occam, and Ada compilers have a clue about what's going on, and
prevent you writing code that has data races and so on in the
first place.

If you want high performance, then in a multicore world turning
threads into an event loop is the very worst thing you can do,
because that one thread is going to run on one core and the
others will sit idle. Again, whatever validity the advice may
once have had was limited to single-core cpus with heavy-weight
threads and high thread switching overheads. Not Erlang.

[Michael Truog]

There are 4 main ways to integrate Erlang and C/C++ source code:
- NIF
- port driver
- port
- cnode

The NIF and port driver approach can expose the Erlang VM to any memory problems the C code creates, though it is the most efficient.  Usually legacy C or C++ code is best integrated with a port, which is ran as a separate OS process communicating with pipes.  I have an open source project here: http://cloudi.org, which can simplify this integration and also provide natural scalability for your legacy source code if: you use the CloudI API (in this case, in C or C++) to communicate with an Erlang CloudI service that handles your external communication that needs to scale.  Then you can easily migrate parts of your system, or the whole system, over to Erlang, which will naturally improve your testing, debugging, scalability, reliability, concurrency and the source code size.  The integration is similar to integrating with an Erlang port, but it provides features that prevent integration problems, like:
- stdout/stderr handling
- automatic OS process failure handling (similar to supervisor restarts)
- request UUIDs
- requests as transactions (i.e., each request receives a response asynchronously)
- timeouts are enforced for soft-realtime constraints on the external software
- native language integration (both C and C++ have separate interfaces which provide the CloudI API)

As with all open source projects, it needs more documentation, but tell me if you have any questions.

[Rustom Mody]

On Fri, Oct 12, 2012 at 9:23 AM, Michael Truog <mjt...@gmail.com> wrote:

There are 4 main ways to integrate Erlang and C/C++ source code:

- NIF
- port driver
- port
- cnode

Thanks thats a good set of starting points
 

The NIF and port driver approach can expose the Erlang VM to any memory problems the C code creates, though it is the most efficient.  Usually legacy C or C++ code is best integrated with a port, which is ran as a separate OS process communicating with pipes.  I have an open source project here: http://cloudi.org, which can simplify this integration and also provide natural scalability for your legacy source code …

Great! I dont yet know our project or Erlang well enough; yet I expect that this will be a help
 

if: you use the CloudI API (in this case, in C or C++) to communicate with an Erlang CloudI service that handles your external communication that needs to scale.  Then you can easily migrate parts of your system, or the whole system, over to Erlang, which will naturally improve your testing, debugging, scalability, reliability, concurrency and the source code size.  The integration is similar to integrating with an Erlang port, but it provides features that prevent integration problems, like:
- stdout/stderr handling
- automatic OS process failure handling (similar to supervisor restarts)
- request UUIDs
- requests as transactions (i.e., each request receives a response asynchronously)
- timeouts are enforced for soft-realtime constraints on the external software
- native language integration (both C and C++ have separate interfaces which provide the CloudI API)

As with all open source projects, it needs more documentation, but tell me if you have any questions.

As we start making headway I would be happy to contribute patches to docs.
Maybe even code but I am some way off yet :-)

Rusi

--

http://blog.languager.org

[Jesper Louis Andersen]

On Oct 10, 2012, at 8:30 PM, Joe Armstrong <erl...@gmail.com> wrote:

> Even if you know how long computations take, and you know what
> resources you have
> to solve the computations, then mapping the computations onto the
> resources involves
> solving the knapsack problem which is NP hard.

One would think so. But in practice, the Knapsack problem is pseudo-polynomial and there exists very efficient algorithms for its solution. They are so fast that they often find the knapsack before an O(n lg n) sort has run
on the items.

So this particular problem is solved.

Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen

[Jesper Louis Andersen]

> How do I understand the '... and constant-factors…'

I think, but may be wrong, that it refers to the fact that Erlang is an interpreted language, which is also dynamically typed. There is no way you get such a language to have as low a constant factor for a computation in general as a solution baked up in C. A good example is if you want to do an FFT on some data. Since you have the interpreter overhead, there is no way to win here. At all. The constant factor on this piece of code is going to be higher.

But there are two things to be said about this: One, a typical program does not spend its time in a small kernel of computation anymore. It is way more complex and has a lot of interaction with the real world. And if there is a small kernel, you can implement the kernel in a different language and get away with it.

Two, I've found that a very large factor of performance stems from the ability to try out clever solutions and be clever about what you do. And in my experience, this is easier to achieve in sane, modern languages like ML, Erlang or (to a certain extent) Haskell. Thus, if you can trade a slow solution for a faster one quickly, then you end up with a faster program in the end. Another factor is time constraint, where a higher level language lets you express the problem quicker.

But Erlang does take some knowledge to make perform.

[Rustom Mody]

On Mon, Oct 15, 2012 at 2:15 PM, Jesper Louis Andersen <jesper.lou...@erlang-solutions.com> wrote:

Two, I've found that a very large factor of performance stems from the ability to try out clever solutions and be clever about what you do. And in my experience, this is easier to achieve in sane, modern languages like ML, Erlang or (to a certain extent) Haskell. Thus, if you can trade a slow solution for a faster one quickly, then you end up with a faster program in the end. Another factor is time constraint, where a higher level language lets you express the problem quicker.

Thanks Jesper for your comments.
I was brought up on lisp and have spent a couple of decades on Haskell and such languages.  So the functional aspect of Erlang is not (likely to be) an issue for me.

Concurrency however is new!

So any directions of exercises that I could try to get into 'the Erlang groove' would be helpful.

[I am of course reading Joe's book]

Rusi