We are considering Erlang for a project involving significant
data-processing under severe time-crunch
I am trying to understand the sentence (from
http://www.erlang.org/faq/introduction.html#id49576 )
----------
The most common class of 'less suitable' problems is characterised by
performance being a prime requirement *and* constant-factors having a large
effect on performance.
----------
How do I understand the '... and constant-factors...'
> We are considering Erlang for a project involving significant data-processing under severe time-crunch
> I am trying to understand the sentence (from http://www.erlang.org/faq/introduction.html#id49576 )
> ----------
> The most common class of 'less suitable' problems is characterised by performance being a prime requirement and constant-factors having a large effect on performance.
> ----------
> How do I understand the '... and constant-factors…'
For example, if the problem involves a significant sequential
component that needs to be very fast, and cannot be made fast by distributing the work across multiple work units.
In some cases, even if the problem can scale, it may be the case
that Erlang is simply too slow to ever reach parity with very fast
sequential solutions. One such example is the ant colony optimization experiment [1], where the Erlang solution did scale
well, but started off being ca 130x slower on one core than the MLTon Standard ML solution. The fact that the Erlang solution got 1.9x speedup with two cores then becomes rather academic.
This is of course only speaking in general terms. In many cases,
small parts of performance-critical parts can be factored out and
written in e.g. C, with the rest of the application written in Erlang.
On Wednesday, October 10, Rustom Mody wrote:
> We are considering Erlang for a project involving significant
> data-processing under severe time-crunch
> I am trying to understand the sentence (from
> http://www.erlang.org/faq/introduction.html#id49576 )
> ----------
> The most common class of 'less suitable' problems is characterised by
> performance being a prime requirement *and* constant-factors having a large
> effect on performance.
> ----------
> How do I understand the '... and constant-factors...'
I agree, that statement doesn't make a whole heap of sense. Looking at
it now, I'm no longer sure what I was thinking then. Maybe it was
"If it matters that your pure-Erlang system runs, say, 4x slower than
a competitor's C system, then Erlang alone isn't a good choice". I'll
think about it a bit and try and come up with something which makes
more sense and isn't a novel.
Back to your implied question:
If you haven't worked with Erlang before, then picking a small part of
what you want to do and implementing it is going to give you a much
better feel for performance than replies on a mailing list. E.g. you
have no idea whether I have a clue or whether I'm just making stuff up
because I like seeing myself on a mailing list.
Many (most? all?) systems which use Erlang and which do "significant
data-processing" use C to improve performance on the parts where
performance is important. E.g. if you're converting JPEG images to
PNG, then Erlang is going to be several times slower than C.
Another thing: you write "under severe time-crunch", but that could
mean "we need to ship a product in a very short time" or it could mean
"the data has to be processed in a very short time".
On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang <matth...@corelatus.se>wrote:
> On Wednesday, October 10, Rustom Mody wrote:
> Another thing: you write "under severe time-crunch", but that could
> mean "we need to ship a product in a very short time" or it could mean
> "the data has to be processed in a very short time".
> Matt
Well mainly the latter
But the former is also true. (When is it ever not :-))
My concern is that there is some desire to try out erlang which may be
fine... However I am wondering whether the difference between concurrency
and parallelism has not been appreciated.
http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/ is a haskell implementers view of the terminology
I wonder where erlang fits in on this
Simon Marlow is a very smart guy, and knows what he's talking about.
Erlang doesn't really target the domain of parallelism, but to some
extent, you can roll your own. Your mileage will depend in part on how heavy your jobs are, and there are some other interesting differences between the Haskell approach and Erlang's approach.
- While you can do something similar to Erlang-style concurrency
in Haskell, to a certain extent this has been a research area.
Nowadays, you can play with Cloud Haskell [1], which closely mimicks Erlang's semantics (well, actually a proposed semantics
for 'future Erlang'). Not least exception handling has long been
a tricky issue to figure out, due to Haskell's strict type system.
- Haskell's main focus so far (I believe) has been what Simon calls
deterministic parallelism. There is quite some promise that Haskell's type system will keep you sane when trying this style
of programming, whereas in Erlang, you tend to end up treating
it as an explicit coordination problem - quite doable, and better
than in most other languages, but more error-prone than it could be in Haskell.
- Erlang's origin is telecoms, where systems are supposed to serve and evolve for many years in the field. This makes it important to support code evolution - given telecom uptime requirements, notably code evolution in a _running system_.
Erlang's support for in-service code change is difficult to reconcile with a strict static type system, and I believe this feature is also absent from Cloud Haskell (at least I don't
see it mentioned). If you're building server systems, this is a feature to die for.
- Haskell is a superb research language, and it has seen some success in commercial settings, but primarily in areas where it
has been possible to maintain a very high skill level across the
board in the design teams (I'm speculating here; I haven't actually verified it). Erlang, in contrast, has been picked up by
companies across the globe and used in commercial products
pretty much after self-study and experimentation. You might say (although (the) Simon(s) might flog me for this), that Erlang
is quite easy to get into, whereas Haskell becomes easy once
you've finished your PhD in Functional Programming. :)
- So basically, some of the smartest programmers in the world
choose Haskell as their weapon, and much of the cutting-edge
programming language research today is done in Haskell.
Erlang is a great craftsman's tool for building robust distributed
systems, with many of the niceties of Haskell, but much less of
the academic rigour. When deep in the trenches, this is not always a bad thing. ;-)
> On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang <matth...@corelatus.se> wrote:
> On Wednesday, October 10, Rustom Mody wrote:
> Another thing: you write "under severe time-crunch", but that could
> mean "we need to ship a product in a very short time" or it could mean
> "the data has to be processed in a very short time".
> Matt
> Well mainly the latter
> But the former is also true. (When is it ever not :-))
> My concern is that there is some desire to try out erlang which may be fine... However I am wondering whether the difference between concurrency and parallelism has not been appreciated.
> http://ghcmutterings.wordpress.com/2009/10/06/parallelism-concurrency/ > is a haskell implementers view of the terminology
> I wonder where erlang fits in on this
On Wed, Oct 10, 2012 at 6:06 PM, Rustom Mody <rustompm...@gmail.com> wrote:
> On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang <matth...@corelatus.se>
> wrote:
>> On Wednesday, October 10, Rustom Mody wrote:
>> Another thing: you write "under severe time-crunch", but that could
>> mean "we need to ship a product in a very short time" or it could mean
>> "the data has to be processed in a very short time".
>> Matt
> Well mainly the latter
> But the former is also true. (When is it ever not :-))
> My concern is that there is some desire to try out erlang which may be
> fine... However I am wondering whether the difference between concurrency
> and parallelism has not been appreciated.
Very true - I have given several lectures on this ... many people use the
words concurrent and parallel as synonyms - but there is a real difference.
In a nutshell - parallel has to do with real hardware than can do
several things "at the same time"
whereas concurrency is a software structuring abstraction which
pretends that different
parts of the software can run at the same time (if suitable hardware
were available)
Parallel has to do with hardware - on a single cpu you can write your programs
using pmap (instead of map) and parBegin and parEnd and it won't make
any difference
since the machine is still only doing one instruction at a time.
It's only because machine are fast that we think they do several
things in parallel -
in the single CPU + time sharing case if we slowed the clock down we would see
that no parallelism is involved - the CPU does one thing a time - even
when executing
concurrent programs :-)
There are only a few real sources of true parallelism - multi-cores,
what happens in the pipeline, and how data is fetched/stored
Multicores do give you true parallelism, but you must be careful to
about the granularity
of computation - ie it should not be more work to shift the work to a
remote CPU than do it
locally. Pipeline messing is old for compiler writers. Data fetching
and storing can be done in
parallel, this needs a bit of thought.
Parallel programs are inherently non portable - in the sense that they
specifically depend
upon details of the hardware.
Concurrent programs are program written using a concurrency
abstraction. In Erlang
processes are the unti of concurrency - processes are pretty coarse-grain things
ie they take a wee bit of effort to get going, so they actually map
well onto multi-cores,
even if you can detect parallelism in your program (easy in a pure FP,
just evaluate
all the arguments to a function in parallel) - it's pretty difficult
(ie an open research problem)
to map this onto the available hardware.
Even if you know how long computations take, and you know what
resources you have
to solve the computations, then mapping the computations onto the
resources involves
solving the knapsack problem which is NP hard.
It you write program in Erlang you at least have a head start over
sequential code
since the programmer has at least said (using processes) which
activities should be run in parallel.
Optimally scheduling these is NP hard - but non-optimal first on
demand, best effort scheduling
works pretty well for non-pathological code.
On Thu, Oct 11, 2012 at 12:00 AM, Joe Armstrong <erl...@gmail.com> wrote:
> On Wed, Oct 10, 2012 at 6:06 PM, Rustom Mody <rustompm...@gmail.com>
> wrote:
> > On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang <matth...@corelatus.se>
> > wrote:
> >> On Wednesday, October 10, Rustom Mody wrote:
> >> Another thing: you write "under severe time-crunch", but that could
> >> mean "we need to ship a product in a very short time" or it could mean
> >> "the data has to be processed in a very short time".
> >> Matt
> > Well mainly the latter
> > But the former is also true. (When is it ever not :-))
> > My concern is that there is some desire to try out erlang which may be
> > fine... However I am wondering whether the difference between concurrency
> > and parallelism has not been appreciated.
> Very true - I have given several lectures on this ... many people use the
> words concurrent and parallel as synonyms - but there is a real difference.
> In a nutshell - parallel has to do with real hardware than can do
> several things "at the same time"
> whereas concurrency is a software structuring abstraction which
> pretends that different
> parts of the software can run at the same time (if suitable hardware
> were available)
> Parallel has to do with hardware - on a single cpu you can write your
> programs
> using pmap (instead of map) and parBegin and parEnd and it won't make
> any difference
> since the machine is still only doing one instruction at a time.
> It's only because machine are fast that we think they do several
> things in parallel -
> in the single CPU + time sharing case if we slowed the clock down we would
> see
> that no parallelism is involved - the CPU does one thing a time - even
> when executing
> concurrent programs :-)
> There are only a few real sources of true parallelism - multi-cores,
> what happens in the pipeline, and how data is fetched/stored
> Multicores do give you true parallelism, but you must be careful to
> about the granularity
> of computation - ie it should not be more work to shift the work to a
> remote CPU than do it
> locally. Pipeline messing is old for compiler writers. Data fetching
> and storing can be done in
> parallel, this needs a bit of thought.
> Parallel programs are inherently non portable - in the sense that they
> specifically depend
> upon details of the hardware.
> Concurrent programs are program written using a concurrency
> abstraction. In Erlang
> processes are the unti of concurrency - processes are pretty coarse-grain
> things
> ie they take a wee bit of effort to get going, so they actually map
> well onto multi-cores,
> even if you can detect parallelism in your program (easy in a pure FP,
> just evaluate
> all the arguments to a function in parallel) - it's pretty difficult
> (ie an open research problem)
> to map this onto the available hardware.
> Even if you know how long computations take, and you know what
> resources you have
> to solve the computations, then mapping the computations onto the
> resources involves
> solving the knapsack problem which is NP hard.
> It you write program in Erlang you at least have a head start over
> sequential code
> since the programmer has at least said (using processes) which
> activities should be run in parallel.
> Optimally scheduling these is NP hard - but non-optimal first on
> demand, best effort scheduling
> works pretty well for non-pathological code.
> My 10 c.
> /Joe
Thanks Joe for a detailed answer.
I have one more question that sits somewhere between parallelism and
concurrency.
In the 'normal' world (ie C++, Java, Python) there is a general
recommendation that converting a threaded solution to an event-driven
solution usually speeds up the program and removes hard-to-find bugs.
And at the heart of an event-driven system is usually an FSM.
Now I am particularly interested in the combined (regular-exp+FSM) approach
that ragel provides: http://www.complang.org/ragel/
Now since ragel is more or less language agnostic -- it has backends for
generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make
sense?
> I have one more question that sits somewhere between parallelism and concurrency.
> In the 'normal' world (ie C++, Java, Python) there is a general recommendation that converting a threaded solution to an event-driven solution usually speeds up the program and removes hard-to-find bugs.
> And at the heart of an event-driven system is usually an FSM.
> Now I am particularly interested in the combined (regular-exp+FSM) approach that ragel provides:http://www.complang.org/ragel/
> Now since ragel is more or less language agnostic -- it has backends for generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make sense?
As far as I can tell, Ragel is mainly for parser FSMs. As such, it seems
great, but I doubt whether it would add value to Erlang as a way to specify concurrent (message-passing) state machines.
The thing it lacks that Erlang has, is selective message reception.
This allows Erlang to ignore a message in its inbox, and keep waiting
for the desired message. In coordination problems, this is a great way to avoid complexity explosion.
Basically, you _could_ generate Erlang FSM code from Ragel code,
I'm sure, but I think this would mainly be useful for porting existing
Ragel models over to Erlang. Ultimately, I think it would still be better
to do the 'porting' by hand, to give you a chance to think through the problem in a more Erlang-ish style.
The FSMs that are easy to write in Ragel, also ought to be easy to
write in Erlang.
If I missed some clever device in Ragel that handles this, please correct me.
On Thu, Oct 11, 2012 at 7:17 AM, Rustom Mody <rustompm...@gmail.com> wrote:
> In the 'normal' world (ie C++, Java, Python) there is a general
> recommendation that converting a threaded solution to an event-driven
> solution usually speeds up the program and removes hard-to-find bugs.
please note that you are talking about problems that exist in those
paradigms. there are other paradigms that do not have the exact same
problems (and of course then have their own problems :-). i suspect
The Erlang Way is different enough that it is probably a mistake to
just take something for the usual crowd of programming language
paradigms, and dump it onto erlang.
On Thu, Oct 11, 2012 at 4:17 PM, Rustom Mody <rustompm...@gmail.com> wrote:
> On Thu, Oct 11, 2012 at 12:00 AM, Joe Armstrong <erl...@gmail.com> wrote:
>> On Wed, Oct 10, 2012 at 6:06 PM, Rustom Mody <rustompm...@gmail.com>
>> wrote:
>> > On Wed, Oct 10, 2012 at 3:45 PM, Matthias Lang <matth...@corelatus.se>
>> > wrote:
>> >> On Wednesday, October 10, Rustom Mody wrote:
>> >> Another thing: you write "under severe time-crunch", but that could
>> >> mean "we need to ship a product in a very short time" or it could mean
>> >> "the data has to be processed in a very short time".
>> >> Matt
>> > Well mainly the latter
>> > But the former is also true. (When is it ever not :-))
>> > My concern is that there is some desire to try out erlang which may be
>> > fine... However I am wondering whether the difference between
>> > concurrency
>> > and parallelism has not been appreciated.
>> Very true - I have given several lectures on this ... many people use the
>> words concurrent and parallel as synonyms - but there is a real
>> difference.
>> In a nutshell - parallel has to do with real hardware than can do
>> several things "at the same time"
>> whereas concurrency is a software structuring abstraction which
>> pretends that different
>> parts of the software can run at the same time (if suitable hardware
>> were available)
>> Parallel has to do with hardware - on a single cpu you can write your
>> programs
>> using pmap (instead of map) and parBegin and parEnd and it won't make
>> any difference
>> since the machine is still only doing one instruction at a time.
>> It's only because machine are fast that we think they do several
>> things in parallel -
>> in the single CPU + time sharing case if we slowed the clock down we would
>> see
>> that no parallelism is involved - the CPU does one thing a time - even
>> when executing
>> concurrent programs :-)
>> There are only a few real sources of true parallelism - multi-cores,
>> what happens in the pipeline, and how data is fetched/stored
>> Multicores do give you true parallelism, but you must be careful to
>> about the granularity
>> of computation - ie it should not be more work to shift the work to a
>> remote CPU than do it
>> locally. Pipeline messing is old for compiler writers. Data fetching
>> and storing can be done in
>> parallel, this needs a bit of thought.
>> Parallel programs are inherently non portable - in the sense that they
>> specifically depend
>> upon details of the hardware.
>> Concurrent programs are program written using a concurrency
>> abstraction. In Erlang
>> processes are the unti of concurrency - processes are pretty coarse-grain
>> things
>> ie they take a wee bit of effort to get going, so they actually map
>> well onto multi-cores,
>> even if you can detect parallelism in your program (easy in a pure FP,
>> just evaluate
>> all the arguments to a function in parallel) - it's pretty difficult
>> (ie an open research problem)
>> to map this onto the available hardware.
>> Even if you know how long computations take, and you know what
>> resources you have
>> to solve the computations, then mapping the computations onto the
>> resources involves
>> solving the knapsack problem which is NP hard.
>> It you write program in Erlang you at least have a head start over
>> sequential code
>> since the programmer has at least said (using processes) which
>> activities should be run in parallel.
>> Optimally scheduling these is NP hard - but non-optimal first on
>> demand, best effort scheduling
>> works pretty well for non-pathological code.
>> My 10 c.
>> /Joe
> Thanks Joe for a detailed answer.
> I have one more question that sits somewhere between parallelism and
> concurrency.
> In the 'normal' world (ie C++, Java, Python) there is a general
> recommendation that converting a threaded solution to an event-driven
> solution usually speeds up the program and removes hard-to-find bugs.
> And at the heart of an event-driven system is usually an FSM.
> Now I am particularly interested in the combined (regular-exp+FSM) approach
> that ragel provides: http://www.complang.org/ragel/
> Now since ragel is more or less language agnostic -- it has backends for
> generating C, C++, Java, Ruby etc -- would an Erlang backend for ragel make
> sense?
I have no idea - you haven't told me what problem you want to solve -
you've told me the
technique you wish to use to solve the problem.
This is a bit "back to front" - I'd like to *start* with the the
problem then figure out how to solve
it - not *start* with the idea that you want FSMs and events and so on
and not tell me what the
problem is.
I can't possibly recommend any technology without knowing what the problem is.
whatever - not really worth getting grey hair over
Event driven programming is with Erlang spectacles on - not a wonderful idea.
Events are a poor substitue for the lack of concurrency and non-blocking reads.
If I had a language with a) only one thread of execution and b)
blocking I/O then you
know what I'd do - I'd invent event based programming.
Let me explain what I think happened ...
Once upon a time there was a sequential processor, that could run only
one sequential program.
ie it interleaved computation with reads and writes.
If the reads blocked - who cared - it's only doing one thing - a
consequence of this
architecture was "the program should not crash" - since there is only
one thread if it crashes
you have a BIG problem - so we invent "defensive programming"
Now we want to do TWO or more things at the same time:
Now - horrors - read stops the system - it blocks, and writes are
problematic we two things
might write the same object at the same time.
Enter - the big global write lock and event based programming
We transform a: into
a: start -> read and on_read_complete(a1)
a1: compute -> read and on_read_complete(a2)
a2: compute -> read and on_read_complete(a3)
etc. and the same for b - the result is the software is a big mess
look at Javascript
$('#result').load('ajax/test.html', function() {
alert('Load was performed.');
});
This is typical jquery javascript code
load("blaa", fun)
means read fun and when the read completes call fun.
Now look what happened:
Simple code like
a: start -> read -> computer -> write -> read ->
got broken into a mess of callbacks and you need a TOOL to help you - enter
FSM modeling the callback style a -> ... read on-read_complete(a1) becomes
a total mess and needs a graphic tool to solve the problem that should never
occurred in the first place.
If read is NON blocking (ie does not block other processes) then you
don't need callbacks.
What about crashing?
When you have ONE thread crashing is a BIG DEAL
But now we have millions of thread - who cares if one crashes - nobody
- "let it crash"
But wait a moment - fault tolerance then? - "let some other guy fix the error"
Who? some other process. Which process? - enter the link - "the linked
processess"
So now we can chuck the idea of defensive programming - don't do it.
Bye bye to 80% of your code that did error checking :-)
Erlang encourages a different mind-set - you can have hundeds and of
thousands of processes
each running sequential easy to write code
Try it - since (again) I don't know what your problem is I can't
really make more than generic observation - but from the Erlang point
of view events and so on are a very "70's" way of programming.
On 11 October 2012 12:38, Raoul Duke <rao...@gmail.com> wrote:
> On Thu, Oct 11, 2012 at 7:17 AM, Rustom Mody <rustompm...@gmail.com> wrote:
>> In the 'normal' world (ie C++, Java, Python) there is a general
>> recommendation that converting a threaded solution to an event-driven
>> solution usually speeds up the program and removes hard-to-find bugs.
> please note that you are talking about problems that exist in those
> paradigms. there are other paradigms that do not have the exact same
> problems (and of course then have their own problems :-). i suspect
> The Erlang Way is different enough that it is probably a mistake to
> just take something for the usual crowd of programming language
> paradigms, and dump it onto erlang.
Let me pitch in an agreement here and on top of this mention that
there are several reasons the "general recommendation" doesn't even
make sense for Erlang:
1. there are no threads in Erlang
2. the standard programming patterns in Erlang (gen_server etc) are
already very event driven
3. the paradigm for Erlang in general is different from C++ or Java
On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong <erl...@gmail.com> wrote:
> I have no idea - you haven't told me what problem you want to solve -
> you've told me the
> technique you wish to use to solve the problem.
> This is a bit "back to front" - I'd like to *start* with the the
> problem then figure out how to solve
> it - not *start* with the idea that you want FSMs and events and so on
> and not tell me what the
> problem is.
Legacy system in C -- Sits on top of an event-oriented middleware.
Since the C system is at core FSM-oriented, it is quite unstructured (from
the normal programming POV)
Ideally one should throw out the mess and start fresh with an appropriate
technology, perhaps Erlang.
Realistically, the stakeholders always prefer evolution to revolution.
If not for anything else, just for unraveling the old-code semantics some
re-engineering is inevitably required
On Thu, Oct 11, 2012 at 8:47 PM, Rustom Mody <rustompm...@gmail.com> wrote:
> On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong <erl...@gmail.com> wrote:
>> I have no idea - you haven't told me what problem you want to solve -
>> you've told me the
>> technique you wish to use to solve the problem.
>> This is a bit "back to front" - I'd like to *start* with the the
>> problem then figure out how to solve
>> it - not *start* with the idea that you want FSMs and events and so on
>> and not tell me what the
>> problem is.
> Legacy system in C -- Sits on top of an event-oriented middleware.
> Since the C system is at core FSM-oriented, it is quite unstructured (from
> the normal programming POV)
Ohhhhh - Ummm - tricky - no easy thing to do here.
New technologies have their best chance a) immediately after a disaster
or b) at the start of a new project.
Cheers
/Joe
> Ideally one should throw out the mess and start fresh with an appropriate
> technology, perhaps Erlang.
> Realistically, the stakeholders always prefer evolution to revolution.
> If not for anything else, just for unraveling the old-code semantics some
> re-engineering is inevitably required
From my perspective, the event loop approach has always been
a bug-ridden nightmare. The idea of turning a multi-thread
program into an event loop one in order to *remove* bugs is,
wow, magno bizarro. I mean, it's like trying to cope with
mice in the house by releasing a lot of large hungry rats.
Yes, they'll eat up the mice, but then you have worse problems.
People who recommend rats (I mean event loops) are thinking
about threads in languages that don't really understand them.
It's hard to imagine a Concurrent Pascal, Occam, or Ada
programmer thinking their program would be improved by turning
it inside out and back to front. But then, Concurrent Pascal,
Occam, and Ada compilers have a clue about what's going on, and
prevent you writing code that has data races and so on in the
first place.
If you want high performance, then in a multicore world turning
threads into an event loop is the very worst thing you can do,
because that one thread is going to run on one core and the
others will sit idle. Again, whatever validity the advice may
once have had was limited to single-core cpus with heavy-weight
threads and high thread switching overheads. Not Erlang.
> On Thu, Oct 11, 2012 at 11:12 PM, Joe Armstrong <erl...@gmail.com <mailto:erl...@gmail.com>> wrote:
> I have no idea - you haven't told me what problem you want to solve -
> you've told me the
> technique you wish to use to solve the problem.
> This is a bit "back to front" - I'd like to *start* with the the
> problem then figure out how to solve
> it - not *start* with the idea that you want FSMs and events and so on
> and not tell me what the
> problem is.
> Legacy system in C -- Sits on top of an event-oriented middleware.
> Since the C system is at core FSM-oriented, it is quite unstructured (from the normal programming POV)
> Ideally one should throw out the mess and start fresh with an appropriate technology, perhaps Erlang.
> Realistically, the stakeholders always prefer evolution to revolution.
> If not for anything else, just for unraveling the old-code semantics some re-engineering is inevitably required
There are 4 main ways to integrate Erlang and C/C++ source code:
- NIF
- port driver
- port
- cnode
The NIF and port driver approach can expose the Erlang VM to any memory problems the C code creates, though it is the most efficient. Usually legacy C or C++ code is best integrated with a port, which is ran as a separate OS process communicating with pipes. I have an open source project here: http://cloudi.org, which can simplify this integration and also provide natural scalability for your legacy source code if: you use the CloudI API (in this case, in C or C++) to communicate with an Erlang CloudI service that handles your external communication that needs to scale. Then you can easily migrate parts of your system, or the whole system, over to Erlang, which will naturally improve your testing, debugging, scalability, reliability, concurrency and the source code size. The integration is similar to integrating with an Erlang port, but it provides features that prevent integration problems, like:
- stdout/stderr handling
- automatic OS process failure handling (similar to supervisor restarts)
- request UUIDs
- requests as transactions (i.e., each request receives a response asynchronously)
- timeouts are enforced for soft-realtime constraints on the external software
- native language integration (both C and C++ have separate interfaces which provide the CloudI API)
As with all open source projects, it needs more documentation, but tell me if you have any questions.
On Fri, Oct 12, 2012 at 9:23 AM, Michael Truog <mjtr...@gmail.com> wrote:
There are 4 main ways to integrate Erlang and C/C++ source code:
> - NIF
> - port driver
> - port
> - cnode
Thanks thats a good set of starting points
> The NIF and port driver approach can expose the Erlang VM to any memory
> problems the C code creates, though it is the most efficient. Usually
> legacy C or C++ code is best integrated with a port, which is ran as a
> separate OS process communicating with pipes. I have an open source
> project here: http://cloudi.org, which can simplify this integration and
> also provide natural scalability for your legacy source code …
Great! I dont yet know our project or Erlang well enough; yet I expect that
this will be a help
> if: you use the CloudI API (in this case, in C or C++) to communicate with
> an Erlang CloudI service that handles your external communication that
> needs to scale. Then you can easily migrate parts of your system, or the
> whole system, over to Erlang, which will naturally improve your testing,
> debugging, scalability, reliability, concurrency and the source code size.
> The integration is similar to integrating with an Erlang port, but it
> provides features that prevent integration problems, like:
> - stdout/stderr handling
> - automatic OS process failure handling (similar to supervisor restarts)
> - request UUIDs
> - requests as transactions (i.e., each request receives a response
> asynchronously)
> - timeouts are enforced for soft-realtime constraints on the external
> software
> - native language integration (both C and C++ have separate interfaces
> which provide the CloudI API)
> As with all open source projects, it needs more documentation, but tell me
> if you have any questions.
As we start making headway I would be happy to contribute patches to docs.
Maybe even code but I am some way off yet :-)
On Oct 10, 2012, at 8:30 PM, Joe Armstrong <erl...@gmail.com> wrote:
> Even if you know how long computations take, and you know what
> resources you have
> to solve the computations, then mapping the computations onto the
> resources involves
> solving the knapsack problem which is NP hard.
One would think so. But in practice, the Knapsack problem is pseudo-polynomial and there exists very efficient algorithms for its solution. They are so fast that they often find the knapsack before an O(n lg n) sort has run
on the items.
So this particular problem is solved.
Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen
On Oct 10, 2012, at 8:37 AM, Rustom Mody <rustompm...@gmail.com> wrote:
> Hello
> We are considering Erlang for a project involving significant data-processing under severe time-crunch
> I am trying to understand the sentence (from http://www.erlang.org/faq/introduction.html#id49576 )
> ----------
> The most common class of 'less suitable' problems is characterised by performance being a prime requirement and constant-factors having a large effect on performance.
> ----------
> How do I understand the '... and constant-factors…'
I think, but may be wrong, that it refers to the fact that Erlang is an interpreted language, which is also dynamically typed. There is no way you get such a language to have as low a constant factor for a computation in general as a solution baked up in C. A good example is if you want to do an FFT on some data. Since you have the interpreter overhead, there is no way to win here. At all. The constant factor on this piece of code is going to be higher.
But there are two things to be said about this: One, a typical program does not spend its time in a small kernel of computation anymore. It is way more complex and has a lot of interaction with the real world. And if there is a small kernel, you can implement the kernel in a different language and get away with it.
Two, I've found that a very large factor of performance stems from the ability to try out clever solutions and be clever about what you do. And in my experience, this is easier to achieve in sane, modern languages like ML, Erlang or (to a certain extent) Haskell. Thus, if you can trade a slow solution for a faster one quickly, then you end up with a faster program in the end. Another factor is time constraint, where a higher level language lets you express the problem quicker.
But Erlang does take some knowledge to make perform.
Jesper Louis Andersen
Erlang Solutions Ltd., Copenhagen
> Two, I've found that a very large factor of performance stems from the
> ability to try out clever solutions and be clever about what you do. And in
> my experience, this is easier to achieve in sane, modern languages like ML,
> Erlang or (to a certain extent) Haskell. Thus, if you can trade a slow
> solution for a faster one quickly, then you end up with a faster program in
> the end. Another factor is time constraint, where a higher level language
> lets you express the problem quicker.
Thanks Jesper for your comments.
I was brought up on lisp and have spent a couple of decades on Haskell and
such languages. So the functional aspect of Erlang is not (likely to be)
an issue for me.
Concurrency however is new!
So any directions of exercises that I could try to get into 'the Erlang
groove' would be helpful.
