[erlang-questions] trouble with erlang or erlang is a ghetto
Joel Reymont
http://www.unlimitednovelty.com/2011/07/trouble-with-erlang-or-erlang-is-ghetto.html
Bring it on!
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Bob Ippolito
resulted in that blog post:
http://erlang.org/pipermail/erlang-questions/2011-July/060104.html
Joel Reymont
On Jul 26, 2011, at 8:20 PM, Bob Ippolito wrote:
> "Funargs: Ruby-like blocks for Erlang" is the discussion thread that
> resulted in that blog post:
Yes, I've been dutifully ignoring it for the past few days :-).
This post warrants a new discussion then!
Joel Reymont
I hate Erlang, although it's been a great way for me to make money. OpenPoker, for example, keeps on giving and is being licensed by a major game development company.
To Tony's list of Erlang warts I would also add the complete lack of transparency during performance optimization. This has bit me during OpenPoker development and also during the "Hot wheels" optimization contest [1].
You do have the facility for call counting (eprof) as well as for per-process profiling with fprof. A regular Erlang system these days will likely have thousands of processes running, processing requests and returning a response. What you get with fprof is a useless profiling listing of thousands of processes.
What you really want is to know what happens along the request processing path. It took N seconds to process the request. Where did this time go? You want to average this over hundreds of requests and see where the bottlenecks are. Do you really want this? You are out of luck, I'm sorry. No tools exist to help you.
That said, I recently came off a project where we build an ad serving platform with OCaml and ZeroMQ. I pushed for this technology stack as I wanted to see if this is a suitable replacement for Erlang. The short answer is no.
We ended up defining tons of Google Protocol Buffers messages for everything we needed and writing boring serialization and deserialization code. Also, what would normally be an Erlang process turned out to be a separate executable. There was some loss of compilation speed as the number of these executables multiplied. Still, it was quite manageable until we got to sharing data.
We started by using plain Redis. This required a server to "front" Redis and process various requests, updating other processes of state changes via ZeroMQ. Serializing things to strings and back is an incredible pain in the ass!
Later, it turned out that Redis was not the ideal choice since a lot of the long-running processes required most of the data in Redis and it was too slow to suck it out upon startup of each of these processes. We ended up dumping data into "bootstrap files", easily and quickly loadable into OCaml.
What I'm trying to say here is that our application clearly benefited from code running "on top" of the data, e.g. Erlang processes using ETS or Mnesia. Our scores of Protobuf messages would have become simple Erlang tuples or records. Data sharing would have become a non-issue.
I wish Tony good luck with his search for the Erlang replacement. I haven't found one yet.
[1] https://groups.google.com/d/topic/erlang-programming/2pRrWneJwG8/overview
Frédéric Trottier-Hébert
--
Fred Hébert
http://www.erlang-solutions.com
Frédéric Trottier-Hébert
--
Fred Hébert
http://www.erlang-solutions.com
Jesper Louis Andersen
> What you really want is to know what happens along the request processing path. It took N seconds to process the request. Where did this time go? You want to average this over hundreds of requests and see where the bottlenecks are. Do you really want this? You are out of luck, I'm sorry. No tools exist to help you.
I agree this thing would be really neat to have. To make it somewhat
efficient, I'd definitely propose an sFlow-like approach. Add a flag
to erlang:trace/3 such that we can trace a function based upon a
sampling value. For instance, that 1/8192 messages on average is
traced. Together with call/return we now know how much time was spent
in that function as a whole. If you also add dependent trace probes
(if the 1/8192 trigger, then these should be enabled as well) you
should in principle be able to build such a tool out of the trace
facilities. It is *almost* there.
--
J.
Joel Reymont
On Jul 26, 2011, at 9:23 PM, Jesper Louis Andersen wrote:
> Add a flag to erlang:trace/3 such that we can trace a function based upon a
> sampling value. For instance, that 1/8192 messages on average is
> traced. Together with call/return we now know how much time was spent
> in that function as a whole.
There may be message passing along the request path, processes talking to one another.
You want to capture and time this interaction.
You can do all this by capturing and processing gigabytes of trace messages, figuring out where the request starts and ends and taking it from there.
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Loïc Hoguin
> Did I miss a lively and heated discussion?
>
> http://www.unlimitednovelty.com/2011/07/trouble-with-erlang-or-erlang-is-ghetto.html
>
> Bring it on!
All I'm seeing there is "Make Erlang like language XYZ". Not that Erlang
can't learn from other languages and platforms, I'm sure it does, but
Erlang focuses on reliability and you can't be reliable and add new
trendy features every year while remaining reliable. Adding features to
Erlang is a long and progressive process and this sort of development
pace doesn't with the "following trends" approach most programmers use
with regards to programming languages. Two days ago Java was trendy,
yesterday it was Ruby, today it's Javascript, tomorrow it'll be another,
everytime reinventing the loop or partially taking from other languages.
Personally I like the slow but certain approach better.
So yeah, maybe Erlang doesn't have the best features of Lisp, and
Clojure, and Java, and Ruby, and Javascript, and-- but neither do them.
I'm a firm believer of the "right tool for the right job", and it turns
out that Erlang actually is good for most server application jobs, so it
pleases me quite well. And it doesn't need to have 10 million features
to fit that role, it just needs to be reliable, concurrent and distributed.
I disagree about Erlang's syntax though. Not that it doesn't suck, but I
don't know any language syntax which isn't sucking (I'm sure you'll tell
me your favorite language has great syntax). They all have issues, they
all have traps you must learn to avoid, and nothing you can do will
change that. Maybe you can add some syntax sugar to avoid typing 5
characters here and there, but seriously what's the point? If the syntax
changes don't make me think differently about the way I program, and
don't allow me to do things I couldn't do before, then they're pretty
much irrelevant to the matter at hand: getting things done. There *are*
interesting concepts that could be good, one day, in Erlang. But not
having them doesn't prevent us from being efficient so they should go
after making Erlang even more reliable, and also after adding missing
library components to the pool of applications. Unless you're doing your
project for fun or something, in which case have fun. But toying with
syntax doesn't make much sense if you need to bring in money.
Erlang does very well what it's designed for, and I'm thankful for that.
I'm using Erlang to do a lot of things. I wouldn't use Erlang to do
everything. But Erlang allows me to get the work done and more
importantly it allows me to move on to other projects quickly because
Erlang provides me with everything needed to avoid almost all possible
bugs (Dialyzer, PropEr, eunit and ct are wonderful to work with), and
once in production I'm confident my applications will always be running
and available. That's something I can't say for any other language or
platform out there today.
--
Loïc Hoguin
Dev:Extend
Jesper Louis Andersen
>
> On Jul 26, 2011, at 9:23 PM, Jesper Louis Andersen wrote:
>
>> Add a flag to erlang:trace/3 such that we can trace a function based upon a
>> sampling value. For instance, that 1/8192 messages on average is
>> traced. Together with call/return we now know how much time was spent
>> in that function as a whole.
>
> There may be message passing along the request path, processes talking to one another.
>
> You want to capture and time this interaction.
Ah yes, that is something I completely missed. Indeed you want to
trace along messages also, but that makes it less obvious what to do.
I am more for capturing such instrumentation information on a random
sampling basis. You could do it manually though by inserting
instrumentation functions along the path and then use those functions
as hooks for a unique tag. That is probably what I'd do today if I
needed this. Then I'd trace the instrumentation functions at random
intervals.
--
J.
Tim Watson
>
> You do have the facility for call counting (eprof) as well as for per-process profiling with fprof. A regular Erlang system these days will likely have thousands of processes running, processing requests and returning a response. What you get with fprof is a useless profiling listing of thousands of processes.
>
> What you really want is to know what happens along the request processing path. It took N seconds to process the request. Where did this time go? You want to average this over hundreds of requests and see where the bottlenecks are. Do you really want this? You are out of luck, I'm sorry. No tools exist to help you.
>
Better profiling tool support would be nice. A few people started
trying to pull together something that would provide better visibility
- my own project, nodewatch, which is heavily based on eper and more
recently Richard Jones and friends produced
https://github.com/beamspirit/bigwig during the spawnfest competition.
I've stopped work on nodewatch for now, to see whether bigwig will
move faster - I don't have that much spare time and if others are
doing the same thing as their day job then I needn't bother. My
ultimate aim was to provide configurable aggregated stats as well as
trace support with time budgeting in mind (e.g., how long in the web
tier, how long hitting the back end, etc).
Jon Watte
--
Americans might object: there is no way we would sacrifice our living standards for the benefit of people in the rest of the world. Nevertheless, whether we get there willingly or not, we shall soon have lower consumption rates, because our present rates are unsustainable.
Richard O'Keefe
On 27/07/2011, at 9:05 AM, Loïc Hoguin wrote:
> On 07/26/2011 09:07 PM, Joel Reymont wrote:
>> Did I miss a lively and heated discussion?
>>
>> http://www.unlimitednovelty.com/2011/07/trouble-with-erlang-or-erlang-is-ghetto.html
>>
>> Bring it on!
There are some substantive issues in that blog entry.
(1) Frames have not been implemented yet.
Now if I had provided a model implementation, things might have been
different. Or if Joe had provided a model implementation of his
earlier and fundamentally similar "proper structs", again things
might have been different.
My excuse is that the BEAM architecture is simply not documented
anywhere that I can find, and I have too many other things to do
to grovel around in the guts in Erlang to figure it out. So let
me add my item here:
(2) BEAM has no usable documentation.
One reason this is important is because some people have liked the
ideas underneath Erlang well enough to try to build other syntaxes
for it. But that is inexcusably hard with BEAM undocumented.
(3) There doesn't seem to be anything in the Erlang _approach_ that
should interfere with scaling, but the _implementation_ appears
not to scale well much past 16 cores.
I don't know if that is current information. If true, it's an
important limitation of the implementation which I'm sure will be
given a lot of attention. I don't expect it to stay true.
(4) He doesn't seem to like the Erlang garbage collector, but that's
something which has changed more than once, and he does not
offer any actual _measurements_.
I tried the experiment of allocating 1,000,000,000 list cells
(but never keeping more than 10,000 of them at a time).
Erlang, byte-codes: 7.58 seconds (= 7.58 nsec/allocation).
Erlang, native : 3.92 seconds (= 3.92 nsec/allocation).
Java -O -server : 11.40 seconds (= 11.40 nsec/allocation).
Java -O -client : 12.26 seconds (= 12.26 nsec/allocation).
Java has come a long way. I don't have an Azul system to try.
He praised tcmalloc. I note that it only recently became usable
without pain on my laptop (MacOS X) and that building it produced
reams of warning messages about using a deprecated interface, so
it may not work much longer. It doesn't work at all on the other
machine on my desk. (Erlang works on both.) I wrote a similar
benchmark in C and linked it with libtcmalloc.a. I killed that
program after it had run for more than 10 times as long as the
Erlang code. So when he says "Erlang ... can't take advantage of
libraries like tcmalloc", there doesn't appear to be any
advantage that Erlang *could* take.
In short, Erlang's memory management is criticised, and it MAY be
that this is justified, but the blog entry provides no EVIDENCE.
By the way, savour the irony. "Erlang's approach [of] using
separate heaps per process", which he criticises, is in fact
used elsewhere: ptmalloc does it, the tcmalloc documentation
makes it absolutely clear that tcmalloc does this also (more
precisely, it uses a per-thread cache, which is what the "tc"
part of the name means), and some recent Java systems have
done the same thing, with a per-thread cache for memory
management, backed by a shared heap. The point of the per-
thread cache is to reduce locking. There is a spectrum of
approaches from nothing shared to everything shared, and it
seems clear that everyone sees merit in not being at either
extreme. Progress must be driven by measurement.
(5) He doesn't like HiPE. For myself, I don't _care_ whether HiPE is
a JIT or a jackal, as long as it gives me a useful improvement in
performance. Inlining across module boundaries _has_ been tried,
I believe (there's a paper about it somewhere), but it's hard to
reconcile with hot loading. HiPE was, of course, a project
contributed by "the community", and depended on funding which I
believe has come to an end. Anyone who wants a better compiler
should try to find funds for it. Sun and Google have vastly
deeper pockets than Kostis Sagonas!
I think it is particularly unfair to criticise HiPE for having a
limited range of back ends when it works on *MORE* systems than
the tcmalloc library he praised.
(6) Erlang is not general purpose.
But it was never intended to be, and isn't advertised as such.
In fact Erlang loves state, but it wants state to be encapsulated
within processes.
"What should you do if you want to deal with a shared-state
concurrency program in Erlang?"
Lie down until the feeling passes off.
If you want shared-state concurrency, I can tell you where to find
Ada. I can tell you where to find concurrent ML (Mlton does it OK).
I can tell you where to find Haskell, which is actually pretty
amazing these days.
One can respect Erlang without being married to it.
(7) He doesn't like the syntax.
Well, it's not quite as much of a disaster as Java syntax, and for
sure it's not as ugly as CAML or F#. (They are so ugly that they
make SML look beautiful, and for someone who prefers Haskell to
SML on aesthetic grounds, that's saying a lot.) The funny thing
is that what makes Erlang syntax clunky is precisely its *similarity*
to classical languages like Pascal and C...
Given documentation for BEAM, we might get more alternative syntaxes
to play with.
It's fair to point out that Erlang resulted from an experiment with
several approaches, so responsible steps were taken to make sure that
it wasn't _too_ bad.
(8) He criticises immutable state on the grounds that while you can share
tails of a list, you can't share prefixes or infixes. The answer, of
course, is multifold:
- there are immutable data structures where you *can* share infixes
and you *can* use them in Erlang, it's just that they don't have
built in syntax. (For that matter, it would be possible to implement
Erlang so that slices of tuples could be shared just like slices of
strings in Java. SML does this. In fact, Concurrent SML would answer
so many of his issues that I'm surprised he didn't mention it.)
- this is only a problem if you *want* to share prefixes or infixes,
and somehow I never do
- in languages with mutable state you cannot safely share ANYTHING.
The argument that allocating pure objects is a bad match for modern
hardware is a non sequitur. Let me quote a paper about Fork/Join
parallelism for Java: "In many ways, modern GC facilities are perfect
matches to fork/join frameworks: These programs can generate enormous
numbers of tasks, nearly all of which quickly turn into garbage after
they are executed." That is, generating enormous amounts of garbage
can be a >good< thing, provided it's the kind that garbage collectors
manage well. I've been on the garbage collection mailing list for a
while, and the Memory Management proceedings have contained papers
showing that garbage collection can be *worse* for locality (and thus
modern hardware) and papers showing that it can be *better* for
locality (and thus modern hardware). What this means is that one
cannot simply *assume* "side effects = good for cache, immutability
= bad", one must *measure*.
(9) He doesn't like the standard library.
Interestingly enough, I hear the same kind of thing in the Haskell
mailing list about the Haskell "standard Prelude". And there is a
project to develop a new standard Prelude.
I believe there is general agreement that an improved Erlang library
could be developed and would be very nice to have. (I've always
found the differences between say ETS and DETS more confusing than
helpful.)
This is something that can be done piecemeal and by individuals.
The things I would like to say about the Java libraries would have to
be displayed on asbestos screens... Heck, I like the Ada libraries
less than I used to; the additions are *pointful* but not to my mind
*tasteful*.
And so it goes.
There *are* things about Erlang that can be improved.
Some things *have* been improved, some things are being improved,
and some things don't need anyone to wait for Ericsson to do them.
Michael Truog
> (3) There doesn't seem to be anything in the Erlang _approach_ that
> should interfere with scaling, but the _implementation_ appears
> not to scale well much past 16 cores.
>
> I don't know if that is current information. If true, it's an
> important limitation of the implementation which I'm sure will be
> given a lot of attention. I don't expect it to stay true.
Any problems scaling on systems with more than 16 cores just seems to be related to the current cost of those systems (would love to know what the tests currently show on the Tileras > 16 cores though). Erlang appears to have much more natural scalability when you compare it to Java, and the criticism of the Erlang garbage collector is unsubstantiated (as previously mentioned). You can not say an approach is wrong because it isn't the Java-way, or perhaps the Sun-way. The per-process garbage collection avoids central state, so it encourages scalability with a design for parallelism. Throwing a ton of money at Azul to push a single heap garbage collector beyond normal limits might sound fun, but it only shows how long and drawn out technical failure can be (like Sun's stock price was for instance, an almost perfect bell curve).
It is hard to believe that there is controversy still here. Per-process garbage collection avoids shared state which avoids the need for low-level locking which makes Erlang scalable. To argue for some other approach in Erlang seems like idiocy to me, because it wouldn't be a real Actor model that can provide fault-tolerance (actually keep failures isolated). Why would you want to fool around with some broken fake Actor implementation that is unscalable (like your operating system, for instance)? Seems like a waste of time, just like this garbage collector complaint.
Yes, HiPE has issues, Erlang is not meant for all programs, and the syntax is different. Nothing is perfect.
Not even the standard library is perfect. I have yet to hear of a perfect standard library in any language. I think having problems with a standard library is a natural problem because the people that write the standard library impose a taxonomy on functionality that not everyone shares naturally, because we are not psychic. Learning the standard library just seems like a natural process in computer science when you get to relate a standard library to the others you had the misfortune to use in the past. Why bother complaining about a taxonomy that is just as bad as any other? If it serves the purpose it was designed for well, then there is no reason to care, it just happens to be different from what you might expect based on your own limited knowledge.
> And so it goes.
>
> There *are* things about Erlang that can be improved.
> Some things *have* been improved, some things are being improved,
> and some things don't need anyone to wait for Ericsson to do them.
The frustration with these issues seems natural and common, both on this mailing list and elsewhere. However, I think it is important to be proactive rather than giving in to emotional arguments that lack justification or evidence.
Ulf Wiger
On 27 Jul 2011, at 08:08, Michael Truog wrote:
> On 07/26/2011 09:42 PM, Richard O'Keefe wrote:
>> (3) There doesn't seem to be anything in the Erlang _approach_ that
>> should interfere with scaling, but the _implementation_ appears
>> not to scale well much past 16 cores.
>>
>> I don't know if that is current information. If true, it's an
>> important limitation of the implementation which I'm sure will be
>> given a lot of attention. I don't expect it to stay true.
>
> Any problems scaling on systems with more than 16 cores just seems to be related to the current cost of those systems (would love to know what the tests currently show on the Tileras > 16 cores though).
I guess it's fair to say that the Erlang/OTP doesn't have a "dangerously sexy" approach to SMP scalability. They recognise that their sponsors [1] would have their heads if BEAM started hanging or dumping core in the sole interest of scaling to ridiculously many cores on hardware that none of the sponsors are using. Instead, they try to deliver incremental improvements without ever sacrificing the rock-solid performance of BEAM that Erlang users have come to expect. They're not batting 100% in that regard, but close enough to be very impressive, IMHO.
They *have* run benchmarks on 64-core machines, but mainly to learn about what's around the corner, and understand what changes will be needed to get there. You will soon hear about some pretty exciting developments in the area of Erlang for massive scalability, but there is a time and a place for everything… ;-)
What is interesting for most Erlang users (I think), is not how well Erlang can scale with synthetic benchmarks, but on *real applications*, where lots of other factors affect the result, and scalability is only one parameter (albeit a very important one).
Having had the privilege of participating in key roles throughout the development lifecycle of a number of telecom-class Erlang-based systems, I've come to think that constant-factor nuisances (like syntax quibbles) are simply unimportant. What matters in these projects is that you get the fundamentals right, esp. in regard to messaging, fault-tolerance and provisioning. If you don't, you will never be able to control schedules and costs, and your product is unlikely to ever be profitable.
In this environment, very few programming languages are even considered: usually, it's either C/C++ or Java, and in some companies, Erlang as well. The .NET languages are usually ruled out because their commercial viability is largely tied to the Windows platform, and such a single-platform focus is an issue when you expect your product to serve for decades. Ruby, Python etc. can be useful for writing test scripts, and perhaps some supporting tools, or perhaps even some peripheral component in the system, but (at this point in time) not much more than that.
I realise that most people today are not in this environment, but rather developing small [2] projects, often in areas where Ruby, Python, Clojure, Scala, Haskell et al are perfectly viable alternatives. I think it's great that the Erlang/OTP team is getting feedback from this crowd, and even pressure to compete with the "coolest" languages out there. Erlang will be better for it.
But let's do this with some mutual respect. I'm not even going to begin commenting on the "Erlang cargo culters" references, and find it curious that Erlang is so often said to "suck", even though the same people often admit that there is no viable replacement - isn't this much more of a criticism against those other languages, which apparently don't just suck enough to give discomfort, but to the point of being near unusable in this realm?
Having been in the position, some years ago, where I was thinking about changing jobs, and couldn't really find any jobs where I could make good use of my Erlang experience (even if not in Erlang), I find today's software market much more exciting - where other languages are drawing inspiration from Erlang, and in some cases really challenging it, even in the concurrency domain.
[1] Sponsors = commercial projects within Ericsson that use Erlang to build 5-nines-style complex messaging products for commercial gain. They typically need to use NEBS-compliant ATCA hardware from reputable vendors, which is standardised across the corporation to minimise supply chain costs. In these settings, I would expect most systems to be running on 4- or 8-core machines today, perhaps starting to move up to 16-core in a few places.
[2] To me, anything under 100 KLOC is small.
BR,
Ulf W
Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com
Max Lapshin
<ulf....@erlang-solutions.com> wrote:
>
> I guess it's fair to say that the Erlang/OTP doesn't have a "dangerously sexy" approach to SMP scalability. They recognise that their sponsors [1] would have their heads if BEAM started hanging or dumping core in the sole interest of scaling to ridiculously many cores on hardware that none of the sponsors are using. Instead, they try to deliver incremental improvements without ever sacrificing the rock-solid performance of BEAM that Erlang users have come to expect. They're not batting 100% in that regard, but close enough to be very impressive, IMHO.
>
Many people forget about it. Erlang messaging system seems to be
99,999% bug free. Scala is full of silly bugs.
I'd preferer to build software, that I sell on bug free platform and
great thanks to your team for it.
The only thing that really affects me is lack of profiling tools or
lack of documentation on them.
Thomas Lindgren
----- Original Message -----
> From: Ulf Wiger <ulf....@erlang-solutions.com>
> To: Michael Truog <mjt...@gmail.com>
> Cc: "erlang-q...@erlang.org Questions" <erlang-q...@erlang.org>
> Sent: Wednesday, July 27, 2011 9:16 AM
> Subject: Re: [erlang-questions] trouble with erlang or erlang is a ghetto
>
> ...
> I guess it's fair to say that the Erlang/OTP doesn't have a
> "dangerously sexy" approach to SMP scalability. They recognise that
> their sponsors [1] would have their heads if BEAM started hanging or dumping
> core in the sole interest of scaling to ridiculously many cores on hardware that
> none of the sponsors are using.
Furthermore, IMO it's something of a mistake to obsess over single-process SMP. I think distributed erlang is pretty sexy.
Best,
Thomas
Hendrik Visage
<thomasl...@yahoo.com> wrote:
> ----- Original Message -----
>> From: Ulf Wiger <ulf....@erlang-solutions.com>
>
>> ...
>> I guess it's fair to say that the Erlang/OTP doesn't have a
>> "dangerously sexy" approach to SMP scalability. They recognise that
>> their sponsors [1] would have their heads if BEAM started hanging or dumping
>> core in the sole interest of scaling to ridiculously many cores on hardware that
>> none of the sponsors are using.
>
>
> Furthermore, IMO it's something of a mistake to obsess over single-process SMP. I think distributed erlang is pretty sexy.
And *that* my dear friends, is where the "clouds" are hanging out ;)
A rack in each city centre with a bunch of 1u/2u boxes, using
something like GoogleFS, and we have close to a petabytes of
distributed disaster tolerant storage and processing, and then I also
say: Bring me my Erlang :)
Dmitrii Dimandt
On 27 Jul 2011, at 08:08, Michael Truog wrote:On 07/26/2011 09:42 PM, Richard O'Keefe wrote:(3) There doesn't seem to be anything in the Erlang _approach_ thatshould interfere with scaling, but the _implementation_ appearsnot to scale well much past 16 cores.I don't know if that is current information. If true, it's animportant limitation of the implementation which I'm sure will begiven a lot of attention. I don't expect it to stay true.Any problems scaling on systems with more than 16 cores just seems to be related to the current cost of those systems (would love to know what the tests currently show on the Tileras > 16 cores though).
I guess it's fair to say that the Erlang/OTP doesn't have a "dangerously sexy" approach to SMP scalability. They recognise that their sponsors [1] would have their heads if BEAM started hanging or dumping core in the sole interest of scaling to ridiculously many cores on hardware that none of the sponsors are using. Instead, they try to deliver incremental improvements without ever sacrificing the rock-solid performance of BEAM that Erlang users have come to expect. They're not batting 100% in that regard, but close enough to be very impressive, IMHO.
They *have* run benchmarks on 64-core machines, but mainly to learn about what's around the corner, and understand what changes will be needed to get there. You will soon hear about some pretty exciting developments in the area of Erlang for massive scalability, but there is a time and a place for everything… ;-)
dmi...@dmitriid.com
Erlang in Russian
http://erlanger.ru/
Ulf Wiger
On 27 Jul 2011, at 12:27, Dmitrii Dimandt wrote:
>> They *have* run benchmarks on 64-core machines, but mainly to learn about what's around the corner, and understand what changes will be needed to get there. You will soon hear about some pretty exciting developments in the area of Erlang for massive scalability, but there is a time and a place for everything… ;-)
>>
>
>
> Also don't forget that there's a grant given to universities to further develop Erlang to become massively multicore, as Kostis Sagonas said at Erlang Factory in London
Well, that's what I was referring to, but there will be a more coordinated announcement soon. :)
…That, and another one, with a different slant.
Lukas Larsson
Lukas
[1] http://kth.diva-portal.org/smash/record.jsf?pid=diva2:392243
----- Original Message -----
From: "Michael Truog" <mjt...@gmail.com>
To: "erlang-q...@erlang.org Questions" <erlang-q...@erlang.org>
Sent: Wednesday, 27 July, 2011 8:08:15 AM
Subject: Re: [erlang-questions] trouble with erlang or erlang is a ghetto
Valentin Micic
Yes, HiPE has issues, Erlang is not meant for all programs, and the syntax is different. Nothing is perfect.
Mihai Balea
On Jul 27, 2011, at 7:29 AM, Ulf Wiger wrote:
>
> On 27 Jul 2011, at 12:27, Dmitrii Dimandt wrote:
>
>>> They *have* run benchmarks on 64-core machines, but mainly to learn about what's around the corner, and understand what changes will be needed to get there. You will soon hear about some pretty exciting developments in the area of Erlang for massive scalability, but there is a time and a place for everything… ;-)
>>>
>>
>>
>> Also don't forget that there's a grant given to universities to further develop Erlang to become massively multicore, as Kostis Sagonas said at Erlang Factory in London
>
> Well, that's what I was referring to, but there will be a more coordinated announcement soon. :)
>
>
> …That, and another one, with a different slant.
Are you referring to this announcement?
http://www.erlang-solutions.com/press-releases/3/entry/1253
Mihai
Dmitrii Dimandt
On Jul 27, 2011, at 3:58 PM, Mihai Balea wrote:
>
> On Jul 27, 2011, at 7:29 AM, Ulf Wiger wrote:
>
>>
>> On 27 Jul 2011, at 12:27, Dmitrii Dimandt wrote:
>>
>>>> They *have* run benchmarks on 64-core machines, but mainly to learn about what's around the corner, and understand what changes will be needed to get there. You will soon hear about some pretty exciting developments in the area of Erlang for massive scalability, but there is a time and a place for everything… ;-)
>>>>
>>>
>>>
>>> Also don't forget that there's a grant given to universities to further develop Erlang to become massively multicore, as Kostis Sagonas said at Erlang Factory in London
>>
>> Well, that's what I was referring to, but there will be a more coordinated announcement soon. :)
>>
>>
>> …That, and another one, with a different slant.
>
> Are you referring to this announcement?
>
> http://www.erlang-solutions.com/press-releases/3/entry/1253
Oh wow. Congrats to Erlang Solutions!
Ulf Wiger
Yeah, something like that. Being on vacation, I haven't kept up with the announcements. :)
BR,
Ulf W
On 27 Jul 2011, at 14:58, Mihai Balea wrote:
>
> On Jul 27, 2011, at 7:29 AM, Ulf Wiger wrote:
>
>>
>> On 27 Jul 2011, at 12:27, Dmitrii Dimandt wrote:
>>
>>>> They *have* run benchmarks on 64-core machines, but mainly to learn about what's around the corner, and understand what changes will be needed to get there. You will soon hear about some pretty exciting developments in the area of Erlang for massive scalability, but there is a time and a place for everything… ;-)
>>>>
>>>
>>>
>>> Also don't forget that there's a grant given to universities to further develop Erlang to become massively multicore, as Kostis Sagonas said at Erlang Factory in London
>>
>> Well, that's what I was referring to, but there will be a more coordinated announcement soon. :)
>>
>>
>> …That, and another one, with a different slant.
>
> Are you referring to this announcement?
>
> http://www.erlang-solutions.com/press-releases/3/entry/1253
>
> Mihai
Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com
_______________________________________________
Ulf Wiger
Are you referring to this announcement?http://www.erlang-solutions.com/press-releases/3/entry/1253
Oh wow. Congrats to Erlang Solutions!
Ulf Wiger
On 27 Jul 2011, at 18:58, Richard Bucker wrote:
> There is nothing in the language or VM that makes it's sigma rating any better or worse than any other language or framework.
Erlang doesn't pretend to have a sigma rating. What language does (except perhaps in some very narrow context)?
Erlang has been used to build systems with better than 5-nines availability, according the rules for measuring In-Service Performance (ISP) in Telecoms. That's what it was made for.
> Indeed Mnesia and Mnesia's replication cannot add to it's rating as it fails all the time. And hello-world is not an acceptable app to test or rate.
As it happens the system responsible for the 99.99999% availability figure floating around [1], did in fact use mnesia. Klarna uses mnesia, and has best-in-class availability among online factoring services in Scandinavia (I believe - I have no hard data, but their CTO was pretty confident at the last Erlang Factory in London). There are other examples, and just about every database used in anger has its share of horror stories too. Perhaps you shouldn't believe everything you read in the blogosphere? ;-)
[1] That was a real data point, made official by British Telecom - which is why it could be used by Joe in a talk at MIT. The average ISP reported from all our deployments was lower, but consistently better than 99.999% while I was keeping track. I cannot give you the exact numbers.
> Erlang solves some interesting problems and creates some new ones. It's more of a "religion of functionalism" than anything else. And for the same reasons it's a little dangerous and risky.
I'm not sure what you base that on. Erlang's history has clearly been driven by pragmatism, and has made it this far mainly because it proved excellent for solving the problems it was designed for.
But I admit - Erlang has gone through some different phases:
* from being young and hyped,
* to being banned, bashed and nearly killed,
* to seeing some kind of truce, where it was tolerated, as long as its proponents didn't try to make a big deal out of it,
* to wooing part of the Open Source crowd and seeing some hype again (and some bashing too, for good measure).
Having experienced all of those, I have long since given up on trying to convince anyone who doesn't want to be convinced. For the purpose of the list, I will try to comment on things that I find incorrect, and share my experience when asked.
If you don't see anything special about Erlang, and consider using it a risk, I'd assume the sensible thing is not to use it. Are you willing to entertain the possibility that other people reach a very different conclusion, without being blinded by religious fervour?
BR,
Ulf W
Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com
_______________________________________________
Max Lapshin
"best garbage collection", but one serious problem is really
mentioned.
It is lack of data structure, which can survive upgrade of only one
part of software.
I can't create redistributable plugins for erlyvideo that can use
headers with record description, I have to write such things:
ems_media:get(Media, last_dts)
and frankly speaking I don't understand how do other people live without this.
I understand, that C language has the same problem, but maybe there
are some ideas to make some delayed instantiation of record usage in
modules?
I.e. add "record_get(Record, name)" instruction to beam, which should
be translated to "element(Record, N)" on loading?
Joel Reymont
On Jul 27, 2011, at 9:36 PM, Max Lapshin wrote:
> I can't create redistributable plugins for erlyvideo that can use
> headers with record description, I have to write such things:
> ems_media:get(Media, last_dts)
> and frankly speaking I don't understand how do other people live without this.
What exactly is the problem with this?
Records are just tuples, no?
P.S. I don't think one needs to post to the Google group as it mirrors this list.
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Max Lapshin
>
> What exactly is the problem with this?
>
> Records are just tuples, no?
>
Problem happens, when plugin is compiled against version 2, where
record rtmp_session was looking like:
#rtmp_session{pid, user_id}
but launched against version 3, where rtmp_session is:
#rtmp_session{pid, user_id, session_id}
Yes, records are only tuples and their field names are lost in runtime.
Loïc Hoguin
>
> On Jul 27, 2011, at 9:36 PM, Max Lapshin wrote:
>
>> I can't create redistributable plugins for erlyvideo that can use
>> headers with record description, I have to write such things:
>> ems_media:get(Media, last_dts)
>> and frankly speaking I don't understand how do other people live without this.
>
>
> What exactly is the problem with this?
>
> Records are just tuples, no?
I think that's what confuses most people. They see records as something
more, when they're just tuples and aren't designed to be anything else.
Records need to be upgraded the same as tuples when changes are made and
of course it takes a little more care than struct/frames would.
--
Loïc Hoguin
Dev:Extend
Ulf Wiger
Richard O'Keefe
On 27/07/2011, at 6:08 PM, Michael Truog wrote:
>
> Not even the standard library is perfect. I have yet to hear of a perfect standard library in any language. I think having problems with a standard library is a natural problem because the people that write the standard library impose a taxonomy on functionality that not everyone shares naturally, because we are not psychic. Learning the standard library just seems like a natural process in computer science when you get to relate a standard library to the others you had the misfortune to use in the past. Why bother complaining about a taxonomy that is just as bad as any other? If it serves the purpose it was designed for well, then there is no reason to care, it just happens to be different from what you might expect based on your own limited knowledge.
Let me offer a two-fold contrast with Eiffel here.
When Betrand Meyer designed Eiffel, he didn't just design a language,
he designed a *consistent* naming convention to be applied across
*all* library classes. From time to time this got revised, but it
did mean that there was a *documented* convention: once you had learned
the names for one class, you knew a great deal about what the names in
another class meant.
I tried to follow a similar policy in the Quintus Prolog library.
That's the kind of "inconsistency" people talk about when they criticise
the Erlang/OTP libraries: same name for different things, different names
for same things.
That's the contrast that makes Eiffel look good and Erlang look bad.
Now for the other contrast.
The Eiffel libraries were proprietary, so other Eiffel implementors
had to come up with others. Eventually the N.I.C.E. came up with a
common but minimal library. Oy was it minimal! The nearest thing
there was to a _useful_ Eiffel library was the GOBO library, and one
of the Eiffel implementations (as it happened, the one I used) changed
too much for the GOBO maintainer to keep up (or me; I stopped using
Eiffel).
The *coverage* of a library can be more important than its *consistency*.
And to be honest, I think coverage is going to drive Erlang/OTP library
development more than consistency is. If someone wants to use SCTP, it
is probably more important to them to have a finished SCTP module they
can use now than to have a beautiful module they might be able to use
next year, maybe.
It might be interesting to run some sort of poll to see what currently
unsupported areas people care about most, particularly the people who
would say "I'd like to use Eiffel but I cannot because it does not
have library support for <whatever>."
Loïc Hoguin
> It might be interesting to run some sort of poll to see what currently
> unsupported areas people care about most, particularly the people who
> would say "I'd like to use Eiffel but I cannot because it does not
> have library support for <whatever>."
Proper Unicode support in Erlang would probably get many votes.
--
Loïc Hoguin
Dev:Extend