Erlang vs Clojure

[Robin]

Erlang: The Movie <=> Clojure: The Podcast
Single Assignment <=> Immutable Data Structures
Mnesia <=> STM
ErlangVM <=> JVM
Hipe <=> JIT
Pattern Matching <=> Multimethods
Erlang Shell <=> REPL
Hot Code Reload <=> Dynamic Compilation
Behaviours <=> Extensible Abstractions
Tail Recursion <=> recur
fun <=> fn
syntax from 1987 <=> syntax from 1958
EMP2 <=> CL style macros
Fiber/Actor <=> Termite?

The burning question:

Can Termite be ported to Clojure?

[Rich Hickey]

On Nov 22, 4:57 pm, Robin <robi...@gmail.com> wrote:
> Erlang: The Movie <=> Clojure: The Podcast
> Single Assignment <=> Immutable Data Structures
> Mnesia <=> STM
> ErlangVM <=> JVM
> Hipe <=> JIT
> Pattern Matching <=> Multimethods
> Erlang Shell <=> REPL
> Hot Code Reload <=> Dynamic Compilation
> Behaviours <=> Extensible Abstractions
> Tail Recursion <=> recur
> fun <=> fn
> syntax from 1987 <=> syntax from 1958
> EMP2 <=> CL style macros
> Fiber/Actor <=> Termite?
>

It's a fun comparison, but one I'd like to be careful about. Clojure
has a different philosophy about concurrency than does Erlang, neither
being right or wrong, but yielding somewhat different results as you
encounter each decision point in the design. The difference is, I
think, (and I'm hesitant to speak for Erlang, which I quite respect
and am no expert on):

In Erlang the concurrency model is (always) a distributed one and in
Clojure it is not.

I have some reservations about unifying the distributed and non-
distributed models (see e.g. http://research.sun.com/techrep/1994/smli_tr-94-29.pdf),
and have decided not to do so in Clojure, but I think Erlang, in doing
so, does the right thing in forcing programmers to work as if the
processes are distributed even when they are not, in order to allow
the possibility of transparent distribution later, e.g. in the failure
modes, the messaging system etc. However, issues related to latency,
bandwidth, timeouts, chattiness, and costs of certain data structures
etc remain. My experiences with transparent distribution were with COM
and DCOM, and, quite frankly, not happy ones. I think Erlang has a
much better story there, but the fact is that distributed programming
is more complex, and I personally wouldn't want to incur that
complexity all the time. I think it is ok for a system designer to
decide one part of a system will be distributed and another not, and
incur some rewrite if they are wrong. If I wrote phone switches I
might think otherwise :)

> The burning question:
>
> Can Termite be ported to Clojure?

Clojure has a synchronous reference mutation model in its STM and I am
wrapping up an asynchronous reference (actor) model for Clojure. The
basic idea is that actors are mutable references to immutable values.
An actor reference can be made to refer to a new immutable value
(only) by sending it a message. Messages are functions (and,
optionally, additional arguments) that are applied to an actor's value
and whose return value becomes the actor's new value. Message sends
return immediately and the actual work happens asynchronously in a
thread pool. Because messages are functions they can also be
multimethods and therefore messages are potentially polymorphic. Also,
because the set of functions is open, the set of messages supported by
an actor is also open, a sharp contrast to pattern matching message
handling loops. Another significant feature of Clojure's actors is
that the value of an actor is always immediately available for reading
without any messages, i.e. observation does not require cooperation/
coordination. This represents, IMO, a substantial reduction in
complexity versus other models, but is incompatible with distribution.
Clojure actors are integrated with its STM - any messages sent in a
transaction are held until it commits, and are discarded if it is
retried or aborted. Clojure's actors are reactive - there is no
imperative message loop and no blocking receive.

Even with actors, Clojure will not yet have a distributed concurrency
story, but I am considering just adopting Erlang's wholesale, using
Jinterface for Clojure<->Clojure or even Clojure<->Erlang distributed
processes. Maybe that will look like Termite when it is done. Stay
tuned.

Rich

[Robin]

The ErlangVM has a scheduler and fly-weight processes on the order of
300 bytes each, so you can have hundreds of thousands of tasks in
flight on conventional hardware.

If I understand Clojure actors correctly, actors will not need a
dedicated stack, heap, or mailbox, so an actor could be as simple as a
reference (4 bytes on 32 bit machine), so you could theoretically have
tens of millions of actors in flight and process them with a
conventional thread pool?

Wow, can't wait to try it out.

[Rich Hickey]

Well, they are bigger than 4 bytes each, having a per-actor linked-
queue for pending messages (there are some delivery order semantics)
and some flags, but pretty small overall, with nothing heavyweight
(e.g. threads) per actor. I've had millions of (idle) actors loaded
with no CPU overhead.

[Robin]

Will the actors have any kind preemptive scheduling?
I think Erlang does call counting, for example after 12 method calls,
the process is preemptively switched out.
What happens in the case that a Clojure Actor enters an infinite
loop?
Does that tie up a thread indefinitely?

[Robin]

Or the transaction timeout would abort the misbehaving message...nice.

[Rich Hickey]

On Nov 24, 7:38 pm, Robin <robi...@gmail.com> wrote:
> Will the actors have any kind preemptive scheduling?

Not under the control of Clojure. Messages are processed on Java
threads.

> I think Erlang does call counting, for example after 12 method calls,
> the process is preemptively switched out.
> What happens in the case that a Clojure Actor enters an infinite
> loop?
> Does that tie up a thread indefinitely?

Yes.