Distributed concurrent applications in Clojure?

[dokondr]

Any ideas how Clojure can be used for distributed concurrent
applications ?
To my mind it should be possible to implement in Clojure mechanism
similar to Erlang light-weight processes running on distributed
'nodes' that can be addressed by 'ports' to send them 'commands'.
How this can be done with Clojure 'agents'?
An obvious solution would be to create an 'agent proxy' on every node
that will maintain a list of its agents and will forward (send ...)
messages to other nodes, as well as receive messages addressed to its
own agents from remote nodes and then deliver these messages to
correct agents.
Addressing scheme and message protocols should be developed as well.
Other ways to provide for communication between nodes?

[Monsieur Pinocchio]

There was some talk about Clojure running on terracotta (www.terracotta.org) which is a distributed JVM (a single JVM instance running on distributed nodes). Depending on how well terracotta can pull it off, clojure's own implementation may remain simple.

However, I am wondering if having a "clojure conscious" distributed version of agents, STM, Threads and the global namespace can prove to be better at efficiency and providing time-memory-network_traffic guarantees to the programmer. Immutability of objects and the design of the global and thread local namespace should reduce implementation hassles a lot.

I will really appreciate some comments.

Pinocchio

[Rich Hickey]

There's JMS:

http://en.wikipedia.org/wiki/Java_Message_Service
https://mq.dev.java.net/
http://joram.objectweb.org/
http://activemq.apache.org/

XMPP:

http://en.wikipedia.org/wiki/Xmpp
http://www.igniterealtime.org/projects/index.jsp

JXTA/Shoal:

http://en.wikipedia.org/wiki/Jxta
https://shoal.dev.java.net/

JINI:

http://en.wikipedia.org/wiki/Jini
http://incubator.apache.org/projects/river.html

DHTs like Pastry:

http://freepastry.org/

JGroups:

http://www.jgroups.org/javagroupsnew/docs/index.html

Terracotta:

http://www.terracotta.org

Jinterface:

http://www.erlang.org/doc/apps/jinterface/

NetKernel:

http://www.1060.org/

and more. All useful from Clojure.

Given the diversity, sophistication, maturity, interoperability,
robustness etc of these options, it's unlikely I'm going to fiddle
around with some language-specific solution.

That said, I have been thinking about putting a simple wrapper API
around queues that would work both locally and over something like
JMS.

Rich

[dokondr]

Rich,
I readily acknowledge the diversity of message passing frameworks for
Java.
Notwithstanding, I think it makes sense to think about distributed
message passing mechanism inherent to Clojure language, like the one
Erlang has.
It is Erlang abstraction of light-weight processes and extreme ease of
reliable, synchronized communication between these processes running
on one or
many distributed 'nodes', it is all that that makes Erlang so great
for real world distributed tasks!
Clojure could have become a great next step in this direction!

Dima

On Nov 25, 4:11 pm, Rich Hickey <richhic...@gmail.com> wrote:
> On Nov 24, 6:41 pm, dokondr <doko...@gmail.com> wrote:
>
> > Any ideas how Clojure can be used for distributed concurrent
> > applications ?
> > To my mind it should be possible to implement in Clojure  mechanism
> > similar to Erlang light-weight processes running on distributed
> > 'nodes' that can be addressed by 'ports' to send them 'commands'.
> > How this can be done with Clojure 'agents'?
> > An obvious solution would be to create an 'agent proxy' on every node
> > that will maintain a list of its  agents and will forward (send ...)
> > messages to other nodes, as well as receive messages addressed to its
> > own agents from remote nodes and then deliver these messages to
> > correct agents.
> > Addressing scheme and message protocols should be developed as well.
> > Other ways to provide for communication between nodes?
>
> There's JMS:
>
> http://en.wikipedia.org/wiki/Java_Message_Servicehttps://mq.dev.java.net/http://joram.objectweb.org/http://activemq.apache.org/
>
> XMPP:
>
> http://en.wikipedia.org/wiki/Xmpphttp://www.igniterealtime.org/projects/index.jsp
>
> JXTA/Shoal:
>
> http://en.wikipedia.org/wiki/Jxtahttps://shoal.dev.java.net/
>
> JINI:
>

> http://en.wikipedia.org/wiki/Jinihttp://incubator.apache.org/projects/river.html

[levand]

But couldn't you just write some macros to wrap a message-passing
framework and have it be part of the language, for all practical
purposes? It would seem that this falls under the category "if you
want it, just do it".

If anything, Clojure is potentially more powerful than Erlang for this
kind of thing because it can utilize multiple distribution paradigms
as appropriate. And with macros at your command, you can make the
syntax as terse as the underlying semantics will allow.

Luke

> >http://en.wikipedia.org/wiki/Java_Message_Servicehttps://mq.dev.java....
>
> > XMPP:
>
> >http://en.wikipedia.org/wiki/Xmpphttp://www.igniterealtime.org/projec...

>
> > JXTA/Shoal:
>
> >http://en.wikipedia.org/wiki/Jxtahttps://shoal.dev.java.net/
>
> > JINI:
>

> >http://en.wikipedia.org/wiki/Jinihttp://incubator.apache.org/projects...

[Dave Griffith]

One big issue to note is that, because of Refs, Clojure agent
semantics can't simply be remoted the way Erlang processes can be.
This is because a message send could include a references to a Ref,
thus exposing mutable state remotely. This breaks, well, just about
everything.

If you restrict the acceptable arguments to message sends to include
only fully functional values (no passing Refs), then your agent-proxy
idea could probably be made to work. I'm guessing the tricky code
won't be the remote equivalent of "send", but rather the remote
equivalent of "wait", as that requires keeping track of where the
remote sends come from, in a stable and machine-independent way. The
current Clojure implementation seems to do that internally via uuids,
but accessing those remotely could be tricky.

For extra bonus points, allow for multiple pluggable protocol stacks
and automatic protocol negotiation (which Erlang has, IIRC).

Unlike some of the other comments in this thread, I'll say I believe
that remote agents in Clojure could be a very powerful idea,
particularly due to integration with the STM. Orchestrating in-
memory and eternal communications takes an enormous amount of effort
in many systems. Having a software transaction which automatically
queues up remote agent sends so that they only occur once the
transaction completes sucessfully would be a very powerful language-
level primitive.

Code it up and give it a try!

[Stuart Sierra]

On Nov 26, 11:19 am, Dave Griffith <dave.l.griff...@gmail.com> wrote:
> Unlike some of the other comments in this thread, I'll say I believe
> that remote agents in Clojure could be a very powerful idea,
> particularly due to integration with the STM.   Orchestrating in-
> memory and eternal communications takes an enormous amount of effort
> in many systems.  Having a software transaction which automatically
> queues up remote agent sends so that they only occur once the
> transaction completes sucessfully would be a very powerful language-
> level primitive.

Hi Dave,

If I recall correctly, agent sends are already queued inside
transactions until the transaction succeeds. So maybe all you need is
a proxy agent that receives agent sends from the local process and
forwards them across the network to another process. Just a random
idea; I haven't used this sort of distributed system before.

-Stuart Sierra

[Alex Miller]

Hi all, I work at Terracotta and I think there are many things about
Clojure that make it very interesting as a Terracotta target. The
focus on immutable data structures is particularly fascinating and
presents some unique challenges for Terracotta. One thing that we
spend a lot of time focusing on how is to deal with minimizing
broadcast of changes in collections like maps. I haven't decided yet
whether the immutable data structure focus actually makes this harder
or easier. I suspect it gives us the tools to do more but will
require customization to take best advantage of it.

I'm a total clojure newb but I am currently rotating between reading
Hoare's "Communicating Sequential Processes", Armstrong's Erlang book,
and Stu's new Clojure beta book. I will emerge on the other side
either as a guru or more likely hopelessly confused. :)

At some future point, I'd love to work on clustering Clojure with
Terracotta if only to learn more about it and understand it more
deeply. From the discussion above, I think the key thing to keep in
mind with Terracotta is that it's not a message-passing architecture;
rather it's clustered memory (and notification). So it helps make
changes in one node appear in all nodes. In that sense, it's very
much targeted towards a shared-memory approach, not an actor/messaging
approach. However, many people have built pipe and messaging
abstractions with TC as a substrate so I think that's an interesting
approach.

Looking forward to learning more about it...

Alex

> >http://en.wikipedia.org/wiki/Java_Message_Servicehttps://mq.dev.java....
>
> > XMPP:
>
> >http://en.wikipedia.org/wiki/Xmpphttp://www.igniterealtime.org/projec...
>

> > JXTA/Shoal:
>
> >http://en.wikipedia.org/wiki/Jxtahttps://shoal.dev.java.net/
>
> > JINI:
>

> >http://en.wikipedia.org/wiki/Jinihttp://incubator.apache.org/projects...

[dokondr]

Dave,
Sure thing, only immutable values will cross network, no Refs.
As for the "wait" problem of, as you put it: "the tricky code won't be

the remote equivalent of "send", but rather the remote equivalent of
"wait", as that requires keeping track of where the remote sends come

from" - this can be solved with a system where every node runs a
blackboard, that node agents use to "subscribe" for messages of
particular type from other nodes.
Blackboard is a very well known publish/subscribe mechanism widely
used in distributed systems.
One old and famous blackboard example is Linda:
http://en.wikipedia.org/wiki/Linda_(coordination_language

[Eric Sessoms]

For kicks, I've uploaded a little wrapper around JInterface to make it
a little more lispy. It might make it a little easier for anyone
interested in playing around with that option.

http://clojure.googlegroups.com/web/erlang.clj

On Nov 25, 8:11 am, Rich Hickey <richhic...@gmail.com> wrote:
> Jinterface:
>
> http://www.erlang.org/doc/apps/jinterface/

[Michael Wood]

On Fri, Nov 28, 2008 at 2:33 AM, Eric Sessoms <nubg...@gmail.com> wrote:
>
> For kicks, I've uploaded a little wrapper around JInterface to make it
> a little more lispy. It might make it a little easier for anyone
> interested in playing around with that option.
>
> http://clojure.googlegroups.com/web/erlang.clj

Interesting :)

> ;;; erlang.clj -- Clojure wrapper for Erlang's JInterface.
>
> (ns erlang)
>
> ;; YMMV
> (add-classpath "file:///opt/local/lib/erlang/lib/jinterface-1.4/priv/OtpErlang.jar")

Using add-classpath, except at the REPL, is discouraged.

[...]
> ;; atom
> (defmethod to-erlang clojure.lang.Symbol [s]
> (new OtpErlangAtom (str s)))
>
> (defmethod from-erlang OtpErlangAtom [s]
> (symbol (.atomValue s)))
>
> (defmethod to-erlang clojure.lang.Keyword [s]
> (new OtpErlangAtom (subs (str s) 1)))
[...]

I think you want (name s) for both the Symbol and Keyword cases:

user=> (defmulti to-erlang class)
#'user/to-erlang
user=> (defmethod to-erlang clojure.lang.Symbol [s] (name s))
#<MultiFn clojure.lang.MultiFn@14384c2>
user=> (defmethod to-erlang clojure.lang.Keyword [s] (name s))
#<MultiFn clojure.lang.MultiFn@14384c2>
user=> (to-erlang 'symbol)
"symbol"
user=> (to-erlang :keyword)
"keyword"

--
Michael Wood <esio...@gmail.com>

[Eric Sessoms]

On Nov 28, 5:01 am, "Michael Wood" <esiot...@gmail.com> wrote:
> > ;; YMMV
> > (add-classpath "file:///opt/local/lib/erlang/lib/jinterface-1.4/priv/OtpErlang.jar")
>
> Using add-classpath, except at the REPL, is discouraged.

True, it felt dirty doing that. My intent was to give someone who
might not be as familiar with Erlang a clue as to where to find the
required supporting libraries. I'll comment it out.

> I think you want (name s) for both the Symbol and Keyword cases:
>
> user=> (defmulti to-erlang class)
> #'user/to-erlang
> user=> (defmethod to-erlang clojure.lang.Symbol [s] (name s))
> #<MultiFn clojure.lang.MultiFn@14384c2>
> user=> (defmethod to-erlang clojure.lang.Keyword [s] (name s))
> #<MultiFn clojure.lang.MultiFn@14384c2>
> user=> (to-erlang 'symbol)
> "symbol"
> user=> (to-erlang :keyword)
> "keyword"

Thanks!

Eric

[Razvan Ludvig]

@dokondr: the blackboard mechanism you mention is just a distributed
associative memory, and there is already a Java specification for such
a system (search for Jini and Javaspaces), with several open source
and commercial implementations available. Unfortunately, anyone who
has ever worked with such a mechanism can probably tell you how
difficult it is to go beyond some simple use cases. Also, the fact
that it's not language agnostic (Javaspaces works with Java objects)
doesn't actually make it better, since it imposes big limitations on
the objects you can use efficiently with this mechanism. There are
other limitations I have encountered in my work with such systems,
mostly related to the lack of support tools (administration,
monitoring, etc).

From my experience with distributed Java applications, JMS is
generally a good fit for most systems. It offers transactions (even
XA) and persistence, which are both necessary more often than not in
such systems, while the API is simple. Additionally, most JMS
implementations can be clustered (giving you fail-over and/or load-
balancing functionality) and the message content can be language
agnostic. Implementing even a subset of these features "naitvely" in
Clojure would represent a very big effort, and for what gains? A JMS
wrapper library for Clojure would make much more sense, especially if
it could integrate with the STM (a successful JMS commit has side-
effects, so maybe this is impossible ) / Agent support. Also, there
are good implementations and support tools for JMS, since it's been
around for a long time.