[erlang-questions] Erlang-based opensource publish-subscribe servers

[Zvi]

Hi,

I looking for the opensource examples of Erlang-based publish-
subscribe servers.
Any suggestions?

Thanks in advance,
Zvi

________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questio...@erlang.org

[Ulf Wiger]

Hi Zvi,

One way to implement publish/subscribe is with gproc properties.

-define(KEY, {?MODULE, subscr}).

subscribe() ->
gproc:add_local_property(?KEY, undefined).

publish(Event) ->
[P ! {self(), ?KEY, Event} || {P,_} <- gproc:lookup_local_properties(?KEY)].

Gproc takes care of monitoring the subscribers. Change local to global
for global publish/subscribe.

http://github.com/esl/gproc

BR,
Ulf W

Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com

[Toby Thain]

On 24/10/10 1:10 PM, Zvi wrote:
> Hi,
>
> I looking for the opensource examples of Erlang-based publish-
> subscribe servers.
> Any suggestions?

Not sure if you mean applications -- if so, ejabberd implements the
publish/subscribe parts of XMPP.

--Toby

[Tony Arcieri]

On Sun, Oct 24, 2010 at 11:10 AM, Zvi <zvi.a...@gmail.com> wrote:

> Hi,
>
> I looking for the opensource examples of Erlang-based publish-
> subscribe servers.
> Any suggestions?

RabbitMQ also provides pubsub

--
Tony Arcieri
Medioh! A Kudelski Brand

[Zvi .]

no,
I need something lightweight, but able to work in a large cluster (10s of
nodes).

The ideas I have:
1. Use ets to store mappings of Subscriber Pids to TopicIDs - this ets will
obviously be a bottleneck.
2. Have a process per TopicID, which hold list of all it's Subscriber Pids
in it's state. Now the bottleneck is mapping TopicID to it's Pid
3. Use Ulf's gproc (looks it uses gen_leader abd requires patching Erlang?)

I currently working on 1.

Zvi

On Sun, Oct 24, 2010 at 8:41 PM, Andrew Stone <stond...@yahoo.com> wrote:

> If you want something powerful you can checkout rabbitmq. It may be
> overkill but
> it's pretty damn easy to get up and running.
>
> http://www.rabbitmq.com/
>
> -Andrew

--
Zvi Avraham, CTO
Nivertech Ltd

[Andrew Stone]

If you want something powerful you can checkout rabbitmq. It may be overkill but
it's pretty damn easy to get up and running.

http://www.rabbitmq.com/

-Andrew

----- Original Message ----
From: Zvi <zvi.a...@gmail.com>
To: erlang-q...@erlang.org
Sent: Sun, October 24, 2010 1:10:46 PM
Subject: [erlang-questions] Erlang-based opensource publish-subscribe servers

[Ulf Wiger]

On 24 Oct 2010, at 21:22, Zvi . wrote:

> 3. Use Ulf's gproc (looks it uses gen_leader abd requires patching Erlang?)

No, it requires no patches of Erlang. I patched Erlang in the first prototype,
but only to demonstrate what it could look like if gproc were started as
part of the kernel application, and the functionality integrated into the
standard behaviours.

It does use gen_leader, though. As far as I can tell, quite a few people
actually use gen_leader in real systems nowadays. There are a few
forks on github to choose from.

BR,
Ulf W

Ulf Wiger, CTO, Erlang Solutions, Ltd.
http://erlang-solutions.com

[Ivan Uemlianin]

Dear Zvi

gproc looks useful.

And have you read Richard Jones' three-part tutorial, starting here:

http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1/

Best

Ivan

On 24/10/2010 20:22, Zvi . wrote:
> no,
> I need something lightweight, but able to work in a large cluster (10s of
> nodes).
>
> The ideas I have:
> 1. Use ets to store mappings of Subscriber Pids to TopicIDs - this ets will
> obviously be a bottleneck.
> 2. Have a process per TopicID, which hold list of all it's Subscriber Pids
> in it's state. Now the bottleneck is mapping TopicID to it's Pid
> 3. Use Ulf's gproc (looks it uses gen_leader abd requires patching Erlang?)
>
> I currently working on 1.
>
> Zvi

[Augusto Becciu]

Here's my fork of gen_leader where I fixed all the bugs I encountered while
running gproc in a distributed setting:

http://github.com/abecciu/gen_leader_revival

Augusto

On Sun, Oct 24, 2010 at 6:28 PM, Ulf Wiger
<ulf....@erlang-solutions.com>wrote:

[Ulf Wiger]

Hi Augusto,

Thanks. Could you explain your approach to handling the dynamic
adding/removal of candidates, how you've tested it and why you
believe it's safe?

Note that I'm not suggesting it isn't safe. :)
Just that it touches on the core algorithm, and I think many would feel
more comfortable using it if the changes were thoroughly explained.

BR,
Ulf W

[Augusto Becciu]

Hi Ulf,

On Tue, Oct 26, 2010 at 3:58 AM, Ulf Wiger
<ulf....@erlang-solutions.com> wrote:
>
> Hi Augusto,
> Thanks. Could you explain your approach to handling the dynamic
> adding/removal of candidates, how you've tested it and why you
> believe it's safe?

The algorithm to add nodes dynamically, along with some failure
scenarios, is explained here: http://gist.github.com/499825
I haven't worked on dynamic removal, but you can just shut down nodes
and they will be put in the 'down' list.

I haven't done rigorous analysis or testing on it. Just tested the
cases I documented in the gist linked above and some other similar
ones. I started working on a test suite using eunit and mcErlang but
ran out of time in the last months. I'm still interested in developing
it though, and hope to get more free time to finish it by December.

I very much appreciate any feedback you can provide on this. :)

>
> Note that I'm not suggesting it isn't safe. :)
> Just that it touches on the core algorithm, and I think many would feel
> more comfortable using it if the changes were thoroughly explained.

It doesn't modify the core algorithm. It works like an extension to
it. The code is only used if you add the option {seed, Node} to start
the node.

I encourage people to use my fork not because of the possibility to
add nodes dynamically, but because it contains many fixes (major and
minor), cleaned up code, and proper project organization with
rebarized build system.

Of particular interest to gproc_dist users are these two fixes:

1) http://github.com/abecciu/gen_leader_revival/commit/686da14f9a349ce4891ca910ba997e30b46b3860

Without this fix, when a leader goes down and other node takes over
leadership, all the elements registered by the old leader won't be
removed.

2) http://github.com/abecciu/gen_leader_revival/commit/64903e4d4a765ce8e9ca765f0a1a8ce59f833627

Without this fix, any request made to gproc_dist during the election
process will make it crash.

Cheers,
Augusto

[Zvi]

Hi,

Just found about pg module (not pg2).
http://erldocs.com/R14B/stdlib/pg.html?i=18&search=pg

Looks like it can be used to implement pubsub.
For each topic create process group (pg:create/1), then each
subsriber's process joins topic's process group (pg:join/2).
Publish maps cleanly to pg:send/2.

Any ideas, why not to use this module?
I guess process groups use registered names, so that's a
disadvantage.
Another problem is, that docs call this module "experimental".

thanks,
Zvi

On Oct 24, 9:22 pm, "Zvi ." <zvi.avra...@gmail.com> wrote:
> no,
> I need something lightweight, but able to work in a large cluster (10s of
> nodes).
>
> The ideas I have:
> 1. Use ets to store mappings of Subscriber Pids to TopicIDs - this ets will
> obviously be a bottleneck.
> 2. Have a process per TopicID, which hold list of all it's Subscriber Pids
> in it's state. Now the bottleneck is mapping TopicID to it's Pid
> 3. Use Ulf's gproc (looks it uses gen_leader abd requires patching Erlang?)
>
> I currently working on 1.
>
> Zvi
>

________________________________________________________________

[Banibrata Dutta]

But then, why not pg2 ? It says:

"In pg, each message is sent to all members in the group. In this module,
each message may be sent to one, some, *or all members*. "

With the option of sending to all-members, it looks like something that is
quite similar to pg, and isn't tagged "experimental" and also more proven
thanks to it being used in the disk_log module !!

--
regards,
Banibrata
http://www.linkedin.com/in/bdutta
http://twitter.com/edgeliving

[bi...@landofbile.com]

Speaking of pg. pg:members/1 claims to accept a term() as pg:create/1
but in fact doesn't. It only handles pids and atoms. Other functions
have the same issue. Meaning you can create a process group which can't
be joined or queried.