[erlang-questions] Distributed publish/subscribe system
Kaiduan Xie
Any one has experience to build distributed publish/subscribe system?
The system has the following requirements,
1. Subscriber subscribes to the interested topic
2. Publisher publishes event related to the topic
3. System delivers the event to all the users subscribed to this
topic. If subscriber is connected, the event is delivered immediately.
If the subscriber is not connected, system should persist the
event. When the user next time becomes online, system delivers the
event to the user.
Can you share the experience?
Best regards,
/Kaiduan
________________________________________________________________
erlang-questions (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:erlang-questio...@erlang.org
Jack Moffitt
> 2. Publisher publishes event related to the topic
> 3. System delivers the event to all the users subscribed to this
> topic. If subscriber is connected, the event is delivered immediately.
> If the subscriber is not connected, system should persist the
> event. When the user next time becomes online, system delivers the
> event to the user.
You get most (if not all of this) from either XMPP (via ejabberd's
mod_pubsub) or rabbitmq. I've used both for various things and was
quite happy.
jack.
Tony Arcieri
> You get most (if not all of this) from either XMPP (via ejabberd's
> mod_pubsub) or rabbitmq. I've used both for various things and was
> quite happy.
Note that if you're looking for a simpler solution than either XMPP or
RabbitMQ, Redis Cluster will eventually support distributed pubsub whenever
it's released.
--
Tony Arcieri
Medioh! Kudelski
Toby Thain
> Hi,
>
> Any one has experience to build distributed publish/subscribe system?
> The system has the following requirements,
>
> 1. Subscriber subscribes to the interested topic
> 2. Publisher publishes event related to the topic
> 3. System delivers the event to all the users subscribed to this
> topic. If subscriber is connected, the event is delivered immediately.
> If the subscriber is not connected, system should persist the
> event. When the user next time becomes online, system delivers the
> event to the user.
>
> Can you share the experience?
>
I have done this with single publisher and single subscriber using
pubsub module and ejabberd. The client was a Java application using
Spark for the xmpp library. I would certainly prefer to use the existing
xmpp design and ejabberd implementation over designing, building and
testing my own.
--Toby
Joe Armstrong
How big? - how scalable - what latency - fault tolerance requirements do you
have.
Distributed can mean 2 nodes - or 10^9
There might be ten channels or 10^12
How fault tolerant? - 2 replicas, 100 replicas.
What the latency? - do you want the message to be delivered with 10 ms or
100 years
What's the security model - who owns the data - is there centralized
security or weak
per-node security policies.
The point is your requirements need to say things like this otherwise they
are not much use.
The architectures for these different cases will be *completely different*
/Joe
Thomas Lindgren
----- Original Message ----
> From: Kaiduan Xie <kaid...@gmail.com>
...
> Any one has experience to build distributed publish/subscribe system?
> The system has the following requirements,
>
> 1. Subscriber subscribes to the interested topic
> 2. Publisher publishes event related to the topic
> 3. System delivers the event to all the users subscribed to this
> topic. If subscriber is connected, the event is delivered immediately.
> If the subscriber is not connected, system should persist the
> event. When the user next time becomes online, system delivers the
> event to the user.
>
> Can you share the experience?
I haven't tried it myself, but RabbitMQ sounds like it might be suitable:
http://www.rabbitmq.com/tutorial-three-python.html
I think you can have persistent queues too.
Best regards,
Thomas
Kaiduan Xie
resources. Joe pointed out a lot of excellent points. The requirements
are listed below,
1) The system should be able to support tens to hundreds of millions
users spreading around the world, so the system will consist hundred
of nodes located in different physical locations.
2) Any user can subscribe to the interested topic, but only topic
owner can publish message to the group. We have no limit on the number
of subscribers in each group. It can be huge, for example, the fans of
Rolling Stone around the world.
3) The number of topics/groups in the system is unlimited.
4) User is not always connected to the system, not always connected to
the same node in the system, and the message delivery should be
guaranteed.
5) As to the latency, it should be in the range of 1 minute if
subscriber is connected.
I am doing research on this topic now, the other requirements are not
sorted out yet.
Thanks,
/Kaiduan
Alex Arnon
subscriber is connected.
Well then, you don't need a software system at all - humans could do this
for you :)
j...@duomark.com
> 1) ... support tens to hundreds of millions
> users spreading around the world ...
> 2) ... no limit on the number of subscribers in each group ...
> 3) The number of topics/groups in the system is unlimited.
> 4) User is not always connected to the system...
> 5) ... latency ... of 1 minute if subscriber is connected.
> I am doing research on this topic now, the other requirements
> are not sorted out yet.
CouchDB is a possibility, and you could test it out using
a service such as http://www.cloudant.com/ which uses
quorum reads and writes to ensure data accuracy. The
requirements I left in above are the ones that Couch would
purport to satisfy, although on the scale of users you
are talking about you would definitely need services
distributed to different continents.
jay
Brian Candler
> 1) The system should be able to support tens to hundreds of millions
> users spreading around the world, so the system will consist hundred
> of nodes located in different physical locations.
>
> 2) Any user can subscribe to the interested topic, but only topic
> owner can publish message to the group. We have no limit on the number
> of subscribers in each group. It can be huge, for example, the fans of
> Rolling Stone around the world.
>
> 3) The number of topics/groups in the system is unlimited.
>
> 4) User is not always connected to the system, not always connected to
> the same node in the system, and the message delivery should be
> guaranteed.
>
> 5) As to the latency, it should be in the range of 1 minute if
> subscriber is connected.
Sounds like NNTP to me!
Joe Armstrong
> Thanks Jack, Tony, Toby, Joe and Thomas for the helpful comments and
> resources. Joe pointed out a lot of excellent points. The requirements
> are listed below,
>
> 1) The system should be able to support tens to hundreds of millions
> users spreading around the world, so the system will consist hundred
> of nodes located in different physical locations.
>
Who own the nodes? are they all owned by the same organisation?
Who is legally responsible for the system?
>
> 2) Any user can subscribe to the interested topic, but only topic
> owner can publish message to the group. We have no limit on the number
> of subscribers in each group. It can be huge, for example, the fans of
> Rolling Stone around the world.
>
>
How do your stop spam - bogus groups etc?
> 3) The number of topics/groups in the system is unlimited.
>
How do you stop name squatting - ie flooding the system with stupid channel
names?
>
> 4) User is not always connected to the system, not always connected to
> the same node in the system, and the message delivery should be
> guaranteed.
>
Uuug - impossible - it's the Byzantine generals problem. How do you know
they got a
message? - send an ack - how do you know if they got the ack?- ack the ack -
how do you know
if they acked the ack, ack the acked ack ...
/Joe
Malcolm Spence
Newsbrief on combining Erlang with CORBA and DDS.
http://mnb.ociweb.com/mnb/MiddlewareNewsBrief-201101
DDS is the OMG spec for a real time data distribution service (versus a
queue). We developed an open source implementation for large scale messaging
systems.
It has the kind of QoS capabilities you need. Can have federated servers to
help with FT and scaling etc. Has a Java interface and can be used as a JMS
provider in a JBoss/ESB/SOA stck.
regards Malcolm
Malcolm D. Spence
Director of Business Development
OCI "Use our reach to exceed your grasp."
Voting member of the OMG.
(Phone 1- 314-579-0066 ext. 206 or FAX -0065)
www.theaceorb.com www.ociweb.com
James Churchman
Jerome Martin
and leverage what Erlang has to offer, as described
here<http://armstrongonsoftware.blogspot.com/2008/05/road-we-didnt-go-down.html>and
there <http://steve.vinoski.net/pdf/IEEE-RPC_Under_Fire.pdf> ? To re-quote
Steve Vinoski after Joe Armstrong:
*"What all those years of CORBA taught me, BTW, is that RPC, for a*
*number of reasons, is generally A Really Bad Idea."*
Or is it just me not understanding what years of Erlang can teach us
regarding good practice, smart technology choices and mistakes of the past,
sometimes leading to widespread adoption of products implementing probably
the worse part of otherwise interesting research (like mostly all of OMG's
so-called standards) ?
--
Jérôme Martin
Malcolm Spence
We are an open source software engineering shop. We integrate disparate
systems for a living. If we did not have open source we would not be able to
do a lot of what we do. We try to not to argue with clients over their past
technology choices. We just try to make them work.
This paper is example of us sharing our thoughts with others in case they
run into similar situations. Our "Middleware Newsbrief" and our more OO
language and tools oriented "Tech Trends" are ways in which we try to
encourage fact based discussion with example code and observations.
We are not advocating one technology over another. We like to understand the
problem before offering a solution. And will readily admit no solution is
perfect. In fact we do like to temper people's expectations when we first
engage because we are pragmatists.
regards Malcolm
Malcolm D. Spence
Director of Business Development
OCI "Use our reach to exceed your grasp."
Voting member of the OMG.
(Phone 1- 314-579-0066 ext. 206 or FAX -0065)
www.theaceorb.com www.ociweb.com
From: Jerome Martin [mailto:tramjo...@gmail.com]
Sent: Thursday, January 27, 2011 4:00 AM
To: Malcolm Spence
Cc: erlang-q...@erlang.org
Subject: Re: [erlang-questions] Distributed publish/subscribe system
Wouldn't using this defeat the whole purpose of moving beyond the RPC model,
and leverage what Erlang has to offer, as described here
<http://armstrongonsoftware.blogspot.com/2008/05/road-we-didnt-go-down.html>
and there <http://steve.vinoski.net/pdf/IEEE-RPC_Under_Fire.pdf> ? To
re-quote Steve Vinoski after Joe Armstrong:
"What all those years of CORBA taught me, BTW, is that RPC, for a
number of reasons, is generally A Really Bad Idea."
Or is it just me not understanding what years of Erlang can teach us
Jerome Martin
> Thanks for the comments Jerome.
>
Thank you for taking my comment so peacefully.
> We are an open source software engineering shop. We integrate disparate
> systems for a living. If we did not have open source we would not be able to
> do a lot of what we do. We try to not to argue with clients over their past
> technology choices. We just try to make them work.
>
I understand your position. It is an honorable business stance. Please note
that my comment was not targeting OSS vs non OSS, but more the pervasiveness
in the "industry" of methodologies and techniques for mostly the wrong
reasons, from a purely technical standpoint (better marketing does not make
a solution better at problem-solving). I realize that there are probably a
lot of different reasons for one individual to adopt or support those, and
it is not my intent to judge or criticize those. However, I have to admit I
have an epidermic reaction to what I feel is a regression compared to the
state of art in CS, and even more so when I see advocacy for such technology
choice on this mailing list.
> This paper is example of us sharing our thoughts with others in case they
> run into similar situations. Our "Middleware Newsbrief" and our more OO
> language and tools oriented "Tech Trends" are ways in which we try to
> encourage fact based discussion with example code and observations.
>
Nice. If only you applied just that to technology that solves more problems
than it creates (and no, I won't even attempt to prove that, it is an
obvious exaggeration to make my point :-) ), I would be a big fan!
> We are not advocating one technology over another. We like to understand
> the problem before offering a solution. And will readily admit no solution
> is perfect. In fact we do like to temper people's expectations when we first
> engage because we are pragmatists.
>
Again, this is a most honorable stance.
Regards,
--
Jérôme Martin
Steve Vinoski
> Wouldn't using this defeat the whole purpose of moving beyond the RPC model,
> and leverage what Erlang has to offer, as described
> here<http://armstrongonsoftware.blogspot.com/2008/05/road-we-didnt-go-down.html>and
> there <http://steve.vinoski.net/pdf/IEEE-RPC_Under_Fire.pdf> ? To re-quote
> Steve Vinoski after Joe Armstrong:
>
> *"What all those years of CORBA taught me, BTW, is that RPC, for a*
> *number of reasons, is generally A Really Bad Idea."*
I couldn't agree more! :-)
In Malcolm's defense, though, there's still a lot of CORBA out there,
a fair amount of it in embedded systems, and it's not going anywhere
anytime soon. For example, to the best of my knowledge certain
telecommunications interfaces required by law are based on CORBA. A
dozen years ago at the height of CORBA we always predicted it would
still be in use in 2020, and so far it seems we might have been fairly
accurate with that prediction. Malcolm and folks like him specialize
in helping folks who have no choice but to maintain such systems try
to fit it together with newer technologies changing and advancing
around it. I'm happy to see Malcolm and his colleagues trying to bring
Erlang into such systems in order to reduce maintenance costs, enhance
functionality, and improve integration capabilities -- I think we
should encourage more of that.
--steve
Stu Bailey
with some interesting features like stateful aggregation and search. It
looks like it would be simple to write an erlang client for it.
Stu
Jerome Martin
> On Thu, Jan 27, 2011 at 4:59 AM, Jerome Martin <tramjo...@gmail.com>
> wrote:
> > Wouldn't using this defeat the whole purpose of moving beyond the RPC
> model,
> > and leverage what Erlang has to offer, as described
> > here<
> http://armstrongonsoftware.blogspot.com/2008/05/road-we-didnt-go-down.html
> >and
> > there <http://steve.vinoski.net/pdf/IEEE-RPC_Under_Fire.pdf> ? To
> re-quote
> > Steve Vinoski after Joe Armstrong:
> >
> > *"What all those years of CORBA taught me, BTW, is that RPC, for a*
> > *number of reasons, is generally A Really Bad Idea."*
>
> I couldn't agree more! :-)
>
Why am I not surprised ? :-)
> In Malcolm's defense, though, there's still a lot of CORBA out there,
> a fair amount of it in embedded systems, and it's not going anywhere
> anytime soon.
Yes, and this is why I said I understand his business stance. Someone has to
maintain these systems now I suppose. However, advising to create some more
is a different thing. Well, that is unless if you specialize in maintenance
of said systems :-) (just a humorous remark, I am not accusing anyone of
anything there)
> For example, to the best of my knowledge certain
> telecommunications interfaces required by law are based on CORBA. A
> dozen years ago at the height of CORBA we always predicted it would
> still be in use in 2020, and so far it seems we might have been fairly
> accurate with that prediction. Malcolm and folks like him specialize
> in helping folks who have no choice but to maintain such systems try
> to fit it together with newer technologies changing and advancing
> around it.
<nod>
I'm happy to see Malcolm and his colleagues trying to bring
> Erlang into such systems in order to reduce maintenance costs, enhance
> functionality, and improve integration capabilities -- I think we
> should encourage more of that.
>
Yes, in the same sense as I am happy to see functional idioms in python:
this is far from ideal, and brings only an order of magnitude less than the
benefits of pure FP. But it has the merit of exposing more people to FP, at
the risk or diluting it, just like smalltalk ideas have been diluted (and
the best ones mostly dropped) in "modern OO".
--
Jérôme Martin
French, Mike
> -----Original Message-----
> From: erlang-q...@erlang.org
> [mailto:erlang-q...@erlang.org]On
> Behalf Of Malcolm Spence
> Sent: 26 January 2011 22:50
> To: erlang-q...@erlang.org
> Subject: Re: [erlang-questions] Distributed publish/subscribe system
>
>
> Newsbrief on combining Erlang with CORBA and DDS.
Integration with DDS is interesting.
DDS is definitely 'sympathetic' to The Erlang Way.
Almost the only things it has in common with CORBA
are reusing IDL as the message specification format
and CDR as the on-the-wire binary data encoding.
DDS is loosely-coupled publish-subscribe, not RPC.
DDS is decentralized peer-to-peer, as opposed to
the centralized hub-and-spoke topology for CORBA,
and many other pub-sub message brokers, like AMQP.
DDS specifies a binary wire protocol, unlike, say, JMS.
DDS is designed for low-latency and high-throughput,
and can operate in soft-realtime systems, but it goes
to the next step and supports quality of service (QoS)
parameters for individual pub-sub channels,
so you can explicitly allocate a latency budget
and configure what happens when messages are late.
etc. etc.
A native implementation of DDS in Erlang would be fascinating.
Mike
Thales UK Ltd (Wells) DISCLAIMER: The information contained in this e-mail
is confidential. It may also be legally privileged. It is intended only for
the stated addressee(s) and access to it by any other person is
unauthorised. If you are not an addressee, you must not disclose, copy,
circulate or in any other way use or rely on the information contained in
this e-mail. Such unauthorised use may be unlawful. We may monitor all
e-mail communications through our networks. If you have received this e-mail
in error, please inform us immediately on sender's telephone number above
and delete it and all copies from your system. We accept no responsibility
for changes to any e-mail which occur after it has been sent. Attachments
to this e-mail may contain software viruses which could damage your system.
We therefore recommend you virus-check all attachments before opening.
Thales UK Ltd. Registered Office: 2 Dashwood Lang Road, The Bourne Business
Park, Addlestone, Weybridge, Surrey KT15 2NX Registered in England No.
868273