Existing models for PUB/SUB

[Martin Sustrik]

Hi all,

Below I am trying to collect links to the existing models for PUB/SUB
messaging.

JMS, PubSubHubbub, Redis and STOMP are the simplest. As far as I can say
they don't account for complex distribution topologies and define only
the simple case of single-hop (disregarding the broker) PUB/SUB.

OpenAMQ and RabbitMQ are based on the same idea of building complex
topologies by making one AMQP broker becoming a client of another AMQP
broker.

ZeroMQ carries the idea further by omitting brokers and speaking of
pub/sub topologies composed of equal peers.

While XEP-0060 defines only a simple pub/sub akin to JMS, experimental
XEP-0248 account for more complex topologies.

Finally, Hermes is an interesting alternative view on pub/sub coming
from academic environment.

Enjoy!
Martin

Hermes:

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.105.9899&rep=rep1&type=pdf

JMS:

See chaper 6 in JMS 1.1 specs (not available at HTML AFAICS)

OpenAMQ:

http://www.openamq.org/user:3-advanced#toc46

PubSubHubbub:

http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-core-0.3.html

RabbitMQ:

http://hg.rabbitmq.com/rabbitmq-federation/file/default/README-hacking

Redis:

http://redis.io/topics/pubsub

STOMP:

http://stomp.codehaus.org/Protocol

XMPP:

http://xmpp.org/about-xmpp/technology-overview/pubsub/

ZeroMQ:

http://www.250bpm.com/pubsub

[Martin Sustrik]

On 06/25/2011 12:41 PM, Martin Sustrik wrote:

> Below I am trying to collect links to the existing models for PUB/SUB
> messaging.

Oops. Apache Qpid (RedHat MRG) link got accidentally lost:

https://cwiki.apache.org/qpid/using-broker-federation.html

Martin

[Tim Fox]

HornetQ extends the standard JMS pub/sub by allowing servers to form
"clusters". Subscribers can be created on any particular node to consume
from the same "topic", and will receive messages for that topic sent to
any node in the cluster. Since subscribers can have filters (selectors),
messages are only forwarded if there are matching subscribers on other
nodes.

There can be multiple clusters, which can be joined by "bridges" which
forward messages from one cluster to another (e.g. from London to New
York cluster). Bridges can be configure to filter and transform messages
too.

End result of which you can create very large global pub/sub networks.

[Alexis Richardson]

and in addition to the Microsoft links below, see

Ruby

https://github.com/jcoglan/faye
https://github.com/derekcollison/nats

Python

http://pubsub.sourceforge.net/

On Sat, Jun 25, 2011 at 12:48 PM, Alexis Richardson
<alexis.r...@gmail.com> wrote:
> Microsoft have published the msmq specifications (google to find them) but they may not have a formal pub sub model.
>
> They have also published something at pubsub.codeplex.com that definitely does pub sub.
>
>
>
> Sent from my iPad

>> --
>> Note Well: This discussion group is meant to become an IETF working group in the future. Thus, the posts to this discussion should comply with IETF contribution policy as explained here: http://www.ietf.org/about/note-well.html
>

[Martin Sustrik]

Hi Tim,

> HornetQ extends the standard JMS pub/sub by allowing servers to form
> "clusters". Subscribers can be created on any particular node to consume
> from the same "topic", and will receive messages for that topic sent to
> any node in the cluster. Since subscribers can have filters (selectors),
> messages are only forwarded if there are matching subscribers on other
> nodes.
>
> There can be multiple clusters, which can be joined by "bridges" which
> forward messages from one cluster to another (e.g. from London to New
> York cluster). Bridges can be configure to filter and transform messages
> too.
>
> End result of which you can create very large global pub/sub networks.

Is there any documentation for that?

It doesn't appear to bu pure JMS-based as AFAICS JMS doesn't allow for
passing subscriptions to the publisher (remote cluster).

Martin

[Tim Fox]

On 25/06/2011 12:53, Martin Sustrik wrote:
> Hi Tim,
>
>> HornetQ extends the standard JMS pub/sub by allowing servers to form
>> "clusters". Subscribers can be created on any particular node to consume
>> from the same "topic", and will receive messages for that topic sent to
>> any node in the cluster. Since subscribers can have filters (selectors),
>> messages are only forwarded if there are matching subscribers on other
>> nodes.
>>
>> There can be multiple clusters, which can be joined by "bridges" which
>> forward messages from one cluster to another (e.g. from London to New
>> York cluster). Bridges can be configure to filter and transform messages
>> too.
>>
>> End result of which you can create very large global pub/sub networks.
>
> Is there any documentation for that?

It's really scattered in several places in the user manual. (Bridge and
clustering chapters mainly)
http://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html/index.html

BTW this is not specific to pub/sub, it's use for distributing messages
for work queuing too.

>
> It doesn't appear to bu pure JMS-based as AFAICS JMS doesn't allow for
> passing subscriptions to the publisher (remote cluster).

This functionality is available in HQ irrespective of what protocol/API
you are using to talk to the broker, e.g. JMS, STOMP, REST etc
>
> Martin

[Martin Sustrik]

On 06/25/2011 01:45 PM, Alexis Richardson wrote:

>> https://github.com/jcoglan/faye

The unerlying protocol spec is here:

http://svn.cometd.com/trunk/bayeux/bayeux.html

>> https://github.com/derekcollison/nats

Ah. I've heard Derek have written a messaging system in Ruby but I
haven't actually checked it so fat. I'll give it a look.

>> http://pubsub.sourceforge.net/

I've clicked through the documentation and I am still completely sure of
the following, however, it seems that it's only in-process messaging,
ie. there's no wire protocol to take into account. Also, the design
doesn't seem to go beyond single-hop scenarios.

>> Microsoft have published the msmq specifications (google to find
them) but they may not have a formal pub sub model.

Nice. I wasn't aware of that. This seems to be the link:

http://download.microsoft.com/download/9/5/E/95EF66AF-9026-4BB0-A41D-A4F81802D92C/%5BMS-MQQB%5D.pdf

I haven't checked for the details on how pub/sub is done yet though.

>> They have also published something at pubsub.codeplex.com that definitely does pub sub.

Yes. However, it seems there's documentation whatsoever :|

Martin

[Tony Garnock-Jones]

On 25 June 2011 06:41, Martin Sustrik <sus...@250bpm.com> wrote:

JMS, PubSubHubbub, Redis and STOMP are the simplest. As far as I can say they don't account for complex distribution topologies and define only the simple case of single-hop (disregarding the broker) PUB/SUB.

IRC. NNTP.

STOMP:
http://stomp.codehaus.org/Protocol

That page has languished a bit. STOMP 1.1 lives here: http://stomp.github.com/stomp-specification-1.1.html

Regards,
  Tony
--
Tony Garnock-Jones
tonygarn...@gmail.com
http://homepages.kcbbs.gen.nz/tonyg/

[Martin Sustrik]

On 06/26/2011 04:41 PM, Tony Garnock-Jones wrote:
> On 25 June 2011 06:41, Martin Sustrik <sus...@250bpm.com

> <mailto:sus...@250bpm.com>> wrote:
>
> JMS, PubSubHubbub, Redis and STOMP are the simplest. As far as I can
> say they don't account for complex distribution topologies and
> define only the simple case of single-hop (disregarding the broker)
> PUB/SUB.
>
>
> IRC. NNTP.

Yes. And IP multicast (together IGMP and friends).

Martin

[Gary Berger]

Don¹t forget the DDS crowd.

http://www.prismtech.com/opensplice

http://www.opendds.org/

-g

[Martin Sustrik]

On 06/27/2011 11:24 PM, Gary Berger wrote:
> Don�t forget the DDS crowd.
>
> http://www.prismtech.com/opensplice
>
> http://www.opendds.org/

Right. The specs can be found here:

http://www.omg.org/technology/documents/dds_spec_catalog.htm

Martin

[Alexis Richardson]

Would it be reasonable to include smtp+POP+IMAP

On Jun 28, 2011 9:41 AM, "Martin Sustrik" <sus...@250bpm.com> wrote:

On 06/27/2011 11:24 PM, Gary Berger wrote:

Donšt forget the DDS crowd.

> > http://www.prismtech.com/opensplice > > http://www.opendds.org/

Right. The specs can be found here:

http://www.omg.org/technology/documents/dds_spec_catalog.htm

Martin

-- Note Well: This discussion group is meant to become an IETF working group in the future. Thus,...

[Martin Sustrik]

On 06/28/2011 10:45 AM, Alexis Richardson wrote:
> Would it be reasonable to include smtp+POP+IMAP

Is there a pub/sub functionality in these? Maybe mailing lists?

Martin

[Alexis Richardson]

Yes exactly. It is a good exercise to try to create a pubsub system from these tools

On Jun 28, 2011 9:53 AM, "Martin Sustrik" <sus...@250bpm.com> wrote:

On 06/28/2011 10:45 AM, Alexis Richardson wrote: > > Would it be reasonable to include smtp+POP+IMAP...

[Martin Sustrik]

On 06/28/2011 11:05 AM, Alexis Richardson wrote:
> Yes exactly. It is a good exercise to try to create a pubsub system from
> these tools

I am not an expert on email technologies, but the brief check seems to
suggest following:

1. There's no standard protocol for mailing list. The semantics can be
reviewed only by checking individual implementations.

2. It seems that most implementation have a system for processing
(un)susbscriptions. This is normally done by posting an email to a
specific address (not the mailing list address). Note that in enterprise
messaging area the counterpart for mailing list subscriptions is
creation of links in the distribution tree (e.g. federation) rather than
subscriptions as such.

3. Given that there are no real subscriptions, there's no subscription
forwarding functionality.

3. AFAICS there's no system for detecting loops in the topology.

Martin

[Tony Garnock-Jones]

On 28 June 2011 05:31, Martin Sustrik <sus...@250bpm.com> wrote:

1. There's no standard protocol for mailing list. The semantics can be reviewed only by checking individual implementations.

Correct, though that really is only true of the subscription-management aspect. The publication and receipt of publications is standard. (Though perhaps not formally so?)
 

3. Given that there are no real subscriptions, there's no subscription forwarding functionality.

Hmm, well I'm not sure there "are no real subscriptions", because I certainly feel like I've subscribed to various mailing-lists :-) And re subscription forwarding: that would be using the (out-of-band/ad-hoc/unstandardized) subscription management of a source mailing-list to add the list address of a target mailing-list as a subscriber. Perhaps I missed what you mean.
 

3. AFAICS there's no system for detecting loops in the topology.

Nothing standardised, but in practice all relays use characteristics like presence of the "X-List-Address" header, scanning of the "Received" headers etc to identify looping messages. This of course is a reactive approach, the only thing that can be done in the absence of vocabulary for discussing subscriptions and their management as first-class entities.

[Martin Sustrik]

On 06/28/2011 05:23 PM, Tony Garnock-Jones wrote:

> 3. Given that there are no real subscriptions, there's no
> subscription forwarding functionality.
>
> Hmm, well I'm not sure there "are no real subscriptions", because I
> certainly feel like I've subscribed to various mailing-lists :-) And re
> subscription forwarding: that would be using the
> (out-of-band/ad-hoc/unstandardized) subscription management of a source
> mailing-list to add the list address of a target mailing-list as a
> subscriber. Perhaps I missed what you mean.

Please, refer to the previous email about topology establishment vs.
routing.

When you "subscribe" to the mailing list you join the topology.

After that there's no way to ask for a specific subset of messages, such
as "I am interested only in messages that contain word 'cucumber' in the
subject." That's what I meant by no subscriptions.

Martin

[Tony Garnock-Jones]

(Pardon me; I misaddressed the previous message. Roll on improved internet messaging!)

---------- Forwarded message ----------
From: Tony Garnock-Jones <tonygarn...@gmail.com>
Date: 28 June 2011 15:14
Subject: Re: [SP] Existing models for PUB/SUB (mailing lists)
To: Martin Sustrik <sus...@250bpm.com>

On 28 June 2011 14:57, Martin Sustrik <sus...@250bpm.com> wrote:

When you "subscribe" to the mailing list you join the topology.

Gotcha.
 

After that there's no way to ask for a specific subset of messages, such as "I am interested only in messages that contain word 'cucumber' in the subject." That's what I meant by no subscriptions.

Hmm. I actually think of this as being able to ask for a specific subset of messages—just with a trivially simple query language:

Query ::== ⊥

Similar to joining a multicast group; or a fanout exchange (where the routing_key is effectively uninterpreted).

Subscription, to me, is parameterised over

• the name of the subscription coordinator for the network/data-source being joined/subscribed to (this is where the subscription request is sent)
  
• a query language, specific to the network/data-source being joined/subscribed to
• a specific query in that language (may or may not correspond closely to the notion of "name" on the publish side)
  
• a destination address for deliveries to be sent to, specific to the transport path between the subscriber and the network/data-source being joined/subscribed to.

So for mailing lists that's: the admin email address; the trivial query language; the trivial query; and the target email address.

For IPv4 on a LAN: the DHCP server's broadcast address; the language of DHCPDISCOVER; a specific packet; and the MAC address of the joining party.

For AMQP 0-9-1: the broker itself; the language of Queue.Bind; a specific exchange name and binding key; implicit in the source of the request.

Etc.

(I'm still digesting your other email BTW.)

Regards,
  Tony
--
Tony Garnock-Jones
tonygarn...@gmail.com
http://homepages.kcbbs.gen.nz/tonyg/

[Alexis Richardson]

And now this:

http://code.google.com/p/closure-library/

ALSO we have not considered implicit pubsub which appears in some data
stores and caching systems. Redis is on the list already, because it
is closest to being explicit (it has a polyglot protocol-like model).

[Alexis Richardson]

Better description of what MS AppFabric can do:

http://convective.wordpress.com/2011/06/08/windows-azure-appfabric-service-bus-queues-and-topics/