PUB/SUB protocol (a strawman)
Martin Sustrik
This is a strawman for how protocol for PUB/SUB pattern could possibly
look like:
The PUB/SUB protocol is built on top of the underlying of SP framing
layer (see SP framing I-D).
The PUB/SUB topology is assumed to form a tree. Each node except the
publisher is connected to exactly one "upstream" node. Every node is
connected to 0..N "downstream" nodes.
There are "messages" flowing downstream and "subscription commands"
flowing upstream (if possible).
"Messages" have no SP-specific fields. They are composed entirely of the
opaque payload defined on the application layer.
"Subscription commands" look like this:
+--------+---------+-------+
| cmd-id | algo-id | query |
+--------+---------+-------+
cmd-id is a 16-bit network byte order unsigned integer specifying the
action to take. There are two defined values for this field:
1 - subscribe
2 - unsubscribe
algo-id is a 16-bit network byte order unsigned integer that specifies
the algorithm to use for subscription matching. The values for this
field are managed by IANA. Possible values are: prefix matching, regexp,
x-path query, SQL-like selector etc.
query is an arbitrarily large field, spanning from the end of algo-id
field to the end of the payload. The value of this field is interpreted
by the algorithm specified by algo-id.
Thoughts?
Martin
Ben Kloosterman
1) In proc pub sub
2) a single combined pub sub server but seperate clients.
3) Cross machine large scale systems where there are more than 1 pub/
sub matching nodes .
I will assume the last , 1 ) is mainly done by languages.
For 2) its quite simple and i would leave it that way.
I asume the frame will take into account large payloads ? And if so
why does each packet need a header ?
The biggest issues with pub / sub is not the protocol but how it all
hangs together ( eg how subscriptions are communicated to other nodes
etc) and on busy systems the packet size and format will be trivial
compared to the subscription matching costs and algorithms - eg if
you have 40K notifications per second and you have 40K subscriptions
( remember 1 person can have 200 subscriptions eg stocks he is
watching ) a brute force would result in 1.6 Billion comparisons a
second ( which could be xpath). Topics are best but encouraging people
to use topics and filters by making it easy for the calee is very
critical ,xpath is popular here but iMHO is heavy though there are
some great research algorthms for breaking it down .
So for cross machine i would go for a larger header that makes its
easier and more flexible. Hence Im not sure whether a binary protocal
adds a lot of value vs say text or a tight xml , maye somehting based
on FIX. Consider also a non binary scheme will allow variable sized
packets and be much easier to call from a client api. Larger packets
results in more aggregation of data which is a good thing.
I spent a bit of time with WS-Eventing and the most important feature
is that you can plug in a simple subscription to fan out notifciations
to sub nodes and load this subscription in the configuration ( ie it
is managed by the maintainers of the system not the software clients.
Since xyz subnote now handles abc topic the clients can then look this
up ( either manually in a driectory or via a redirect) and send the
publish (notification/event ) directly to the appropriate node
responsible for that topic ( which can be load balanced) this cuts
all the routing / network path theory out..
I prefer a canonical topic based subscription which are fast , text
and meaningfull ( the server can do a hash for an exact match on each
of the canonical parts which are normally managed by parent nodes) .
Anyway i would have something like this which allows support for
conectionless and seperate nodes.
Subscrbe packet
action ( subscribe , unsubscribe , error ( server going down) or
renew)
sender ( ip and socket so the subscribe knows the subscription has
been sent)
Send notifications to ( where matches are sent to)
optional data
Filter type
FIlter information
Publish packet
action publish
topic
payload
Publish packet forwarded by server
action publish
optional data ( from subscription)
topic
payload
Error Packet
Sender
Error
Ben Kloosterman
Martin Sustrik
> Firstly what is the issue we are addressing ?
> 1) In proc pub sub
> 2) a single combined pub sub server but seperate clients.
> 3) Cross machine large scale systems where there are more than 1 pub/
> sub matching nodes .
>
> I will assume the last , 1 ) is mainly done by languages.
> For 2) its quite simple and i would leave it that way.
Yes. You are right. It's about 3.
> I asume the frame will take into account large payloads ? And if so
> why does each packet need a header ?
The proposal explicitly calls for no header:
"Messages have no SP-specific fields."
> The biggest issues with pub / sub is not the protocol but how it all
> hangs together
That's an important observation!
The important thing here is the semantics of pub/sub network, the
protocol is more or less irrelevant (it can be MQTT, XMPP, AMQP or
whatever).
However, after speaking to IETF folks I've realised that it's much more
viable to have a working group centered around a protocol. Defining
"semantics" or "abstract APIs" is considered somewhat out-of-scope
and/or a bit suspicious.
> ( eg how subscriptions are communicated to other nodes
> etc) and on busy systems the packet size and format will be trivial
> compared to the subscription matching costs and algorithms - eg if
> you have 40K notifications per second and you have 40K subscriptions
> ( remember 1 person can have 200 subscriptions eg stocks he is
> watching ) a brute force would result in 1.6 Billion comparisons a
> second ( which could be xpath). Topics are best but encouraging people
> to use topics and filters by making it easy for the calee is very
> critical ,xpath is popular here but iMHO is heavy though there are
> some great research algorthms for breaking it down .
Yes. That's the point of algo-id field.
The idea is to allow research and progress in the area of matching
algorithms.
We can start with simple prefix-matching a la 0MQ (algo-id=1) and allow
others to register new algo-ids via IANA. Hopefully an evolutionary
process will ensue that will provide us with a sane set of algorithms to
use.
> So for cross machine i would go for a larger header that makes its
> easier and more flexible. Hence Im not sure whether a binary protocal
> adds a lot of value vs say text or a tight xml , maye somehting based
> on FIX.
Why define the generic structure of a message at all?
Deifferent algo-ids can of course assume specific message format. Eg.
x-path algo can assume that the messages are XML, SQL-selector algo can
assume that messages are sets of DB-like fields. Etc.
> Consider also a non binary scheme will allow variable sized
> packets and be much easier to call from a client api. Larger packets
> results in more aggregation of data which is a good thing.
Sorry. I think I don't follow here.
> I spent a bit of time with WS-Eventing and the most important feature
> is that you can plug in a simple subscription to fan out notifciations
> to sub nodes and load this subscription in the configuration ( ie it
> is managed by the maintainers of the system not the software clients.
> Since xyz subnote now handles abc topic the clients can then look this
> up ( either manually in a driectory or via a redirect) and send the
> publish (notification/event ) directly to the appropriate node
> responsible for that topic ( which can be load balanced) this cuts
> all the routing / network path theory out..
I guess nothing prevents you to do that. It has no impact on the
protocol AFAICS though.
> I prefer a canonical topic based subscription which are fast , text
> and meaningfull ( the server can do a hash for an exact match on each
> of the canonical parts which are normally managed by parent nodes).
As already said, the proposal allows for extensible set of matching
algorithms. Thus, you can either use one of the existing algos or define
your own if none of the existing suits you.
> Anyway i would have something like this which allows support for
> conectionless and seperate nodes.
Ack. Like multicast -- publish all messages, filter on the subscriber.
These kind of scenarios should be taken into account when defining
semantics of a pub/sub pattern.
> Subscrbe packet
> action ( subscribe , unsubscribe , error ( server going down) or
> renew)
> sender ( ip and socket so the subscribe knows the subscription has
> been sent)
> Send notifications to ( where matches are sent to)
> optional data
> Filter type
> FIlter information
>
> Publish packet
> action publish
> topic
> payload
>
> Publish packet forwarded by server
> action publish
> optional data ( from subscription)
> topic
> payload
Note that in multi-hop topologies all the hops should use the same
protocol. Ie. there should be only one definiton for the "publish packet".
>
> Error Packet
> Sender
> Error
Martin
Ben Kloosterman
> I asume the frame will take into account large payloads ? And if so
> why does each packet need a header ?
The proposal explicitly calls for no header:
"Messages have no SP-specific fields."
BK> Ok just the subscriptions and other messages which is fine.
> The biggest issues with pub / sub is not the protocol but how it all
> hangs together
That's an important observation!
The important thing here is the semantics of pub/sub network, the
protocol is more or less irrelevant (it can be MQTT, XMPP, AMQP or
whatever).
BK> Yes , the protocol is irrelevant but the overall protocol leads to
certain overall designs ( eg where are the subscriptions handled ,
replicated , how events are broken down etc) . A good design should make it
very easy to use and encourage a competent but simple design - you don't
want to get into routing and filtering subscriptions and the ensuing network
flow .
However, after speaking to IETF folks I've realised that it's much more
viable to have a working group centered around a protocol. Defining
"semantics" or "abstract APIs" is considered somewhat out-of-scope
and/or a bit suspicious.
BK> It may not be in the spec but it should be in the product .
> ( eg how subscriptions are communicated to other nodes
> etc) and on busy systems the packet size and format will be trivial
> compared to the subscription matching costs and algorithms - eg if
> you have 40K notifications per second and you have 40K subscriptions
> ( remember 1 person can have 200 subscriptions eg stocks he is
> watching ) a brute force would result in 1.6 Billion comparisons a
> second ( which could be xpath). Topics are best but encouraging people
> to use topics and filters by making it easy for the calee is very
> critical ,xpath is popular here but iMHO is heavy though there are
> some great research algorthms for breaking it down .
Yes. That's the point of algo-id field.
The idea is to allow research and progress in the area of matching
algorithms.
We can start with simple prefix-matching a la 0MQ (algo-id=1) and allow
others to register new algo-ids via IANA. Hopefully an evolutionary
process will ensue that will provide us with a sane set of algorithms to
use.
BK> Suggest support XPath and canonical topic filters ( over XML messages)
to integrate with WS-Eventing and WS-Notification
BK> Note in the above case the 1.6 Billion comparisons can be improved as
follows ( using STock markets as an example) , server 1 load receives all
subscriptions and messages , it sends all messages to new pub/sub services
according to a canonical id eg Exchange.Banking goes to server 1 ,
EExchange.Resources go to server 2. All subscriptions are forwarder to the
appropriate service.. Now if we break it down to 10 the original server
now has 40K notifications and 10 subscription's for 400K operations per
seconds.. each of the 10 nodes does say 4 K * 4K subscriptions ( assuming
an ideal split) or 16M operations and can break it down further) which is a
lot better than 1.6 Billion.
This sort of thing needs to be very easy and requires deep understanding of
the filter. Hence at least one of the early filters should allow this and
perhaps should be the default. .
> So for cross machine i would go for a larger header that makes its
> easier and more flexible. Hence Im not sure whether a binary protocal
> adds a lot of value vs say text or a tight xml , maye somehting based
> on FIX.
Why define the generic structure of a message at all?
BK> because of the filter integration above ..At least support a topic
filter to xml in the spec ( just reference topic from ws-notify)
Deifferent algo-ids can of course assume specific message format. Eg.
x-path algo can assume that the messages are XML, SQL-selector algo can
assume that messages are sets of DB-like fields. Etc.
BK> Yes but 2comments
1. A good protocol like tcp hides many things and makes it easy to build
systems on tcp the same needs to be done.
2. XML messages may require multiple packets is this taken care of in the
framing ?
> Consider also a non binary scheme will allow variable sized
> packets and be much easier to call from a client api. Larger packets
> results in more aggregation of data which is a good thing.
Sorry. I think I don't follow here.
BK> Just saying that a default filter ( and message structure) and topics
like exchange.grouping.share will make it much easier to use from the client
and help the users to create a number of hierarchal events. 2nd bit is
about whether to support multiple packets with 1 subscription .filter like
the xml message above.
> I spent a bit of time with WS-Eventing and the most important feature
> is that you can plug in a simple subscription to fan out notifciations
> to sub nodes and load this subscription in the configuration ( ie it
> is managed by the maintainers of the system not the software clients.
> Since xyz subnote now handles abc topic the clients can then look this
> up ( either manually in a driectory or via a redirect) and send the
> publish (notification/event ) directly to the appropriate node
> responsible for that topic ( which can be load balanced) this cuts
> all the routing / network path theory out..
I guess nothing prevents you to do that. It has no impact on the
protocol AFAICS though.
BK> It needs to be made easy..Consider the case of someone building a pub
sub system based on 0MQ on one server and it all works great and than he
hits the 16G comparisons . He now needs some major changes and he may be
quite disappointed , however if he had hierarchal topics and is using a
filter based on them , than the system people can break them down as above
quite easy..
> Subscrbe packet
> action ( subscribe , unsubscribe , error ( server going down) or
> renew)
> sender ( ip and socket so the subscribe knows the subscription has
> been sent)
> Send notifications to ( where matches are sent to)
> optional data
> Filter type
> FIlter information
>
> Publish packet
> action publish
> topic
> payload
>
> Publish packet forwarded by server
> action publish
> optional data ( from subscription)
> topic
> payload
Note that in multi-hop topologies all the hops should use the same
protocol. Ie. there should be only one definiton for the "publish packet".
bK> Yes - probably a bad name ( i don't like notification or even either)
, but i really think having a topic in there is very important and should be
in a spec even if not all filters and cases will use it , it should be the
default case most people use and given in an example.
Bk> Note you need publish notification and errors , esp to cancel , renew
and know when the subscription is loaded in a multi server environment. This
also requires some for of addressing .
Lastly note the optional data which get appended to each packet forwarded
this can be quite useful.
Martin Sustrik
> However, after speaking to IETF folks I've realised that it's much more
> viable to have a working group centered around a protocol. Defining
> "semantics" or "abstract APIs" is considered somewhat out-of-scope
> and/or a bit suspicious.
>
> BK> It may not be in the spec but it should be in the product .
Rather, it should be semantics camouflaged like a protocol.
In reality each protocol has a semantic part (e.g. how the retransmit is
done in TCP) so it's not that big a cheat.
The only difference is that in SP's case the protocol can be very
simple, while semantics are pretty complex.
> This sort of thing needs to be very easy and requires deep understanding
> of the filter. Hence at least one of the early filters should allow this
> and perhaps should be the default. .
Agreed. I would recommend documenting the use case and use it to check
any future specification.
> Deifferent algo-ids can of course assume specific message format. Eg.
> x-path algo can assume that the messages are XML, SQL-selector algo can
> assume that messages are sets of DB-like fields. Etc.
>
> BK> Yes but 2comments
> 1. A good protocol like tcp hides many things and makes it easy to build
> systems on tcp the same needs to be done.
> 2. XML messages may require multiple packets is this taken care of in
> the framing ?
No.
Imagine you want to introduce a new filter type.
You have to specify two things:
1. The assumed format of the message (XML)
2. The syntax of the query language (x-path)
You ask IANA to assign a algo-id for your filter and implement the
filtering algorithm in the client library. From this point on, users of
your library can filter any existing feed using x-path.
(Of course, we should define how the filter behaves when confronted with
messages not matching the assumed format. My proposal: Drop them.)
As your filtering algorithm gets more popular, vendors of intermediate
devices may opt to support it. Afterwards, admins of the intermediate
nodes may decide to upgrade them to get the functionality.
At that point the filtering can be handled in the middle of the network,
thus saving the bandwidth associated with transferring all the messages
that nobody is subscribed to.
>> Consider also a non binary scheme will allow variable sized
>> packets and be much easier to call from a client api. Larger packets
>> results in more aggregation of data which is a good thing.
In case of TCP the bundling can be done in the implementation with no
effect on the protocol whatsoever.
With packet-based protocols (UDP, PGM etc.) the bundling affects the
protocol, however, I would argue it's out of scope of SP effort: If you
want bundling on top of PGM (as supported in 0MQ, for exmaple) you
should create a separate WG to standardise it. Once done, SP can use the
ensuing specification as an underlying transport.
Note that binary/textual distinction doesn't apply at this level. L4
protocols tend to treat all the content as opaque/binary anyway.
> I guess nothing prevents you to do that. It has no impact on the
> protocol AFAICS though.
>
> BK> It needs to be made easy..Consider the case of someone building a
> pub sub system based on 0MQ on one server and it all works great and
> than he hits the 16G comparisons . He now needs some major changes and
> he may be quite disappointed , however if he had hierarchal topics and
> is using a filter based on them , than the system people can break them
> down as above quite easy..
Yes. Definitely document the use case. We can use that for checking the
spec viability later on.
> Note that in multi-hop topologies all the hops should use the same
> protocol. Ie. there should be only one definiton for the "publish packet".
>
> bK> Yes - probably a bad name ( i don't like notification or even
> either) , but i really think having a topic in there is very important
> and should be in a spec even if not all filters and cases will use it ,
> it should be the default case most people use and given in an example.
What's the use case for that? I mean, having a topic in the message even
though it is not used by anybody?
> Bk> Note you need publish notification and errors , esp to cancel ,
> renew and know when the subscription is loaded in a multi server
> environment. This also requires some for of addressing .
Yes. The question is whether we should implement these on SP level or
whether what the underlying transport layer offers will do.
> Lastly note the optional data which get appended to each packet
> forwarded this can be quite useful.
Can you describe the use case?
Martin