Protocol type checking
Martin Sustrik
I've originally posted this message to zeromq-dev mailing list, but I
guess it makes sense also in context of the SP discussion.
The topic discussed is how to prevent different messaging patterns to
accidentally speak to each another, for example publish/subscribe client
accidentally connect to reques/reply server or similar.
... This one is a complex problem so let me share some
random half-baked thoughts about it:
The obvious solution is that each peer sends its socket type to the
other peer, which in turn verifies it and drops the connection if it is
incompatible with the advertised socket type.
Now, consider that this kind of mechanism is needed at at least three
layers of the stack:
1. TCP. Identifying TCP services such as SMTP vs. SSH vs. HTTP. This
mechanism exists => TCP ports.
2. 0MQ. Identifying different 0MQ services such as PUB/SUB vs. REQ/REP
vs. PUSH/PULL.
3. Application. Identifying different application level services
(topologies). For example, stock quote distribution vs. video chat
distribution.
The above problem can be solved in two basic ways: either use a separate
"type negotiation" mechanism at each level, or try to unify it in some way.
The first option is pretty obvious and easy to implement. However, there
are several problems with it:
a. It doesn't solve the problem of non-0MQ application (SMTP or
whatever) connecting to 0MQ application (pt 1 above).
b. It can be done on 0MQ level (pt 2) but it's not clear how to do it on
the application level (pt 3) especially when the pattern in question is
unidirectional or unreliable. The only option available seems to move
application level type-checking to 0MQ which in turn suffers of the
other problems mentioned in this list.
c. It doesn't play well with unidirectional or unreliable transports. To
do type checking we need to pass the type in both directions and in a
reliable way.
d. The type checking mechanism is not a generic feature. It's different
for each transport protocol. Type checking for TCP is different from
type checking for SCTP as the two cannot speak together anyway.
e. Different messaging patterns are orthogonal non-intersecting
protocols that happen to be bundled in a single library (libzmq). You
can think of it as say a single library providing TCP, UDP and SCTP
support. However, type checking is a universal feature that cuts across
all these protocols. Intuitive reaction of a protocol designer would be
to move it to a common underlying layer thus (TCP and friends).
The above leads us to a second possible solution, which is providing a
unified, transport-agnostic type checking system.
The idea of this solution is to provide the type checking in the single
layer, namely the transport layer which is common to all the patterns.
In practical terms it means, for example, using TCP ports to identify
particular service on TCP, 0MQ and application level.
The second component of the solution should be a system for mapping the
high-layer type (topology, e.g. "NASDAQ stock quotes" to lower-layer
types (i.e. PUB/SUB on 0MQ level and port 5555 on TCP level).
This second solution makes more sense IMHO, but, at the same time,
requires much more work and experimentation to get it done.
Uh. That was long...
Martin
Paul Colomiets
Having experience in setting up complex network topologies with
zeromq, I'd say this problem should be addressed. Typical problems
setting new service are following:
1. Can't connect to some port.
2. Using wrong pattern. Pull vs sub and push vs pub, often thought
like "give me unidirectional input/output" socket.
3. Connecting to wrong port.
The easiest way to see first one within zeromq application is strace
(which shows connecting 10 times a second). Other two are only tracked
by sitting and watching configuration for ages :)
We can't use tcp port per service, partially because of zeromq's
symmetry of binding vs connecting (we usually connect service to a
device instead of binding service to a port), and partially because of
using more than one service instance per machine.
So in perfect world I'd check quadruple (application_id, service_id,
pattern, pattern_version) on connection attempt, so that all three
problems above could be tracked from the log. Simpler solution of
course would be be concatenate application_id and service_id into
single topology_id which is opaque user-specified string. Pattern may
probably be concatenated with a version number.
For connection-less protocols this check should optionally be done
when joining a cluster. For UDP it would be a special packet (and a
reply), and it should be optional because of networks where you want
unidirectinal pattern and can't receive reply by UDP (like firewalled
ones). I'm not very familiar with PGM, but probably retransmission
mechanism could be used for this kind of thing.
There are no inherently unidirectional transports I'm aware of.
--
Paul
Martin Sustrik
> Having experience in setting up complex network topologies with
> zeromq, I'd say this problem should be addressed. Typical problems
> setting new service are following:
>
> 1. Can't connect to some port.
> 2. Using wrong pattern. Pull vs sub and push vs pub, often thought
> like "give me unidirectional input/output" socket.
> 3. Connecting to wrong port.
>
> The easiest way to see first one within zeromq application is strace
> (which shows connecting 10 times a second). Other two are only tracked
> by sitting and watching configuration for ages :)
Yes. True. However, let's distinguish the protocol part of the problem
(how to communicate whether the peers are compatible or not) from the
implementation part of the problem (how to communicate the compatibility
problem to the user). Only the former is of interest here, the former
should be discussed in the scope of individual implementations.
> We can't use tcp port per service, partially because of zeromq's
> symmetry of binding vs connecting (we usually connect service to a
> device instead of binding service to a port), and partially because of
> using more than one service instance per machine.
This is an interesting problem. For those not familiar with 0mq, the
problem can be put this way:
If TCP ports (or similar mechanism) is the only way to define the "type"
of the endpoint and if the messaging pattern defines different kinds of
endpoints (e.g. publishers vs. subscribers), the information about the
type of the endpoint must be conveyed via TCP port number.
Let me give an example. If there's a PUB/SUB broker, there are
publishers and subscribers connecting to it. As assumed above there's no
way for the client to let the server know whether it is a publisher or a
subscriber. Thus, the broker has to bind to two TCP ports, one to be
used by publishers, other one by the subscribers.
As Paul says, in such situation there's no clear mapping from service
name to TCP port. The service actually maps to two different ports, one
for publishers, other one for subscribers.
> So in perfect world I'd check quadruple (application_id, service_id,
> pattern, pattern_version) on connection attempt, so that all three
> problems above could be tracked from the log. Simpler solution of
> course would be be concatenate application_id and service_id into
> single topology_id which is opaque user-specified string. Pattern may
> probably be concatenated with a version number.
I feel we are dealing with two separate problems here:
1. How to find the right endpoint to connect to. If user specifies only
the name of the topology when joining the topology and infrastructure
transparently translates it into TCP address and port, we've solved most
of the problems.
2. If the mis-connection happens anyway, how can we find it have
happened and fail in a decent manner, log it etc.? I think the
proposition above has to do with this point.
As for the first point, some very initial work was done w.r.t. resolving
symbolic names to physical addresses via DNS (thanks to VMware for
sponsoring the development!) The patched version of 0mq can be found here:
https://github.com/mato/libzmq/tree/dns
It's used like this:
connect (sock, "dns://mytopology.example.com");
It doesn't solve the problem of multiple ports for the same service,
etc. but it's at least a nice start.
As for the second point, what about using opaque identifiers, say UUIDs.
instead of structured ones?
> For connection-less protocols this check should optionally be done
> when joining a cluster.
Yes. I think this corresponds to the pt.1 above.
> For UDP it would be a special packet (and a
> reply), and it should be optional because of networks where you want
> unidirectinal pattern and can't receive reply by UDP (like firewalled
> ones).
Ok, kind of makes sense. However, I would say we are speaking about DCCP
here (connection-based unreliable transport). UDP is meant to be
connectionless. Making handshakes on top of it just means artificially
forcing it into being a connection-based protocol.
> I'm not very familiar with PGM, but probably retransmission
> mechanism could be used for this kind of thing.
This is rather interesting topic: PGM uses a mechanism called SPM (RFC
3208, page 4, also section 13.3.1) to communicate metadata from
publishers to subscribers. What it communicates is actually an opaque
BLOB called TSI.
The interesting part is that given that the publisher has no idea about
all the subscribers, it simply broadcasts the SPM packet once in a while
and it's up to subscribers to check whether everything is OK. Note that
this can be though of as a connection-less uni-directional equivalent to
a type-checking handshake.
> There are no inherently unidirectional transports I'm aware of.
Raw IP multicast, for one.
Martin
Paul Colomiets
On Sat, Nov 12, 2011 at 10:28 AM, Martin Sustrik <sus...@250bpm.com> wrote:
> Yes. True. However, let's distinguish the protocol part of the problem (how
> to communicate whether the peers are compatible or not) from the
> implementation part of the problem (how to communicate the compatibility
> problem to the user). Only the former is of interest here, the former should
> be discussed in the scope of individual implementations.
>
It makes some sense, but if pattern match, how you can detect whether it's
right place to connect? Sure, communicating the problem to user is an
implementation part, but detecting the problem can't be done without
protocol part. (unless you will put topology id inside every request, which
could be either ineffecient or ugly)
> As for the second point, what about using opaque identifiers, say UUIDs.
> instead of structured ones?
>
The problem is if I have two instances of same application with
multiple services inside. And we share same hardware for both of them.
Then, to be safe we need to configure each service's UUID separtely.
That process is tedious and error-prone.
Let me show an example from the game industry. Consider you have a
game, e.g. chess, and you have two services:
chess.chat
chess.game
To setup another instance of application you just configure prefix,
and they become:
chess1.chat
chess1.game
So you have no problem of one application sending data to another.
Being them UUIDs, you could not only mess up with IPs and ports, you
could mess up UUIDs all together (like each service can have a
separate config, and they can depend on each other, you must
synchonise them, log messages are mostly unusefull, etc.) Sure you can
generate UUIDs from service names, and translate log messages back
from UUIDs to services, but as UUID generation is irreversible,
translation is complex and unreliable process (you can only translate
known UUIDs).
So do you scared of the long
"com.company.name.department.Product.Name.instance.service"
identifiers?
>> For connection-less protocols this check should optionally be done
>> when joining a cluster.
>
> Yes. I think this corresponds to the pt.1 above.
>
>> For UDP it would be a special packet (and a
>> reply), and it should be optional because of networks where you want
>> unidirectinal pattern and can't receive reply by UDP (like firewalled
>> ones).
>
> Ok, kind of makes sense. However, I would say we are speaking about DCCP
> here (connection-based unreliable transport). UDP is meant to be
> connectionless. Making handshakes on top of it just means artificially
> forcing it into being a connection-based protocol.
>
You can think of it as safety check, not like handshake. And it can be
asked periodically in case the service on the other side changes.
>> I'm not very familiar with PGM, but probably retransmission
>> mechanism could be used for this kind of thing.
>
> This is rather interesting topic: PGM uses a mechanism called SPM (RFC 3208,
> page 4, also section 13.3.1) to communicate metadata from publishers to
> subscribers. What it communicates is actually an opaque BLOB called TSI.
>
> The interesting part is that given that the publisher has no idea about all
> the subscribers, it simply broadcasts the SPM packet once in a while and
> it's up to subscribers to check whether everything is OK. Note that this can
> be though of as a connection-less uni-directional equivalent to a
> type-checking handshake.
>
Good news. Implementations should probably provide some
configuration for trade off between joining cluster faster or do type check
first.
>> There are no inherently unidirectional transports I'm aware of.
>
> Raw IP multicast, for one.
Well, I don't think we should target protocol to raw IP multicast, but as we
see, periodically sending type-checking info solves the problem.
--
Paul
Martin Sustrik
>> Yes. True. However, let's distinguish the protocol part of the problem (how
>> to communicate whether the peers are compatible or not) from the
>> implementation part of the problem (how to communicate the compatibility
>> problem to the user). Only the former is of interest here, the former should
>> be discussed in the scope of individual implementations.
>>
>
> It makes some sense, but if pattern match, how you can detect whether it's
> right place to connect? Sure, communicating the problem to user is an
> implementation part, but detecting the problem can't be done without
> protocol part. (unless you will put topology id inside every request, which
> could be either ineffecient or ugly)
Yes, agreed. The only thing I was trying to say was that implementation
details are irrelevant for the protocol discussion, not that the whole
problem should not be solved.
Not really. The problem I was alluding to is preventing clashes. UUID
contains somewhat less than 128 bits of randomness. Thus, it's almost
impossible that a non-SP application (SSH, SMTP, HTTP etc.) connects to
an SP endpoint and accidentally happens to send the right data to
connect (this could happen especially with short, 1-byte IDs).
It would also prevent all the "test" topologies (every developer is
going to create at least a couple with that name) to clash.
So, my take on it is as follows:
1. Create global unique names using DNS. If you own, say, example.com,
you can be sure that no service will clash with your myservice.example.com
2. Provide a run-time check of the type when TCP connection is
established. There are 3 options AFAICS:
a. Use an arbitrary string to identify the topology. Problem: Many
people are going to use same strings ("test", "publisher" and alike) so
there's rather high chance of clashes.
b. Use DNS addresses to identify the topology. Problem: It tighly couple
SP with DNS. I.e. you wouldn't be able to instantiate your first
"helloWorld" topology without having a domain registered, messing with
DNS etc.
c. Use a random identifier, such as UUID. Problem: It's messy. However,
the UUID can be stored in DNS (a single place) and the infrastructure
can get it from there instead of requiring the user to mess with it
directly.
Finally, I would like to note that some kind of fallback is needed. For
example: If the user doesn't specify the topology ID, the algorithm
should fall back to checking the SP topology type, for example, whether
both endpoints adhere to pub/sub pattern.
This can be achieved by assigning fixed IDs to the messaging patterns.
How could we guarantee that these are unique? It's easy with UUIDs. With
DNS IDs they would have to be managed by some owner of an DNS domain
(iana.org?) In case of textual IDs we would have to specify some IDs to
be reserved (those starting with an underscore or such).
>> Ok, kind of makes sense. However, I would say we are speaking about DCCP
>> here (connection-based unreliable transport). UDP is meant to be
>> connectionless. Making handshakes on top of it just means artificially
>> forcing it into being a connection-based protocol.
>>
> You can think of it as safety check, not like handshake. And it can be
> asked periodically in case the service on the other side changes.
I believe that for connectionless protocols we should not introduce
"per-connection" state, e.g. handshakes. However, you are right we can
use either periodic publishing of the type or, alternatively, attach the
type to every UDP packet.
>>> I'm not very familiar with PGM, but probably retransmission
>>> mechanism could be used for this kind of thing.
>>
>> This is rather interesting topic: PGM uses a mechanism called SPM (RFC 3208,
>> page 4, also section 13.3.1) to communicate metadata from publishers to
>> subscribers. What it communicates is actually an opaque BLOB called TSI.
>>
>> The interesting part is that given that the publisher has no idea about all
>> the subscribers, it simply broadcasts the SPM packet once in a while and
>> it's up to subscribers to check whether everything is OK. Note that this can
>> be though of as a connection-less uni-directional equivalent to a
>> type-checking handshake.
>>
> Good news. Implementations should probably provide some
> configuration for trade off between joining cluster faster or do type check
> first.
Dunno, but I would guess that receiving messages without getting an SPM
first could be problematic in different ways.
> Well, I don't think we should target protocol to raw IP multicast, but as we
> see, periodically sending type-checking info solves the problem.
I wouldn't discard the option to use IP multicast. And yes, periodic
type-checking should help in such circumstances.
Martin
Paul Colomiets
On Tue, Nov 15, 2011 at 11:03 AM, Martin Sustrik <sus...@250bpm.com> wrote:
> Not really. The problem I was alluding to is preventing clashes. UUID
> contains somewhat less than 128 bits of randomness. Thus, it's almost
> impossible that a non-SP application (SSH, SMTP, HTTP etc.) connects to an
> SP endpoint and accidentally happens to send the right data to connect (this
> could happen especially with short, 1-byte IDs).
>
It looks like solution seeking for a problem. We usually use some
range of ports for our application, like 10000-11000. It's hard to put
25 instead of 10025 as a port, but its easy to have off by one bugs.
If it's security problem, you should not allow connect at that port in
the first place. If it's problem of misconfiguration, we should just
implement a handshake that is different from SSH, SMTP, HTTP, etc.
Most network protocols out there have a handshake (Exceptions, like
mongodb, will usually fail when trying to do some request). It's not
that big problem for UDP and PGM, because there are no much UDP and
PGM based protocols in use currently, and there aren't going to be
hundreds of services multicasting something inside a single network
(it would be just ineffecient)
> It would also prevent all the "test" topologies (every developer is going to
> create at least a couple with that name) to clash.
>
How? Be sure that, all the "test" topologies will share same UUID.
> So, my take on it is as follows:
>
> 1. Create global unique names using DNS. If you own, say, example.com, you
> can be sure that no service will clash with your myservice.example.com
>
> 2. Provide a run-time check of the type when TCP connection is established.
> There are 3 options AFAICS:
>
> a. Use an arbitrary string to identify the topology. Problem: Many people
> are going to use same strings ("test", "publisher" and alike) so there's
> rather high chance of clashes.
>
> b. Use DNS addresses to identify the topology. Problem: It tighly couple SP
> with DNS. I.e. you wouldn't be able to instantiate your first "helloWorld"
> topology without having a domain registered, messing with DNS etc.
>
> c. Use a random identifier, such as UUID. Problem: It's messy. However, the
> UUID can be stored in DNS (a single place) and the infrastructure can get it
> from there instead of requiring the user to mess with it directly.
>
I'd vote for arbitrary string with some recommended pattern, like dbus
recommends to use reversed domain names for objects. Checking DNS
isn't going to happen, particularly because we are discussing another
way of name resolution here and because of performance. If they are
not checked against real DNS, then they are inherently arbitrary.
And probably some implementation, like zeromq could use for address
"sp-tcp://name.example.org" topology id of "name.example org" after
resolving DNS (even may be after resolving CNAME's) but it's entirely
implementation's feature.
Resolving UUIDs to names throught DNS is quite inconvenient. First we
probably need to define a protocol above DNS for that. Second it isn't
going to be done when writing logs (because of performance), it is
going to be decoded when reading logs, and DNS records can differ at
that time. And remember we are discussing configuration mistakes, so
we are adding another place to fail process of debugging failed
configuration (Sounds bad, isn't it?)
> Finally, I would like to note that some kind of fallback is needed. For
> example: If the user doesn't specify the topology ID, the algorithm should
> fall back to checking the SP topology type, for example, whether both
> endpoints adhere to pub/sub pattern.
>
I still argue that we need to check both topology id and pattern. (And
maybe only pattern if no topology id configured). Some reasons I've
desribed in zeromq mailing list:
http://lists.zeromq.org/pipermail/zeromq-dev/2011-November/014391.html
More reasons here include protocol versioning (in case we will need
them at any time in future; it of course will be up to an
implementation to support single or all versions of protocol or
particular pattern)
--
Paul
Martin Sustrik
>> Not really. The problem I was alluding to is preventing clashes. UUID
>> contains somewhat less than 128 bits of randomness. Thus, it's almost
>> impossible that a non-SP application (SSH, SMTP, HTTP etc.) connects to an
>> SP endpoint and accidentally happens to send the right data to connect (this
>> could happen especially with short, 1-byte IDs).
>>
> It looks like solution seeking for a problem. We usually use some
> range of ports for our application, like 10000-11000. It's hard to put
> 25 instead of 10025 as a port, but its easy to have off by one bugs.
Ok. If people believe dropping non-SP connections is out-of-scope we can
omit it from the requirements.
> If it's security problem, you should not allow connect at that port in
> the first place.
No, it's not a security problem.
> If it's problem of misconfiguration, we should just
> implement a handshake that is different from SSH, SMTP, HTTP, etc.
That was actually my line of thought: We need a different initiation
sequence / handshake => how do we guarantee that? => send enough random
data to be sure that it's highly improbable to accidentally match an
existing initiation sequence => 128 bits or randomness should be enough
for that => UUID.
>> It would also prevent all the "test" topologies (every developer is going to
>> create at least a couple with that name) to clash.
>>
> How? Be sure that, all the "test" topologies will share same UUID.
Yup. Unless the UUID is auto-generated by the management tools. Imagine
creating topology like this:
$ sp-topology create test.example.org
The above can create new DNS record already tagged by a freshly
generated UUID.
> I'd vote for arbitrary string with some recommended pattern, like dbus
> recommends to use reversed domain names for objects. Checking DNS
> isn't going to happen, particularly because we are discussing another
> way of name resolution here and because of performance. If they are
> not checked against real DNS, then they are inherently arbitrary.
>
> And probably some implementation, like zeromq could use for address
> "sp-tcp://name.example.org" topology id of "name.example org" after
> resolving DNS (even may be after resolving CNAME's) but it's entirely
> implementation's feature.
>
> Resolving UUIDs to names throught DNS is quite inconvenient. First we
> probably need to define a protocol above DNS for that. Second it isn't
> going to be done when writing logs (because of performance), it is
> going to be decoded when reading logs, and DNS records can differ at
> that time. And remember we are discussing configuration mistakes, so
> we are adding another place to fail process of debugging failed
> configuration (Sounds bad, isn't it?)
Ah. I think you've misunderstood me.
I definitely don't think the UUIDs should be used as addresses. Let me
explain...
We need to use human-readable names as addresses. However, if we are
thinking on the Internet scale, we have to guarantee they are unique. We
can do this using existing domain name tree (DNS). The unique name then
maps to an IP address or whatever:
myservice.example.org => tcp://62.128.0.11:5555
That's basically it. Period.
Now, we have a second problem: What if the DNS is misconfigured or maybe
it is circumvented etc.? In such case the applications should be able to
spot the problem, log it, notify the admin etc.
To do so, we can add some kind of handshake between the communicating
endpoints. For example, each endpoint can send string
"myservice.example.org" to its peer and check whether the peer does the
same.
Now, that should do in theory. However, there are couple of reasons to
replace the initiation string by an UUID: First, there's more randomness
in a UUID. Second, it's fixed size which makes it more HW-friendly.
Note that using UUIDs instead of strings is a kind of optimisation
("more random", "more HW-friendly") rather than a hard requirement.
>> Finally, I would like to note that some kind of fallback is needed. For
>> example: If the user doesn't specify the topology ID, the algorithm should
>> fall back to checking the SP topology type, for example, whether both
>> endpoints adhere to pub/sub pattern.
>>
> I still argue that we need to check both topology id and pattern. (And
> maybe only pattern if no topology id configured). Some reasons I've
> desribed in zeromq mailing list:
>
> http://lists.zeromq.org/pipermail/zeromq-dev/2011-November/014391.html
>
> More reasons here include protocol versioning (in case we will need
> them at any time in future; it of course will be up to an
> implementation to support single or all versions of protocol or
> particular pattern)
Yes, I see the problem: Different kinds of endpoints in a single
topology (e.g. publishers vs. subscribers) and no way to distinguish
between them.
Btw, the discussion is getting complex pretty fast. As I've already
mentioned there's a 0MQ branch with DNS lookup implemented. Maybe it
would be worth doing some experimenting with it to find out what works
in real world and what does not.
Martin