Enabling RPC over MQTT, Metadata services, etc...

[Rick Bullotta]

As part of my proposal for adding metadata browsing and type information on MQTT topic(s), I think that we're going to need a means of lightweight RPC riding on top of MQTT.  These could be a special kind of endpoint that you would register as a "provider" of the service (e.g. you subscribe to requests) or a "consumer" of the service (you would "publish" a request).  Normal MQTT topic handling won't work for a request/response scenario where you could have multiple simultaneous requests for the same service endpoint (ok, there are hideous and ugly hacks that could be done, but I'm not going there). 

I'd like to get a dialog going on how this could be implemented - my gut tells me it would be fairly easy to do technically, but we should perhaps think out the protocol and the implications and "get it right the first time".

Ultimately I would then envision built-in broker service endpoints for a variety of functions that could be invoked using this protocol.

My end vision is that MQTT can be used for modeling and implementing well-described, discoverable "things" that expose data, services, and events.

Next phase, I'd like to take a pass at defining metadata formats for MQTT topics, but I'll start that discussion separately.

[Paul Fremantle]

Rick

I have been using a scheme where you can publish to a specific client by publishing to $CLIENT/{clientId}

You need to configure the server to have a security rule to only let {clientid} subscribe to that topic.

This doesn't solve the problem of how to send the clientId to the server in a standard way, only the return path.

Paul

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To post to this group, send email to mq...@googlegroups.com.
Visit this group at http://groups.google.com/group/mqtt.
For more options, visit https://groups.google.com/d/optout.

--

Paul Fremantle

Part-time PhD student - School of Computing

email: paul.fr...@port.ac.uk, pa...@fremantle.org

twitter: pzfreo / skype: paulfremantle / blog: http://pzf.fremantle.org

CTO and Co-Founder, WSO2
OASIS WS-RX TC Co-chair, Apache Member
07740 199 729

[Matteo Collina]

Hi Rick,

You can implement that behavior by using Mosca and overriding https://github.com/mcollina/mosca/blob/master/lib/server.js#L317-L330.

Mosca allows a broker to 'react' and process a single publish packet in a way very similar to a RPC connection, and we even got a router framework for it: https://www.npmjs.org/package/mors.

However, it is far from standard.

Cheers,

Matteo

[Rick Bullotta]

Hi, Paul.

I want to avoid a "topic-per-client" model. In a system of reasonable size, it will create a large number of unnecessary topics, garbage collection load, memory consumption, etc. - in general, scalability issues.  I think it will require a purpose-built extension to the MQTT spec to enable a request/response (async of course) model.  Ideally, there would be one topic per "service" in that scenario.

Rick

[Karl Palsson]

A topic per service has the rather large disadvantage that anyone can see the responses. You're going to want a reply channel that is on demand/ephmeral, that the client
suggests perhaps, but now you need to dynamically add/remove ACLS for temporary topics.

Remember, there's nothing wrong with a topic per client if they don't actually _exist_ all
the time.

if the client only subscribes to $CLIENT/RPC/blah when it wants to get a response, then
sends the publish to start the RPC call, then unsubscribes when it receives the response,
you don't have any floating around topics. (If you really really are concerned about that
at all....) This lets you leave the ACLs permanently in place based around a client
connecting/leaving the system entirely. Also, if you have so many clients that you even
think that a topic per client is a problem, you're probably going to be having all sorts of
other problems that are probably more important.

Sincerely,
Karl P

> >> email to mqtt+uns...@googlegroups.com <javascript:>.
> >> To post to this group, send email to mq...@googlegroups.com <javascript:>
> >> .

> >> Visit this group at http://groups.google.com/group/mqtt.
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> >
> >
> > --
> > Paul Fremantle
> > Part-time PhD student - School of Computing

> > email: paul.fr...@port.ac.uk <javascript:>, pa...@fremantle.org<javascript:>

> > twitter: pzfreo / skype: paulfremantle / blog: http://pzf.fremantle.org
> > CTO and Co-Founder, WSO2
> > OASIS WS-RX TC Co-chair, Apache Member
> > 07740 199 729
> >
>

[Paul Fremantle]

Rick

I'd be very interested to hear in other alternatives that use less overhead.

However, I'm not clear there is any overhead in implementing this. The reason I chose $CLIENT is that it indicates special processing to the broker. The broker doesn't need to do anything other than deliver messages to the named client. A client subscribing to this special topic can be registered in the broker by a single boolean (potentially). 

Paul

[Dominik Obermaier]

Rick,
I absolutely love the idea of a Metadata "discovery" mechanism. But I think do this properly, we should - as you suggested - think out of the protocol "box".

Let me start with a provoking idea: Why not use a RESTful approach for metadata discovery?

>From my point of view it doesn't make much sense to use Pub / Sub  for a discovery, because a concrete client decides itself when and how it wants to discover metadata. There is no chance for the server to figure out when a client wants to discover metadata of one or more topics.

Note, that this does *not* mean to use HTTP. I am just advocating for a RESTful approach, be it HTTP, CoAP or a tiny custom binary "request - response with GET and POST" protocol. I think the RESTful fits very well to the MQTT Topic world. Let me explain this idea on a concrete example. I am using HTTP REST notations for simplicity and to demonstrate the idea. Also note that I am using a JSON format for better readability for the metadata. JSON is not a format I propose for metadata.


Discovery of Topic Metadata
----------------------------------------

1. Client 1 decides he wants to know what metadata he can expect on topic "my/topic".
2. Client 1 requests mqttmetadata://localhost:80/my/topic with an GET
3. Server responds with a message {"metadataAvailable": true, "metadataStructure":  {.....STRUCTURE FORMAT...}};
4. Client 1 can generate the object parser for the topic dynamically / prepare for the data he expects
5. Client 1 subscribes to mqtt://localhost:1883/my/topic



Discovery of Topic Metadata with # Wildcards
----------------------------------------

1. Client 1 decides he wants to know what metadata he can expect on topic "my/topic/#".
2. Client 1 requests mqttmetadata://localhost:80/my/topic/#   with an GET
3. Server responds with a message {{"topic": "my/topic/A", "metadata{metadataAvailable: true, metadataStructure:  {.....STRUCTURE FORMAT...}},......}};
4. Client 1 can generate the object parser for the topic dynamically / prepare for the data he expects
5. Client 1 subscribes to mqtt://localhost:1883/my/topic/#

Discovery of Topic Metadata with + Wildcards
----------------------------------------

1. Client 1 decides he wants to know what metadata he can expect on topic "my/+/A".
2. Client 1 requests mqttmetadata://localhost:80/my/+/A   with an GET
3. Server responds with a message {{"topic": "my/topic/A", "metadata{metadataAvailable: true, metadataStructure:  {.....STRUCTURE FORMAT...}},......}};
4. Client 1 can generate the object parser for the topic dynamically / prepare for the data he expects
5. Client 1 subscribes to mqtt://localhost:1883/my/+/A


Discovery of all Metadata
----------------------------------------

1. Client 1 decides he wants to know all metadata on the MQTT broker.
2. Client 1 requests mqttmetadata://localhost:80/$metadata  with an GET
3. Server responds with a message {{"topic": "my/topic/A", "metadata{metadataAvailable: true, metadataStructure:  {.....STRUCTURE FORMAT...}},......}}; //Only topics which actually have metadata
4. Client 1 can generate the object parser for the topic dynamically / prepare for the data he expects
5. Client 1 subscribes to the topic he desires


The advantages with such an approach would be:

- It would be easy to administrate the allowed topic metadata for concrete topics on the broker
- If it is not desired to have structured data on a topic, the metadata discovery would simply return that there is unstructured data.
- Discovery of all metadata is simple
- The server can validate incoming messages if they match the metadata structure if desired


This example is far from perfect but I hope it's sufficient to get the idea. I'd love to hear your thoughts.

Best,
Dominik


Rick Bullotta wrote:

[Karl Palsson]

On Wed, Apr 23, 2014 at 11:14:30AM +0200, Dominik Obermaier wrote:
> Rick,
> I absolutely love the idea of a Metadata "discovery" mechanism. But I
> think do this properly, we should - as you suggested - think out of the
> protocol "box".
>
> Let me start with a provoking idea: Why not use a RESTful approach for
> metadata discovery?

You've heard of SOAP right? Your examples were basically SOAP wsdl requests, with different
names.

Personally I think good human readable documentation on a single website associated with the
domain of a broker is the best way out of "discovery" I've not yet seen any sort of magical
discovery protocol that really worked well without hand holding on both sides, and at that
point, you're not doing discovery anymore.

Still, I'd love to be wowed :) (I just don't think it has any business inside MQTT itself,
some of us have no need whatsoever for discovery protocols, and are very happy not to have
extra unsused/complicated/unncessary cruft lying around)

Cheers,
Karl P

[Paul Fremantle]

Dominik

I think this sort of thing might be useful additions to servers, and could potentially be "de-facto" standardised across servers, I would push hard not to add anything like this to the spec.

I'm happy for servers to expose stuff by HTTP, but one factor that is important for small devices is to only use a single protocol. 

All the "GET"s you are doing can easily be replicated by subscribes to a topic which would be more MQTTish.

Paul

[Dominik Obermaier]

Hi Karl,

yes, I’ve heard of SOAP and I unfortunately have to deal with SOAP more often than I want to.

Perhaps the examples were a bit unfortunate if it made the impression to you that it’s SOAP style. I absolutely don’t want any WSDL for MQTT discovery (and also no WADL). What exactly made the impression it’s SOAP? You can safely skip the “regenerate code by metadata” part ;-).

However, I don’t agree that human readable documentation is the best way of discovery for all use cases. I agree, that metadata discovery is out of the MQTT scope at the moment (also, the $SYS topic defintions are out of scope for the MQTT spec!). The basic idea is, if I understood Rick correctly, to bring discovery mechanisms to MQTT broker implementations.
Like $SYS topics. Not all MQTT brokers support the $SYS topic and nevertheless some people rely on that mechanism and even do monitoring with that. And the good thing is, if someone doesn’t need that additional stuff, it can be safely deactivated.

As far as I've heard, some people have done such MQTT metadata discovery stuff in projects. I would love to hear how they did it and perhaps it's possible to find the lowest common denominator.

Best,
Dominik

[Peter Hunsberger]

Karl has a sort of sideways point here.  What you describe isn't SOAP in the conventional sense, but the overall pattern is:  ask for a set of service descriptions, get back some metadata describing those services.  As such, maybe just using SOAP with as few modifications as possible makes sense?  I'm not normally a fan of SOAP for intra-enterprise discovery, but if you're talking extra-enterprise it does address some of the issues you'll eventually run into, in particular, trust models and end point authority validation...

Peter Hunsberger

[Rick Bullotta]

Hi, everyone. Yes, I'm an idiot and did "reply to author" for a few comments/responses.  Unfortunately, Google groups crappy UI doesn't include "Post Reply" in its dropdown menu. In any case, could you send me/repost the messages?  There was some good discussion in there that I don't want to try to recreate.  Many thanks and look forward to a healthy discussion.