Using MQTT for storage/persistence?

[Nicholas Humfrey]

Hello,

Is it good or bad practice to use MQTT's retained messages feature for storage/persistence of configuration data?

1) First example: storing settings for a stateless sensor node - polling frequency, unit of measurement etc. Settings would probably be set by a server/configuration tool and then fetched by subscribing to a topic based on the node's id.

2) Second example: storing application settings - a mobile app gets its settings data by reading from a topic. And also writes/updates settings by publishing (JSON for example) to that topic.

3) Finally, a web application, such as Node-RED. Instead of writing the flows to disk, is there any reason why it would be bad to write the flows data to an MQTT topic instead?


The end result is using the broker as a sort of Key-Value store with subscription support.

I guess the main dependency is the MQTT broker having support for persisting the messages to disk. Mosquitto supports this but it is turned off by default.


Thanks,

nick.

[Nicholas O'Leary]

Hi Nick,

I principle there is nothing wrong with this... I've seen it done and done it myself.

You might want to look at the Argot project as an example that uses retained topics to share device api and metadata.

As you say, the safety of it is as good as the brokers persistence. Security is an additional concern as you have made configuration data network accessible, so you better make sure you know who can read/write it.

On the node-red front, you may have seen that I have just this week abstracted out the storage mechanism - primarily to help make the code more testable, but in such a way that an mqtt based storage mechanism could be plugged in (not that I have any intention of doing so).

Personally, if you start looking at network stored configuration, I'd look at something like redis that gives you similar lightweight access, can be setup to do pub/sub notification of changes, but has much richer data structure support. For example built in support for queues, lists, sets, hashes etc.

Regards,

Nick

--
--
To learn more about MQTT please visit http://mqtt.org

To post to this group, send email to mq...@googlegroups.com
To unsubscribe from this group, send email to
mqtt+uns...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/mqtt

---
You received this message because you are subscribed to the Google Groups "MQ Telemetry Transport" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[David Ryan]

Hi Nick(s),

Just wanted to point out that what I did with Argot was really just a test to see if it was a good place to store metadata.  I'm not completely happy with using this feature to store metadata.  The bigger problem of configuration and configuration synchronization is an area I think needs further investigation.  Redis looks interesting for that.

Can anyone point out any underlying theory for master-slave synchronization over publish/subscribe architectures?

Regards,
David.

[Dougie Lawson]

On 13 November 2013 23:09, David Ryan <da...@livemedia.com.au> wrote:

Just wanted to point out that what I did with Argot was really just a test to see if it was a good place to store metadata.  I'm not completely happy with using this feature to store metadata.  The bigger problem of configuration and configuration synchronization is an area I think needs further investigation.  Redis looks interesting for that.

 

Having worked with IBM's first messaging and queuing system for thirty+ years, I've spent 30+ years discouraging folks from keeping their data in my message queues. If you have persistent data then it belongs in a file or in a database. My messaging and queuing system (IMS) happens to include the hierarchical database DL/I and can interface to DB2. Persistent data is less likely to go missing when it's stored (and logged) in a database that can then be backed up.

If you keep data in a MQTT queue there's a high risk that it is going to go missing. There's no logging, there's no restart/recovery, there's no image copying with MQTT.

Put your data in a database where it belongs.

--
http://twitter.com/DougieLawson

[Paul Fremantle]

Dougie

Excellent points. 

What do you think about putting the config in a database and then regularly updating a topic with the config? That way the client can have a simple subscription while there is a clear approach to restart/recovery. 

Paul

[David Ryan]

Hi Dougie,

Thanks for the response.  I've been having a bad niggling feeling about using the feature for storing metadata.  You've settled it for me that this is the wrong way to use it.

This then raises the question, if the configuration/metadata is stored in a database or other persistent storage then what is the best way to manage synchronization between server and device.  The simplest method I've thought of so far; when a device connects it publishes a message with a timestamp of its last configuration update and a hash of its current configuration to /server/sync.  If the server has an configuration or update different to what the device provided, the server sends a message with config changes to /<deviceid>/config.

Is this style of interaction the norm for config synchronization?  Interested in others experience and thoughts on this.

Regards,
David.

--

[Dougie Lawson]

On 14 November 2013 08:31, Paul Fremantle <pa...@wso2.com> wrote:

Dougie

Excellent points. 

What do you think about putting the config in a database and then regularly updating a topic with the config? That way the client can have a simple subscription while there is a clear approach to restart/recovery. 

Paul

Hi Paul,

My approach would be to have a long-running server connected to MQTT subscribed to a special topic.

The user needing config data would publish a message on that topic with the data needed to form a coherent SQL predicate.

The server gets the predicate, sanity checks it (because we should always sanity check SQL predicate that arrive from outside our domain), then feeds it into an SQL SELECT query to retrieve the required rows.

It then publishes those rows on a reply topic.

Effectively you've written a simply transaction processor. You may need to consider some form of authorisation for that.

Should be simple with something like python to knock that up in half a day.

Regards, Dougie.

--
http://twitter.com/DougieLawson

[Matteo Collina]

Hi Everyone,

The MQTT broker I have written, Mosca http://github.com/mcollina/mosca, uses a real database for both retained and offline messages. It performs very well in these scenarios.

Mosca supports Redis, MongoDB, and LevelDB as databases, but it can be easily extended to use SQL databases.

Cheers,

Matteo

[Nicholas Humfrey]

On 2013-11-16 19:47, Matteo Collina wrote:
> Hi Everyone,
>
> The MQTT broker I have

> written, Mosca http://github.com/mcollina/mosca [1], uses a real

> database for both retained and offline messages. It performs very
> well in these scenarios.
> Mosca supports Redis, MongoDB, and LevelDB as databases, but it can
> be easily extended to use SQL databases.
>
> Cheers,
>
> Matteo

Hello,

That is good to know - have you been using it to persist configuration
data?


I have been thinking about it more and the persisting of retained
messages, seems like less of a challenge and a concern than the protocol
itself.

If you are using a Subscribe, as a Get, then all error conditions
appear to be a timeout.
- Client error (invalid topic format)
- Not found (unknown topic)
- Authorisation error
- Server error
- Network error (a real timeout?)
- Error persisting the retained message

Maybe my head has been in the world of HTTP for too long.
I guess one possibility would be to add a return code to the Suback
message.

Is this any more of a concern that using MQTT for normal PubSub?


nick.

[Nicholas Humfrey]

On 2013-11-15 18:42, Dougie Lawson wrote:

> On 14 November 2013 08:31, Paul Fremantle <pa...@wso2.com [1]> wrote:
>
>> Dougie
>>
>> Excellent points. 
>>
>> What do you think about putting the config in a database and then
>> regularly updating a topic with the config? That way the client can
>> have a simple subscription while there is a clear approach to
>> restart/recovery. 
>>
>> Paul
>
> Hi Paul,
>
> My approach would be to have a long-running server connected to MQTT
> subscribed to a special topic.
>
> The user needing config data would publish a message on that topic
> with the data needed to form a coherent SQL predicate.
>
> The server gets the predicate, sanity checks it (because we should
> always sanity check SQL predicate that arrive from outside our
> domain), then feeds it into an SQL SELECT query to retrieve the
> required rows.
>
> It then publishes those rows on a reply topic.
>

> Effectively youve written a simply transaction processor. You may

> need
> to consider some form of authorisation for that.
>
> Should be simple with something like python to knock that up in half
> a
> day.

Thanks for suggesting an alternative approach. Makes a lot of sense
doing this over MQTT, rather than having to implement multiple
application protocols on the device.

The extra layer of protocol also gives a lot more options and would
make auto-provisioning of sensor configuration easier. It would also
make it much easier to report error conditions with persisting to the
data store.

However it feels an awful lot like RPC over PubSub, which scares me a
bit!
Is that a good thing or a bad thing?

I instinctively try and make MQTT resource centric - but then you have
a very limited set of verbs...


nick.

[Matteo Collina]

Il giorno lunedì 18 novembre 2013, Nicholas Humfrey ha scritto:

That is good to know - have you been using it to persist configuration data?

Not on a large scale. I tested the approach and it works.

 

I have been thinking about it more and the persisting of retained messages, seems like less of a challenge and a concern than the protocol itself.

If you are using a Subscribe, as a Get, then all error conditions appear to be a timeout. 

- Client error (invalid topic format)
- Not found (unknown topic)
- Authorisation error
- Server error
- Network error (a real timeout?)
- Error persisting the retained message

We are speaking about an embedded device. Do you think it's possible to recover from any of these situations?

You subscribe to a topic and then you receive your config. If it's not there, maybe you can use some (sane) defaults. 

Mosca has some plugin capabilities. 

Is this any more of a concern that using MQTT for normal PubSub?

I don't think so.. but hey!

Matteo

[Nicholas Humfrey]

On Tuesday, 19 November 2013 09:22:36 UTC, Matteo wrote:

Il giorno lunedì 18 novembre 2013, Nicholas Humfrey ha scritto:

That is good to know - have you been using it to persist configuration data?

Not on a large scale. I tested the approach and it works.

 

I have been thinking about it more and the persisting of retained messages, seems like less of a challenge and a concern than the protocol itself.

If you are using a Subscribe, as a Get, then all error conditions appear to be a timeout. 

- Client error (invalid topic format)
- Not found (unknown topic)
- Authorisation error
- Server error
- Network error (a real timeout?)
- Error persisting the retained message

We are speaking about an embedded device. Do you think it's possible to recover from any of these situations?

You subscribe to a topic and then you receive your config. If it's not there, maybe you can use some (sane) defaults. 

Very good point, yes, very little that can be done.

And not having multiple different types of failure condition makes it easier to code for.

nick.