Publisher persistence in network failures.

[Nowa Concordia]

Hello,

I've come up with the requirement to address the issue of persisting the unpublished messages in case of a connection loss to the broker.

I already use the QOS  and clean session option to retrieve missed messaged for the subscribers, but here it's' the other story of clients 

being unable to publish the messages to the broker. 

Had this been already addressed in the Paho libraries or will i have to implement sort of a persistent queue separately to overcome the problem? 

Any suggestion would be highly appreciated.

Thank you.

-Nowa.

[Hans Jespersen]

How constrained is your device? Can you run a local lightweight mqtt broker like mosquito on the same device as node-red and then back haul messages to a central broker via an mqtt bridge link (i.e. broker to broker link)? This architecture gives you disconnected local buffering of data without having node-red or custom code needed.

[Nowa Concordia]

This is android phone, acting as a gateway for a more constrained device/devices. There is a primary app running on it which does the controlling part. Your suggestion seems be able to do the job, but will that be an acceptable practice for production? 

[Nowa Concordia]

Further i Found MqttDefaultFilePersistence class in paho client library, for what purpose do we need a Client persistence if not for cases like above?

[Dominik Obermaier]

Hi Nowa,

afaik the persistence is used for persisting the QoS message flow state in case of QoS 2 / 1 messages. It does not support buffering message when you are offline, though.

Best,

Dominik

On 5 Aug 2015 at 05:53:19, Nowa Concordia (slam...@gmail.com) wrote:

Further i Found MqttDefaultFilePersistence class in paho client library, for what purpose do we need a Client persistence if not for cases like above?

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To post to this group, send email to mq...@googlegroups.com.
Visit this group at http://groups.google.com/group/mqtt.
For more options, visit https://groups.google.com/d/optout.

[Nowa Concordia]

Hi Dominik,

Thank you for answering and it now make sense :). So I think using a persistent job queue will do the job for me.

-Nowa.

[Laing, Michael]

The Paho python client persists published messages in the absence of a current connection.

It uses an in-memory list and attempts to send them when reconnected.

This is in addition to in-flight message handling.

ml

[Hans Jespersen]

In memory persistence works for short outages or for infrequent publish rates but if you expect long outages, high publish rates, or outages that need to span rebooting the phone, then you need to store the data in a non-volatile message spool.

[Ian Craggs]

Just to clarify.  There are various sorts of "persistence":

1) Continuing the sending and receiving of messages when the connection to the broker is re-established after a break (but not allowing the sending of new messages when the connection is broken).  All client libraries should support this, for QoS 1 and 2 messages and connecting non-clean session. 

2) Allowing the application to publish new messages when the connection to the broker is not established.  This is not supported by the Paho Java and C clients, but is by the Python client.

3) When the application using the client library is stopped, so that the client objects are destroyed.  The state of the client objects is recreated when the application restarts.  The Paho Java and C (not embedded) have disk persistence, implemented as a replaceable module, to support this. 

Ian

[Nicolas Jouanin]

What about other kind of messages like SUBSCRIBE or UNSUBSCRIBE ? 

Should a client "bufferize" such messages while the broker connection is down ?

[Nowa Concordia]

I implemented the offline buffer in job queue manner using Priority Job Queue for Android. It's network state aware so does pretty good job. But the connectivity handling on the Mqtt service has to be so perfect that i noticed atleast one message being lost (QOS2) in possible connection drops. The loss is not consistent, I suspect this is due to a token timeout, if in that case the complete message publish job's getting discarded?

 In the re connection process there is a rare condition i've been seeing lately, that the mqtt re connection process halts with 'unexpected error' and couldn't figure a way to avoid it or recover from it on the fly without restarting the application.

One other thing is the maximum publish rate. Any information regarding this that you may know of?

Cheers.

[Nowa Concordia]

Number 2 is what i'm looking at, I heard it's available in Python client, but i'm working on android, will the future versions of Java client support this?