How to clean the 'inactive topics' published with mqtt client?

[Henry John]

Hi, all

     I see no bytes in MQTT that will clean up inactive topics.  So if somebody published a topic with payload, but nobody would subscribe this topic, Then how can MQTT delete  this topic automaticly?

     Help infomation about the mosquitto server is welcome. Thanks

Regards! 

[Ian Craggs]

Topics do not have to have an independent life in MQTT.  There are subscriptions and messages (or publications).  If a message is sent to a topic for which there are no subscriptions, I would expect the server to drop the message - no other clean up is necessary.

This behaviour could vary between implementations though.

[Laing, Michael]

Huh. I would expect the server to retain the message if the retain bit were set.

From the spec:

If the RETAIN flag is set to 1, in a PUBLISH Packet sent by a Client to a Server, the Server MUST store the Application Message and its QoS, so that it can be delivered to future subscribers whose subscriptions match its topic name [MQTT-3.3.1-5]. When a new subscription is established, the last retained message, if any, on each matching topic name MUST be sent to the subscriber [MQTT-3.3.1-6].

On Sat, Feb 28, 2015 at 3:11 PM, Ian Craggs <icr...@gmail.com> wrote:

Topics do not have to have an independent life in MQTT.  There are subscriptions and messages (or publications).  If a message is sent to a topic for which there are no subscriptions, I would expect the server to drop the message - no other clean up is necessary.

This behaviour could vary between implementations though.

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To post to this group, send email to mq...@googlegroups.com.
Visit this group at http://groups.google.com/group/mqtt.
For more options, visit https://groups.google.com/d/optout.

[Darren Clark]

The exception being retained messages, rather than just messages with no subscriber. In the case of retained messages, there is no way for the server to know that there will never be a subscriber to that topic, and clear the message autonomously. 

As the owner of your particular system _you_ may know that there will never be a subscriber to that topic. In that case you can send an empty message to the topic with retained flag set to 1. That will remove that topic from storage on the broker.

Someone correct me if I'm wrong, but I believe that clearing retained messages with an empty message is new in the 3.1.1 spec vs 3.1. However, almost all broker implementations supported this feature anyway.

-Darren

On Sat, Feb 28, 2015 at 12:11 PM, Ian Craggs <icr...@gmail.com> wrote:

Topics do not have to have an independent life in MQTT.  There are subscriptions and messages (or publications).  If a message is sent to a topic for which there are no subscriptions, I would expect the server to drop the message - no other clean up is necessary.

This behaviour could vary between implementations though.

[Laing, Michael]

The length of retention may be up to implementation.

In my implementation based on mosquitto and using Cassandra or PostgreSQL, length of retention is set by configurable metadata associated with an ACL.

[Darren Clark]

Michael,

That's pretty clever. Particularly since a decent amount of my retained data is updated periodically, so data on temperature more than a couple hours old isn't terribly useful since that device is offline anyway. However, other data(like last known WiFi signal strength) can help identify problems.

-D

[Roger Light]

> Someone correct me if I'm wrong, but I believe that clearing retained
> messages with an empty message is new in the 3.1.1 spec vs 3.1. However,
> almost all broker implementations supported this feature anyway.

This feature was added to the spec in v3.1.1, but was present in at
least some brokers since mqtt v3.

There is a utility in the Paho Python examples directory that lets you
clear retained messages based on a wildcard subscription pattern:

http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.python.git/tree/examples/mqtt_clear_retain.py

Cheers,

Roger

[Ian Craggs]

The original question didn't mention retained messages.

I claim the credit for allowing retained messages to be deleted with a zero-length payload in RSMB in about 2008.  Along with Dave Conway-Jones and Andy Stanford-Clark who also wanted the function  :-)  Personally I wanted to be able to fully clean up state before running tests, in a non-broker specific way.

Unless someone wants to put forward an earlier claim...

Ian

[Henry John]

Thanks all for your suggestions.  We want to use mosquitto( the MQTT server)  to push specified message to specified users, so we make every clientid as a 'topic'. Each client should subscribe its clientid as its identical topic,  When we designed to push a message to client_id_X, we will publish our message to the topic 'client_id_X', then this client and ONLY this client could receive the message.

But we don't know user uninstall,  so there is no unsubscribe operation on this client, And there will no 'publish command with retain set to 1 and payload empty' to this clientid (the topic). We need the broker to do automatic cleanning about published topics.

That is the background of my question.

Regards!

在 2015年2月28日星期六 UTC+8下午4:22:55，Henry John写道：

[Roger Light]

Hi Henry,

Do your clients use clean session set to true or false? If clean
session is set to true then when your client disconnects all of its
subscriptions will be removed by the broker, along with any queued
messages. If clean session is set to false in your clients, then the
subscription information and queued messages will be kept by the
broker. You can have these removed automatically by mosquitto by
setting the persistent_client_expiration value in your configuration
file. This will remove the client information in the same way as if it
had connected with clean session set to true, after the specified time
has passed since they last disconnected.

Cheers,

Roger

[Henry John]

Thanks,Roger

    As of our use case, there are two method to let  the client receive messages pushed to it after its reconnection:

    1、like what you said, the client will always set its 'clean session' tag as false, so that this session is saved in mosquitto db even if the user is offline, so that it can receive messages after its reconnection. Yes, we should set  persistent_client_expiration value to eliminate expired clients.

    2、The client will always set 'clean session' as false, so after its lost connect, the session is removed. But the publisher will set message tag as 'retained', so after the client reconnect and subscribe with the same client_id, It will also receive  messages published to it.

    Now I still have no idea which one is better, but it seems that neither of method can clean published topics that were not subscribed for a long time.

Regards!

在 2015年3月3日星期二 UTC+8下午3:40:41，Roger Light写道：

[Henry John]

fix.

 2、The client will always set 'clean session' as true, so after its lost connect, the session is removed. But the publisher will set message tag as 'retained', so after the client reconnect and subscribe with the same client_id, It will also receive  messages published to it.

在 2015年3月3日星期二 UTC+8下午4:35:27，Henry John写道：

[Roger Light]

On Tue, Mar 3, 2015 at 8:37 AM, Henry John <qhlonl...@gmail.com> wrote:
> fix.
> 2、The client will always set 'clean session' as true, so after its lost
> connect, the session is removed. But the publisher will set message tag as
> 'retained', so after the client reconnect and subscribe with the same
> client_id, It will also receive messages published to it.

In this case when the client reconnects it will only receive the last
message published to the topic. There is only ever one retained
message per topic.

Something else to bear in mind - you can use the "will" of a client to
remove retained messages. Set the will topic as appropriate, set the
will retained flag to true and will payload to be empty and if the
client unexpectedly disconnects (i.e. without sending a DISCONNECT),
the retained message at the will topic will be cleared.

Cheers,

Roger

[Henry John]

Other problem： When I published messages, I set retain =1,  Then I use a second client_id to subscribe this topic, Each time I restart mosquitto_sub, I received the same message again. Why? I would like to receive this message only once for certain client_id, no matter it is retained or not .

Regards! 

在 2015年2月28日星期六 UTC+8下午4:22:55，Henry John写道：

Hi, all

[Nicholas O'Leary]

Henry,

that is how retained messages work. It only stores the last message on a given topic that was published with the retained flag, and when every time a client subscribes to a topic, it receives the topic's retained message (if there is one).

The guys at hivemq wrote a blog post this week all about retained messages - they explain them very well: http://www.hivemq.com/mqtt-essentials-part-8-retained-messages/

Nick

--

[Laing, Michael]

We publish our messages as retained to the topic '.../<client-id>/<unique-id>'

Each message expires according to a TTL configured by my internal users.

When the client connects she subscribes to sub '.../<client-id/#', among others, receiving her current message subtree.

At any one time we have millions of such messages, stored in Cassandra, and shared amongst our custom "brokers".

This is the retained message store we are enhancing and porting to mosquitto 1.4 as a proof of concept.

It should be available in github as experimental code based on the 1.4 release by the end of the month.

ml

[Henry John]

Thanks, knolleary,

      Very helpful!

     Thanks

Regards!

在 2015年3月3日星期二 UTC+8下午5:59:04，knolleary写道：