MQTT Client reconnect after unsubscribe is creating queue again

[Nikhil Khatwani]

Hi,

I opened an issue few minutes back and it has been marked invalid and closed. Michaek Klishin commented queues are declared when

• Client connects with clean session = false and a client id

Can someone please help me understand when client has unsubscribed and is reconnecting, why we would queue needs to re created ?

Thanks,

Nikhil

[Nikhil Khatwani]

Let me try to clarify my use case here:

We would like to support millions of mqtt clients and will be doing push notification to them. Message load is very low for us. We wanted to optimize by avoiding millions of queues with metadata in RAM of rabbitmq cluster. 

What we came up with is whenever server want to send push notification it will 

1) create subscription(binding and queue) for existing device/(deviceId will be mqtt client id) 

2) disconnect the server side consumer

3) push the mqtt message over the above created queue

4) Client/Device can come online at any point of time 

5) connect with deviceId as client id

6) receive the message

7) unsubscribe to remove binding and queue

This avoids the lot of queues overhead for us.

After all this, client reconnect is creating the queue again. Can someone please help me understand when client has unsubscribed and is reconnecting, why we would queue needs to re created ?

Thanks,

Nikhil

[Michael Klishin]

There are two cases where queues are declared by the plugin:

 * Client sends a `SUBSCRIBE` frame

 * Client connects with `clean session = false` and a client id

So an MQTT-only solution to your question is "connect with clean session set to true" but queues

can have a TTL in RabbitMQ:

http://www.rabbitmq.com/ttl.html

(queues that are still in use won't be deleted by TTL, see the doc guide above for more info)

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Nikhil Khatwani]

Hi MK,

If i use clean session = true, queue will get deleted as soon as disconnect happens. TTL part you mentioned is kicking in only in case clean session = false. 

I tried below steps as an alternative but facing weird issue at step 3:

1) Server create subscription(binding and queue) for existing device/(deviceId will be mqtt client id) with clean session = false with queue ttl/suscription tll = 24 hrs in mqtt config. 

Disconnect the server thread above acting as consumer

2) Server push the mqtt message over the above created queue

3) Client/Device connect some time later with clean session = true with deviceId as client id

4) client should receive the message

At step 3, there is a pending message to be consumed but on client connect, immediately my queue and binding is getting deleted. 

Can you please help me with why queue and binding are getting deleted at step 3, Please have a look once at attached unit test "testMQTTSubscription" method

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

TTL is unaware of MQTT features and will kick in when a queue is unused (for the definition of

"unused", see the docs).

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nikhil Khatwani]

I tried setting queue TTL via below command and it is showing in policies - management console

sudo rabbitmqctl set_policy expiry ".*" '{"expires":259200000}'

MQTT client connects with clean session = true, queue is getting deleted as soon as mqtt client is disconnected. Nutshell, no effect of TTL policy. 

Please let me know if i am missing something. Going in circles now.

Also, I tried using clean-session=false for first time and clean-session=true afterwards also causing queue deletion issues. Please refer my previous comment step 3.

[Michael Klishin]

Clean session set to true also makes MQTT subscription queues auto-delete,

see https://github.com/rabbitmq/rabbitmq-mqtt/issues/37 and http://www.rabbitmq.com/tutorials/amqp-concepts.html.

If you want to *only* rely on queue (and/or message) TTL, connect with clean session = false and possibly subscribe QoS 1.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nikhil Khatwani]

We have discussed both clean session = true and clean session = false and their consequences on this thread but still use case i mentioned is not supported. 

Can you please suggest something to handle below use case which optimizes to reduce queues present on broker at any point of time, considering millions of ever increasing clients will be registered: 

"Create device subscriptions lazily via server when server has a message to push for client. Once client consumes the message, queue gets deleted and not recreated on client reconnects. Server only will recreate on need basis when it has a message to push"

I have already shared unit tests above in the thread.

Thanks,

Nikhil

[Michael Klishin]

Either queue TTL or subscribing with clean session set to "true" (which will use auto-deleted queues)
can work. Every active subscription will have a queue. 

> >>> wrote:
> >>>
> >>>> Hi MK,
> >>>>
> >>>> If i use clean session = true, queue will get deleted as soon as
> >>>> disconnect happens. TTL part you mentioned is kicking in only in case clean
> >>>> session = false.
> >>>>
> >>>> I tried below steps as an alternative but facing weird issue at step 3:
> >>>>

> >>>> 1) Server create subscription(binding and queue) for existing device/(*deviceId
> >>>> will be mqtt client id*) with *clean session = false *with queue

> >>>> ttl/suscription tll = 24 hrs in mqtt config.
> >>>> Disconnect the server thread above acting as consumer
> >>>> 2) Server push the mqtt message over the above created queue

> >>>> 3) Client/Device connect some time later with *clean session = true *
> >>>> with *deviceId as client id*

> >>>> 4) client should receive the message
> >>>>
> >>>> At step 3, there is a pending message to be consumed but on client

> >>>> connect, immediately my *queue and binding* is getting *deleted. *

> >>>>
> >>>> Can you please help me with why queue and binding are getting deleted
> >>>> at step 3, Please have a look once at attached unit test
> >>>> "testMQTTSubscription" method
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thursday, August 11, 2016 at 7:40:42 PM UTC+5:30, Michael Klishin
> >>>> wrote:
> >>>>>
> >>>>> There are two cases where queues are declared by the plugin:
> >>>>>
> >>>>> * Client sends a `SUBSCRIBE` frame
> >>>>> * Client connects with `clean session = false` and a client id
> >>>>>
> >>>>> So an MQTT-only solution to your question is "connect with clean
> >>>>> session set to true" but queues
> >>>>> can have a TTL in RabbitMQ:
> >>>>> http://www.rabbitmq.com/ttl.html
> >>>>>
> >>>>> (queues that are still in use won't be deleted by TTL, see the doc
> >>>>> guide above for more info)
> >>>>>
> >>>>>
> >>>>> On Thu, Aug 11, 2016 at 4:58 PM, Nikhil Khatwani

> >>>>>>> - Client connects with clean session = false and a client id

> To post to this group, send an email to rabbitm...@googlegroups.com.

[Nikhil Khatwani]

Completely agreed "Every active subscription will have a queue". 

But issue i reported on github was that mqtt subscribe followed by susbcription tll reached with no active consumer leading to susbcription expire. They why we are recreating the queue when client reconnects but not subscribes again. I don't understand why we are doing this in mqtt plugin code flow as you mentioned while closing the ticket. Can you explain rationale behind it please.

Thanks,

Nikhil

[Nikhil Khatwani]

Hi MK,

Can you please help me understand rationale behind creating queue on reconnect(not re susbcribe) of mqtt client even after subscription expiry on broker based on subscription_ttl.

Thanks