Cannot remove x-expires value of 30 minutes on all queues

[Sam]

Copied from my SO post: http://stackoverflow.com/questions/34332367/rabbitmq-randomly-have-x-expires-value-set

up votedown votefavorite

I have a RabbitMQ service running with the MQTT plugin. Everything was working fine and I had set a queue TTL for 3 days using the command sudo rabbitmqctl set_policy expiry ".*" '{"expires":259200000}' --apply-to queues Today, I noticed an issue where some messages were not being delivered and when I looked at the RabbitMQ server dashboard, I saw that every queue under "Features" has "D" (for durable), "expiry" for the 3 day TTL I set and "Exp" with a 30 minute value. Looking at this post RabbitMQ: What is the default x-message-ttl value there should be no default TTL value. I am confused where the 30 minute TTL came from and I am having a hard time figuring out how to remove it. When I go to the Admin tab in the dashboard, I only see my 3 day TTL. I removed that, but nothing happened to the "Exp" value. Before I restart my RabbitMQ server, wanted to see if there is any way to remove this 30 minutes TTL. The 30 minute TTL is making me lose important data when clients are idle for more than 30 minutes and I need to remove it.

[Michael Klishin]

I suspect you use MQTT. It has queue expiration on by default: you can undo it by
configuring rabbitmq_mqtt.subscription_ttl to undefined. The default is 30 minutes.

See http://www.rabbitmq.com/mqtt.html.

The reason why it exists is that without Clean Session flag set, MQTT consumer can go away at any
time and RabbitMQ won't possibly know when to clean up its data.

That said, I think we should revisit the default to be 1 day or so. 

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Sam]

Thanks, will try that. 

PS: would be nice to have default policies showing up on the management console as well.

[Michael Klishin]

On 18 December 2015 at 15:14:27, Sam (samn...@gmail.com) wrote:
> PS: would be nice to have default policies showing up on the management
> console as well.

Well, they are not really policies: they are plugin defaults. We can try making them more visible
in the docs but more often than not, the users don't read those before trying something out.

FYI: https://github.com/rabbitmq/rabbitmq-mqtt/issues/49 

[Sam]

I read the doc a few weeks back when i was setting things up and just ran in to this issue, and forgot what it had mentioned and was going purely on the other SO thread and the management console values :(

I added the following to my config:

{rabbitmq_mqtt, [{subscription_ttl, undefined}]}

but now I am getting an issue where my client seem to get an error from the server when trying to subscribe saying the x-expire value is inconsistent. Anyway to handle this? I was not expecting this to happen, was assuming existing queues will have their expiry remove and things to work smoothly...did I miss something?

=ERROR REPORT==== 18-Dec-2015::12:36:33 ===

** Generic server <0.720.0> terminating 

** Last message in was {'EXIT',<0.716.0>,

                        {{shutdown,

                          {server_initiated_close,406,

                           <<"PRECONDITION_FAILED - inequivalent arg 'x-expires' for queue 'mqtt-subscription-4cab764f-9f24-11e5-b770-12442bac86e6qos1' in vhost '/': received none but current is the value '1800000' of type 'long'">>}},

                         {gen_server,call,

                          [<0.726.0>,

                           {call,

                            {'queue.declare',0,

                             <<"mqtt-subscription-4cab764f-9f24-11e5-b770-12442bac86e6qos1">>,

                             false,true,false,false,false,[]},

                            none,<0.716.0>},

                           infinity]}}}

** When Server state == {state,amqp_direct_connection,

                         {state,'rab...@xxx.xx.xx.xx:',

                          {user,<<"guest">>,

                           [administrator],

                           [{rabbit_auth_backend_internal,none}]},

                          <<"/">>,

                          {amqp_params_direct,<<"guest">>,none,<<"/">>,

                           'rab...@xxx.xx.xx.xx:',

                           {amqp_adapter_info,

                            {0,0,0,0,0,65535,44063,13073},

                            1883,

                            {0,0,0,0,0,65535,31719,11483},

                            53865,

                            <<"xxx.xx.xx.xx::53865 -> xxx.xx.xx.xx::1883">>,

                            {'MQTT',"3.1.1"},

                            [{ssl,false}]},

                           []},

                          {amqp_adapter_info,

                           {0,0,0,0,0,65535,44063,13073},

                           1883,

                           {0,0,0,0,0,65535,31719,11483},

                           53865,

                           <<"xxx.xx.xx.xx::53865 -> xxx.xx.xx.xx::1883">>,

                           {'MQTT',"3.1.1"},

                           [{ssl,false}]},

                          <0.723.0>,undefined,1450442193430},

                         <0.722.0>,

                         {amqp_params_direct,<<"guest">>,none,<<"/">>,

                          'rab...@xxx.xx.xx.xx:',

                          {amqp_adapter_info,

                           {0,0,0,0,0,65535,44063,13073},

                           1883,

                           {0,0,0,0,0,65535,31719,11483},

                           53865,

                           <<"xxx.xx.xx.xx:53865 -> xxx.xx.xx.xx::1883">>,

                           {'MQTT',"3.1.1"},

                           [{ssl,false}]},

                          []},

                         0,

                         [{<<"capabilities">>,table,

                           [{<<"publisher_confirms">>,bool,true},

                            {<<"exchange_exchange_bindings">>,bool,true},

                            {<<"basic.nack">>,bool,true},

                            {<<"consumer_cancel_notify">>,bool,true},

                            {<<"connection.blocked">>,bool,true},

                            {<<"consumer_priorities">>,bool,true},

                            {<<"authentication_failure_close">>,bool,true},

                            {<<"per_consumer_qos">>,bool,true}]},

                          {<<"cluster_name">>,longstr,

                           <<"rabbit@xxx-xx-x-xxx-xx">>},

                          {<<"copyright">>,longstr,

                           <<"Copyright (C) 2007-2015 Pivotal Software, Inc.">>},

                          {<<"information">>,longstr,

                           <<"Licensed under the MPL.  See http://www.rabbitmq.com/">>},

                          {<<"platform">>,longstr,<<"Erlang/OTP">>},

                          {<<"product">>,longstr,<<"RabbitMQ">>},

                          {<<"version">>,longstr,<<"3.5.6">>}],

                         none,false}

** Reason for termination == 

** {unexpected_msg,

       {'EXIT',<0.716.0>,

           {{shutdown,

                {server_initiated_close,406,

                    <<"PRECONDITION_FAILED - inequivalent arg 'x-expires' for queue 'mqtt-subscription-4cab764f-9f24-11e5-b770-12442bac86e6qos1' in vhost '/': received none but current is the value '1800000' of type 'long'">>}},

            {gen_server,call,

                [<0.726.0>,

                 {call,

                     {'queue.declare',0,

                         <<"mqtt-subscription-4cab764f-9f24-11e5-b770-12442bac86e6qos1">>,

                         false,true,false,false,false,[]},

                     none,<0.716.0>},

                 infinity]}}}}

[Michael Klishin]

On 18 December 2015 at 15:52:12, Sam (samn...@gmail.com) wrote:
> <<"PRECONDITION_FAILED - inequivalent arg 'x-expires' for
> queue 'mqtt-subscription-4cab764f-9f24-11e5-b770-12442bac86e6qos1'
> in vhost '/': received none but current is the value '1800000'
> of type 'long'">>}}

When you change TTL, existing queues need to be deleted. Queue properties cannot be changed
on the fly in RabbitMQ.

Although for TTL specifically we can improve this by making the plugin define a policy which can be altered
on the fly. 

[Nayana Samaranayake]

Is there a way to restart the server and have only the new queues created inherit the new TTL value, and for old ones to operate as they were? I am trying to figure out how to gracefully change something like this in a production env without losing any messages. 

Also, is there a preferred way to delete all the queues? 

Thanks for all the help.

[Michael Klishin]

On 18 December 2015 at 16:03:38, Nayana Samaranayake (samn...@gmail.com) wrote:
> Is there a way to restart the server and have only the new queues
> created inherit the new TTL value, and for old ones to operate
> as they were? I am trying to figure out how to gracefully change
> something like this in a production env without losing any messages.

That's how it works but if you subscribe with the same client ID and QoS level as before,
queue name will be the same but TTL will differ, so the queue cannot be declared.

Disabling the TTL in the config entirely and using a policy (queue names that back
MQTT subscriptions have standardised names) will allow you to change policy without
the error above, because the TTL won't be really a queue property. 

> Also, is there a preferred way to delete all the queues?

You can do it using a RabbitMQ client, in the management UI or HTTP API.

[Nayana Samaranayake]

most probably I missed some docs. Is there a way to find the values set by default for me to figure out what I need to override? My test service config just had [{rabbit, [{loopback_users, []}]}]. to allow guest access from outside localhost.

Is the list of properties mentioned at https://www.rabbitmq.com/mqtt.html under "Here is a sample configuration that sets every MQTT option:" the default values? Would be great if you can point me at a doc that I might have missed.

[{rabbit, [{tcp_listeners, [5672]}]}, {rabbitmq_mqtt, [{default_user, <<"guest">>}, {default_pass, <<"guest">>}, {allow_anonymous, true}, {vhost, <<"/">>}, {exchange, <<"amq.topic">>}, {subscription_ttl, 1800000}, {prefetch, 10}, {ssl_listeners, []}, %% Default MQTT with TLS port is 8883 %% {ssl_listeners, [8883]} {tcp_listeners, [1883]}, {tcp_listen_options, [binary, {packet, raw}, {reuseaddr, true}, {backlog, 128}, {nodelay, true}]}]} ].

[Michael Klishin]

On 18 December 2015 at 16:16:23, Nayana Samaranayake (samn...@gmail.com) wrote:
> most probably I missed some docs. Is there a way to find the values
> set by default for me to figure out what I need to override? My test
> service config just had [{rabbit, [{loopback_users, []}]}].
> to allow guest access from outside localhost.
>
> Is the list of properties mentioned at https://www.rabbitmq.com/mqtt.html
> under "Here is a sample configuration that sets every MQTT option:"
> the default values? Would be great if you can point me at a doc that
> I might have missed.

They are mostly default.

You can find all defaults here (under the env "section") :
https://github.com/rabbitmq/rabbitmq-mqtt/blob/master/src/rabbitmq_mqtt.app.src#L7