message not_route problems when there is a node restart

280 views
Skip to first unread message

Jiatong Shen

unread,
Dec 4, 2019, 9:11:38 PM12/4/19
to rabbitmq-users

Hi, we are experiencing a problem where some messages are routed to destination queues whenever there is a node restart.

 

  1. Background Info

 

  1. rabbitmq-server version: 3.7.13
  2. cluster with 3 nodes
  3. queues are not ha nor durable
  4. host arch: x86_64
  5. kernel version 4.15.0-65-generic, ubuntu bionic Ubuntu 18.04.3
  6. rabbitmq is deployment by a helm-charm offered by openstack-helm-infra

 

  1. Problem Description

 

we are running a openstack cluster both in dev and production enviroment, and openstack is of Rocky release. We have 3 controller nodes and several compute nodes. The scale is around 500-1000 queues for dev and 4000 queues in production. Whenever there is a node restarting, there are several queues having some problems receiving published messages. When publishing with mandatory bit set to true, I can see rabbitmq-server return "not_route"  to client. Under such circumstances, it seems that there is nothing I can do except manually rebuilding binding by first deleting and then creating an identitical one; or by manually deleting the queue and forcing openstack creating a same queue again.

 

We are not using ha or durable queues for some reasons. Of course, I have tried ha and durable queues, and seems that they are fine. The queues are binding to topic exchanges and having routing key identical to queue name. Every rpc remote synchronous call will create a direct exchange and a queue. It seems that both direct and topic exchange suffering from the same problem.

 

Surprisingly, problems are significantly mitigated when trying with a old release for example 3.7.4, so far we haven't observed any not routable message again. It also seems to work if I let client waiting for some seconds before reconnecting (kombu_reconnect_timeout=30)

 

  1. Reproduce Error

 

I believe problem could be easily reproduced if I have around 100 connections and 300 non-ha queues using a 3-node cluster.

 

  1. Configuration

 

  1. rabbitmq

 

cluster_formation.k8s.address_type = hostname

cluster_formation.k8s.host = kubernetes.default.svc.cluster.local

cluster_formation.node_cleanup.interval = 10

cluster_formation.node_cleanup.only_log_warning = true

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s

cluster_partition_handling = autoheal

listeners.tcp.1 = 0.0.0.0:5672

log.console.level = debug

loopback_users.guest = false

management.load_definitions = /var/lib/rabbitmq/definitions.json

net_ticktime = 5

queue_master_locator = min-masters

 

  1. policies

 

+----------+-----------------+----------+--------------------------------------------------------------------+-----------------------+----------+

|  vhost   |      name       | apply-to |                             definition                             |        pattern        | priority |

+----------+-----------------+----------+--------------------------------------------------------------------+-----------------------+----------+

| cinder   | ha_ttl_cinder   | all      | {"message-ttl": 70000}                                             | ^(?!(amq\.|reply_)).* | 0        |

| glance   | ha_ttl_glance   | all      | {"message-ttl": 70000}                                             | ^(?!(amq\.|reply_)).* | 0        |

| keystone | ha_ttl_keystone | all      | {"message-ttl": 70000}                                             | ^(?!(amq\.|reply_)).* | 0        |

| neutron  | ha_ttl_neutron  | all      | {"message-ttl": 70000}                                             | ^(?!(amq\.|reply_)).* | 0        |

| nova     | ha_ttl_nova     | all      | {"message-ttl": 70000}                                             | ^(?!(amq\.|reply_)).* | 0        |

+----------+-----------------+----------+--------------------------------------------------------------------+-----------------------+----------+

 

 

  1. oslo.messaging

 

[DEFAULT]

amqp_auto_delete = false

amqp_durable_queues = true

 

 

I am happy to provide more information if listed is not enough. And thank you for the help.

 

Jiatong Shen

unread,
Dec 11, 2019, 2:05:31 AM12/11/19
to rabbitmq-users
Hi please help, thx

Luke Bakken

unread,
Dec 11, 2019, 3:03:08 PM12/11/19
to rabbitmq-users
Hello,

Please try to reproduce this problem using the latest 3.7.X version of RabbitMQ: https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.7.23

It would also be helpful to know if you can reproduce this using version 3.8.2 https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.2

Your environment appears unique enough that it would be difficult for us to reproduce.

Thanks,
Luke

Jiatong Shen

unread,
Dec 12, 2019, 8:52:04 PM12/12/19
to rabbitmq-users
Hi Luke,

Thank you for answering. but unfortunately 3.7.23 does not help. I would like to try 3.8.2 but little afraid it will break schema..

Besides, I  have a theory: from this commit https://github.com/rabbitmq/rabbitmq-server/commit/eab08df1ffd3ba21174fa688469b547649bc9db2 , seems that delete queue and delete exchange trie table are done in different transactions. And i try to merge them into one transaction like this https://github.com/jshen28/rabbitmq-server/blob/28602d30b726241e8f297a47008fcfb6aea998a6/src/rabbit_amqqueue.erl#L1166  and rebuild docker image. Seems problems are gone.

Could you please review my code to see if it makes sense? thx

Norman Shen

unread,
Dec 12, 2019, 8:55:35 PM12/12/19
to rabbitmq-users

Luke Bakken

unread,
Dec 13, 2019, 10:54:58 AM12/13/19
to rabbitmq-users
Hi Norman,

Yes, that code changed to address some performance issues when deleting large numbers of queues. We're looking into it, but I remember this issue isn't easy to reproduce.

Thanks -
Luke

Jiatong Shen

unread,
Dec 13, 2019, 6:33:36 PM12/13/19
to rabbitm...@googlegroups.com
but I think it is quite easy as long as you use transient queues and number of queues as well as connections are large enough. whenever there is pod
got deleted or pod restart itself, we saw some queues do not work normally.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/da577dc0-9c93-4760-b0c3-df68cc3fe9b2%40googlegroups.com.


--

Best Regards,

Jiatong Shen

Luke Bakken

unread,
Dec 14, 2019, 10:56:19 AM12/14/19
to rabbitmq-users
Hi Norman,

An issue that is easy to reproduce is one where a script or other single command is provided to the RabbitMQ team that we can run to see the problem, every time. Anything more than that requires quite a bit of time from the team to set up an environment and try to reproduce the issue. The RabbitMQ team is small (< 10 people) and we have a wide range things to work on.

I've added an item to our internal work tracker to investigate this but I can't make guarantees as to when a resolution can be found.

Thanks -
Luke

Michael Klishin

unread,
Dec 16, 2019, 9:03:21 AM12/16/19
to rabbitmq-users
Your understanding is correct and this change was intentional, as were a few more in that area [1][2][3][4][5].

The function in question can cause massive lock contention in the schema database when it has to delete all queues and all bindings
at once. There are some benchmarks in some of those PRs or at least others linked to them. The difference is massive for some workloads.

Yes, this unfortunately introduce a potential observability problem but solves several others. To help reduce the likelihood of both
lock contention and [5] we turned all bindings for the default exchanges to be implicit (keep no rows in the routes/bindings tables).

We would consider undoing the change only with ample evidence that demonstrate that it would not bring back the same massive contention problem,
and it's a fairly involved set of benchmarks to run.

FWIW our primary goal was to address [5] but also contention, and the last 12 months suggest we've achieved it, so I'm inclined to keep
this as they are until a new schema database is adopted in 4.0 (we offer no ETA).

FTR, 3.7.23 would be no different from 3.7.13 in this regard but you should upgrade nonetheless.

Michael Klishin

unread,
Dec 16, 2019, 9:06:56 AM12/16/19
to rabbitmq-users
Also, as you have found out, messages can be published as mandatory [1] and there's a safe publishing mechanism [2]
on top of that. Your publishers should use both of them. I understand that historically OpenStack messaging library has adopted
a client library which lacked publisher confirms support and generally underpaid attention to publisher-side data safety.

We have recommended changes to that library all the way back in 2015 and to our knowledge, only some were adopted.

This is something that publishers should be able to deal with. [1][2] explain the features available in the protocol to cover those exact scenarios.

Jiatong Shen

unread,
Dec 16, 2019, 7:13:31 PM12/16/19
to rabbitm...@googlegroups.com
Hi Micheal,

thank you for you reply. I am newbie to erlang so forgive me what I said does not make sense but look at this https://github.com/rabbitmq/rabbitmq-server/blob/6ca895a30105c0e0553e56cead0d72cb3148a15c/src/rabbit_amqqueue.erl#L1971 code seems that clear up trie table is still done in one transaction why it is a bad idea to modify https://github.com/rabbitmq/rabbitmq-server/blob/6ca895a30105c0e0553e56cead0d72cb3148a15c/src/rabbit_amqqueue.erl#L1939 this code a little bit and delete binding in the transaction. Because, it seems to me that it only uses 10 queues in each iteration so I suppose locking is minimal.  what is your idea about this change https://github.com/jshen28/rabbitmq-server/blob/23bb3a2fcc0722e859ee914f476027cb8e7e8cee/src/rabbit_amqqueue.erl#L1169 ? I have tried my best to apply this piece of me in our test environment, it seems that the locking is not very bad although it only has around 2000 queues and we do not observe un-routed message.

about oslo.messaging, it now supports mandatory message at least in the master branch. But setting message to be mandatory does not solve problem, it only serves as an indicator and tells me there is a queue working abnormally and I should delete it either manually or explicitly by modifying code.

In terms of upgrading, I think I cannot persuade my boss unless I can prove that such binding inconsistent will not happen.. Right now the current solution is still using 3.7.4 which does not adopt that aggressive change. 

Best

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7e13fe82-7c34-41af-b32e-a873015fb920%40googlegroups.com.

Luke Bakken

unread,
Dec 19, 2019, 4:51:58 PM12/19/19
to rabbitmq-users
Hello,

Please see the following issues for more context:



Especially this comment:


The least efficient part of queue deletion is binding deletion because there is currently no way to load bindings of a queue (or exchange) without a full scan on one of the binding tables. This unfortunate design can be mitigated with secondary indices but this would be a breaking schema change and therefore can only go into 3.8.0

We can't consider your change without a way to reliably reproduce this issue, and without performance numbers for your change.

Thanks,
Luke

Jiatong Shen

unread,
Dec 19, 2019, 7:01:53 PM12/19/19
to rabbitm...@googlegroups.com
Hi Luke,

thank you for reply. I am confused.. by looking at this https://github.com/rabbitmq/rabbitmq-server/blob/3d6596cd943215d7ea18d8b458a6247547fcd777/src/rabbit_amqqueue.erl#L2002 function, by saying `binding` do you specifically say rabbit_binding or exchange trie? because if you are referring to the last one, seems although inefficient, they are still deleted in one transaction? why not split them up and put them inside https://github.com/rabbitmq/rabbitmq-server/blob/3d6596cd943215d7ea18d8b458a6247547fcd777/src/rabbit_amqqueue.erl#L1969?

sorry for asking silly questions

best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/6522ba9f-28bf-4705-970d-7e1f30bdfe61%40googlegroups.com.

Luke Bakken

unread,
Dec 19, 2019, 9:12:32 PM12/19/19
to rabbitmq-users
Hello,

These aren't silly questions.

Rather than pointing to specific lines of code, please open a pull request with your suggested changes and we can discuss them there. If your suggested changes aren't obvious, provide comments explaining them.

Thanks,
Luke
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages