Messages not being routed to queue? Direct works

[Roger Lipscombe]

I've got a RabbitMQ consumer that's no longer receiving messages that are sent to a topic exchange, even though it's (apparently) correctly bound.

My consumer claims to be using queue "amq.gen-arQ5ArflOq80ZcFOgs75iQ", which is apparently bound correctly:

$ sudo rabbitmqctl list_bindings | grep arQ exchange amq.gen-arQ5ArflOq80ZcFOgs75iQ queue amq.gen-arQ5ArflOq80ZcFOgs75iQ []

production exchange amq.gen-arQ5ArflOq80ZcFOgs75iQ queue production.models.updates []

If I publish a message using rabbitmqadmin, it doesn't arrive at this consumer:

$ rabbitmqadmin publish exchange=production routing_key=production.models.updates payload='{"command":"ping"}'

Message published

I have another consumer (on another host), which is correctly receiving messages with the same routing key.

However, if I send the message direct:

$ rabbitmqadmin publish routing_key=amq.gen-arQ5ArflOq80ZcFOgs75iQ payload='{"command":"ping"}'

Message published

...then it arrives correctly at the consumer.

I'm using RabbitMQ 3.6.3, on Erlang R16B03 on Ubuntu 14.04 on AWS; this is a three node cluster.

The consumer is written in Erlang, using amqp_client and rabbit_common 3.1.1. If I kill the consumer (so that it gets restarted by the supervisor, and reconnects), then messages start flowing again.

This happens fairly regularly after a (presumed) network glitch causes the consumer to reconnect, without restarting the whole client daemon.

How can I diagnose this further? How can I fix it?

I lied above; I actually had two broken consumers, so if anyone's got any ideas, I can attempt to diagnose with the still-broken one.

Regards,

Roger.

[Michael Klishin]

Hi Roger,

Am I misunderstanding your question or do you

expect a message routed to your queue to be

delivered to two consumers *at the same time*?

That's not how it works. A message is never delivered to more than one consumer at a time. If you want

both consumers to receive a copy, each of them

must have its own queue bound to the exchange in question. Some exchange types will happily

route a message to more than one queue

(topic and fanout ones are recommended

over others for such topologies, though).

This includes the direct exchange type

but not the default exchange

(which is used when you specify no

exchange to rabbitmqadmin).

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Tutorials 2 through 5 should cover routing, competing consumers (N consumers consuming

from a single queue) and most things in between:

http://www.rabbitmq.com/getstarted.html

This guide provides a 2 page long overview of the concepts:

http://www.rabbitmq.com/tutorials/amqp-concepts.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Roger Lipscombe]

On 12 May 2017 at 15:41, Michael Klishin <mkli...@pivotal.io> wrote:

Hi Roger,

Am I misunderstanding your question

Yes. Let me clarify it.

I have a number of consumers, each with their own queue. Those queues are bound to a topic exchange for a particular topic ("production.models.updates"). This works fine: if I publish a message to the topic exchange with that routing key, all of the consumers usually receive the message.

Except that it's not working right now. A few days ago, two of the consumers stopped receiving messages sent to the topic exchange. All other consumers are still working correctly.

Using "rabbitmqctl", I can see that the consumers are connected, that they have queues, and that those queues are bound to the correct exchange with the correct routing key.

If I publish a message via the default exchange to one of the named queues (i.e. direct addressing), then the message arrives at the (single) consumer. It just doesn't work for topic-routed messages.

 

That's not how it works.

I know.

 

A message is never delivered to more than one consumer at a time. If you want

both consumers to receive a copy, each of them

must have its own queue bound to the exchange in question.

They do.

Extra information:

If I use "rabbitmqctl cluster_status", I can see that there was a network partition at one point. I'm unsure how to read the information (I'll raise a separate question for that), but I'm puzzled as to how this can cause topic messages to fail to route, since I thought that unmirrored queues lived on a particular node -- which is the one I'm using.

[Michael Klishin]

Publishing via default exchange only rules out any issues with the queue itself.

There's http://www.rabbitmq.com/firehose.html which exists pretty much exclusively for debugging.

And as with most other issues, see server logs for clues.

--

[vikinghawk]

Roger, did you ever find out more about this?

We have seen this 2 or 3 times now in our dev environment (3.6.9) where messages just stop routing for entire topic exchanges. Other topic exchanges on the same virtual host work just fine. The queues, bindings, and consumers are all in place but messages are not routed via java client or management UI. Logs contain no errors pointing to a routing issue. Both times we have seen it was after node failures or cycles in a 2 node cluster. We use automatic recovery and the cluster appears to recover correctly. Doing a rolling cycle of the nodes will sometimes fix it but not always. In one case it fixed it for some topic exchanges but not all. Rebinding the queue to the exchange does not fix it either, but an unbind and rebind does seem to work. Unfortunately, we have 10000+ queues and bindings and that is not an easy fix. A full cluster downtime/reboot does seem to do the trick as well.

We run the message_timestamp plugin in all our domains. In Dev, we also have the top and firehose plugins enabled (but turned off). Those 2 plugins are the only difference between dev and our other clusters. So I'm not sure if its just a coincidence that we only see this in prod or if perhaps the firehose plugin could be a culprit.

[Jiatong Shen]

vikinghawk, 

   do you have any follow-up on this issue? we have a rabbitmq 3.7.13 cluster , with following configuration 

cluster_formation.k8s.address_type = hostname

cluster_formation.k8s.host = kubernetes.default.svc.cluster.local

cluster_formation.node_cleanup.interval = 10

cluster_formation.node_cleanup.only_log_warning = true

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s

cluster_partition_handling = autoheal

listeners.tcp.1 = 0.0.0.0:5672

loopback_users.guest = false

management.load_definitions = /var/lib/rabbitmq/definitions.json

queue_master_locator = min-masters

and the cluster is running on top of a k8s cluster. We observe messaging not correctly routed after pod deleted or networking flapping.

It seems only be solved after I delete erroneous queue, reboot cluster does not seem to help.