Trying to understand Unroutable (drop) metric

[Vilius Šumskas]

Hello,

 

we have put a new 3.9 three node cluster into production. Everything seems to work fine and we don‘t see any issues from consumers or producers side.

 

However I found a new metric – Unroutable (drop) which is constantly about half of what producers are sending and is always the same as Consumer Ack numbers. Details in the screenshot https://pasteboard.co/jXD0Ozo78ISe.png

 

Since all applications seems to function normally, I guess my question is is this a normal behaviour? Do we have some kind of topology issue?

 

Our few producers are internal and via CachingConnectionFactory they are connected to one of the nodes at any given time (CHANNEL MODE).

Hundreds of consumers are connected externally and they are connecting via loadbalancer so almost equally distributed across nodes.

 

--

   Best Regards,

 

    Vilius Šumskas

 

[Michal Kuratczyk]

Hi,

There could be a bug in the metric of course but quite likely you have publishers publishing to some routing keys that have no bindings.

You can check the rabbitmq_channel_messages_unroutable_dropped_total metric from /metrics/per-object, to see which channel(s)

publish unroutable messages (note, per-object metrics are available only while the object exists - in this case you need to check

this metric while the channel is still open).

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/DBBPR01MB770590835DF57A0A3269F9FE92FA9%40DBBPR01MB7705.eurprd01.prod.exchangelabs.com.

--

Michał

RabbitMQ team

[Vilius Šumskas]

It’s difficult to track this because we have thousands of active channels, but I see a lot of such channels in the metrics:

 

rabbitmq_channel_messages_unroutable_dropped_total{channel="<0.7493.4>",vhost="/",exchange=""} 2023

rabbitmq_channel_messages_unroutable_dropped_total{channel="<0.10325.4>",vhost="/",exchange=""} 250

 

However, I also found that I can view these statistics in management interface per channel, so I sorted all channel by the Dropped column and this is what I see:

https://pasteboard.co/bFl7deLYJQMh.png

 

These were all running channels at the time when a screenshot was made, others did not produce messages. I then opened management view of top 3 channels in the screenshots and ensured that their queues all have routing keys. They do. Also, if I search for the related exchanges in the metrics I see zeroes for dropped messages:

 

rabbitmq_channel_messages_unroutable_dropped_total{channel="<0.16376.259>",vhost="/",exchange="3046164"} 0

 

Looks like a bug to me (?)

 

--

    Vilius

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0tGAuBgjv8WeOfP4vHr%2BB-JxW7p-%2BhaunUhU55iVn7mZg%40mail.gmail.com.

[david....@gmx.de]

1. In the example you provided, you use the default exchange (""). Are you sure that a queue exists with the name of the routing key?

2. You can set the mandatory flag to check what messages cannot be routed. They will be returned to the publisher and will enable you to debug further.

3. If you are sure about 1. and nothing is returned in 2. can you please provide step by step reproduction steps?

[Vilius Šumskas]

Can you elaborate on item #1? I’m not a developer so sorry if the following question will be stupid. I see that default exchange is used, but I thought that that exchange is bound to all queues and no additional configuration is needed to use it? This is what I see when I sort exchanged by messages-in column:

https://pasteboard.co/RPQ5TBEtGxxt.png

Looks like dropped count is indeed coming from the default exchange, however I have no idea how to debug it further.

 

Since I don’t have access to our product architecture knowledge and most of the developers moved on to other projects, is there another way to see what is sent to the default exchange without setting mandatory flag?

 

--

    Vilius

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/2172e7d0-687c-4260-9deb-ae2f42112c2dn%40googlegroups.com.

[david....@gmx.de]

>  I see that default exchange is used, but I thought that that exchange is bound to all queues and no additional configuration is needed to use it?

Correct, you don't need to create any (additional) bindings. The default exchange looks at the routing key of the incoming message. If there is a queue with the same name as the routing key, the default exchange forwards the message to that queue. However, if there is no queue with the name of the routing key, the message is dropped.

So, in your case publishers publish messages to the default exchange with a routing key name that doesn't match any queue name.

> rabbitmq_channel_messages_unroutable_dropped_total{channel="<0.7493.4>",vhost="/",exchange=""} 2023

> rabbitmq_channel_messages_unroutable_dropped_total{channel="<0.10325.4>",vhost="/",exchange=""} 250

You even know what channels these messages come from: They come from a channel with PID <0.7493.4> and from another channel with PID <0.10325.4>. In the Management UI you can pin down the connection and publisher IP address, so you know what publishers cause these dropped messages.

>  is there another way to see what is sent to the default exchange without setting mandatory flag?

https://www.rabbitmq.com/firehose.html could help you (not suitable for production use cases though).

[Vilius Šumskas]

Thank you for the detailed explanation.

 

I tried to pin point exact producers which are sending to default exchange but it seems that they all send to it. Every time I send something via application it looks like one message goes into the correct queue and another into default.

 

I will look further with firehorse.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/932f8233-05f5-4ba7-8965-f2ba5f9dc79fn%40googlegroups.com.