Validate bridge is consuming all messages

[Michael Johnson]

I've configured a bridge from a single node to a cluster of 5 nodes and I'm trying to figure out how to validate that every expected message is arriving on the bridged node.

The bridge is a pull (in) bridge with a wildcard. I'm receiving thousands of messages a second but I'm having trouble validating that I'm receiving every message.

I looked at vmq_bridge_dropped_msgs on the bridged node and it shows 0 in metrics but I'm not sure that that metric would know the queue length from the cluster it's pulling from.

How can I be sure that no messages are being dropped? Is there another metric that shows that? Does vmq_bridge_dropped_msgs metric reflect messages not pulled from another cluster for any reason?

[André Fatton]

Hi Michael,

technically, the "vmq_bridge_dropped_msgs" counter would show whether the bridge has gone over the configured buffer ("max_outgoing_buffered_messages") for its internal queue at some point. If the counter is 0, it hasn't.

Note that on the node where you started the bridge, you will have metrics for the incoming and outgoing messages, like "vmq_bridge_publish_out_0". This might give you more information on the number of messages flowing through.

Also note that a pull bridge creates an MQTT consumer at the remote node. If that bridge consumer is too slow for the sending broker, it (the broker) will drop messages. So you'll want to check for dropped messages logged in the remote sending broker's log files.

If you're overloading the bridge like that, then you have to re-architect/distribute this in a better way.

Hope this helps,

André

[Michael Johnson]

André,

Thank you for the information. I was able to find both metrics.

One follow up question I have is, if I have a cluster of 5 nodes and want to bridge to another single instance, should I configure the bridge from many nodes in the source cluster or just from a single node? I am achieving the throughput I hoped I'm more concerned about resilience to outages now.

Mike

[André Fatton]

Bridges work in a broker-to-broker fashion, and are not cluster aware. But the bridge that connects to a node in your 5-node cluster will make messages flow to that node from all other cluster nodes (by the nature of the created subscriptions). So you still get all the messages from your cluster.

You can increase resilience to outages with a setup where the consuming broker will set up 5 bridges to the 5 remote cluster nodes, but using shared subscriptions so that the messages are loadbalanced. (otherwise you'd get the same message 5 times over the bridges).

The new shared subscriber policy in 1.12.4 (prefer_online_before_local) is probably best suited for a scenario like that.

Best,

André