Consumer not picking messages from sharded queues by passing exchange name as queue name in RabbitMQ

74 views

Skip to first unread message

Ashish Tiwari

unread,

Aug 16, 2019, 3:23:38 AM8/16/19

to rabbitmq-users

I am facing problem with sharding in RabbitMQ at consumer side. I am using latest stable RabbitMQ .Net client from nuget. I have gone through with various documents and conference videos and I get that I have to pass sharding Exchange Name as a queue name to consumer. By this way, consumer is connected but fetching data from first sharded queue only. If I have 5 shards, it's not fetching from other than first.

If I try with the exact queue name that is randomly created by RabbitMQ, consumer is fetching messages successfully. But as RabbitMQ says, Sharding is done at exchange level... and... If you have a shard called images, then you can directly consume from a queue called images. So I am passing shard exchange name to consumer but it is consuming first queue data only.

Enable sharding -
rabbitmq-plugins enable rabbitmq_sharding
Shard creation using RabbitMQ management UI -
1. Created exchange of type x-modulus-hash named as HelloShard
2. Enabled sharding by creating a policy for this exchange by pattern: ^HelloShard$, parameters: shards-per-node = 5, routing-key = 12345

I expect from consumer to fetch messages from all sharded queues.

Michael Klishin

unread,

Oct 3, 2019, 5:53:08 AM10/3/19

to rabbitmq-users

A single consumer will not consume from all shards. It would not achieve the goal this plugin tries to achieve, which is

parallel processing on a single logical queue.

When you add a consumer to a sharded queue, the shard with fewest consumers is picked as a way of balancing the load.

So it is expected that you will have more consumers and each will receive a subset of deliveries/stream of publishes.

It makes zero sense to use that plugin with a single consumer.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/a7f5a0e9-cf2b-4017-8cb1-57398cd37251%40googlegroups.com.