RabbitMQ Sharding Plugin/Consistent Hash Exchange dynamic resizing of number of partitions

[sumit jain]

Hi,

We are thinking of using sharding plugin to implement ordering in our exchange for same keys. This will provide us concurrency over different shards and ordering within a single shard.

But can we change the number of partitions afterwards, to scale consumer concurrency dynamically? The github page doesn't mention this nor does RabbitMQ Consistent Hash Exchange.

Is this behavior possible in rabbitmq?

 

[sumit jain]

To be sure, I am asking if the number of partitions be changed dynamically, and not the number of consumers. The latter is responsibility of the application, of course.

Also, if it can be done, is there some doc which explains the broker behavior during re-sharding?

[Michael Klishin]

On 28 Jul 2015 at 14:05:51, sumit jain (sumit...@gmail.com) wrote:
> To be sure, I am asking if the number of partitions be changed
> dynamically, and not the number of consumers. The latter is responsibility
> of the application, of course.
>
> Also, if it can be done, is there some doc which explains the broker
> behavior during re-sharding?

Changing the number of shards by tweaking policy should work fine [1].
Existing data won’t be moved, as in [2].


1. https://github.com/rabbitmq/rabbitmq-sharding/blob/master/src/rabbit_sharding_exchange_decorator.erl#L52-56
2. https://github.com/rabbitmq/rabbitmq-sharding#auto-scaling
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[sumit jain]

Thanks Michael. 

Few queries on this:

1. Is the policy change detected instantly? 

2. What is the broker behavior when the policy is changed. Existing data will not be moved, but what happens to the queues? Say, if count is increased, are the old ones deleted or reused? 

Does publishing stop until transition is complete? Do consumers need to re-register? In short, what is the observable effect on application producers/consumers during resharding?

Regards,

Sumit

[Michael Klishin]

On 28 July 2015 at 15:37:55, sumit jain (sumit...@gmail.com) wrote:
> 1. Is the policy change detected instantly?

As soon as policy changes reach the node, so pretty much instantly.

> 2. What is the broker behavior when the policy is changed. Existing
> data will not be moved, but what happens to the queues? Say, if
> count is increased, are the old ones deleted or reused?
> Does publishing stop until transition is complete? Do consumers
> need to re-register? In short, what is the observable effect
> on application producers/consumers during resharding?

As the README explains, sharded queues are logical and backed by N "physical"
queues (that can have on disk storage, etc).

When you increase count, the plugin will declare more "physical" queues and bind
them accordingly. Existing queues or consumers won't be affected except that there
will be a moment when bindings are updated in bulk (so in theory some messages
may end up being unrouted — publishing as mandatory will make sure those are returned
to the published using basic.return).

Re-adding consumers is not technically necessary but if you want them to consume from
all the shards, it's a good idea. 

[sumit jain]

Tried the shard plugin, its looks good, with automatic queue creation and fair  message distribution. 

But I am facing problem with consumer allocation, the queues are not getting evenly divided among them. e.g. If have 100 partitions, and I use this spring amqp configuration

        <rabbit:listener-container id="queueListenerContainer" connection-factory="consumerConnectionFactory" 

acknowledge="auto" concurrency="100">

<rabbit:listener ref="workerListener" method="onMessage" queue-names="ShardedExchange" admin="rabbitWorkAdmin" />

</rabbit:listener-container>

The consuming threads are randomly allocated to queues, with some queues getting 10, and few not getting even 1. Once running with 1000 partitions/consumers, one queue ended with 500 consumers !!

[Michael Klishin]

On 29 July 2015 at 14:06:08, sumit jain (sumit...@gmail.com) wrote:
> But I am facing problem with consumer allocation, the queues
> are not getting evenly divided among them. e.g. If have 100 partitions,
> and I use this spring amqp configuration
>

> connection-factory="consumerConnectionFactory"
> acknowledge="auto" concurrency="100">

> > queue-names="ShardedExchange" admin="rabbitWorkAdmin"
> />
>
>
>

> The consuming threads are randomly allocated to queues, with
> some queues getting 10, and few not getting even 1. Once running
> with 1000 partitions/consumers, one queue ended with 500 consumers
> !!

This is likely the sharding plugin limitation and has nothing to do with spring-amqp.
Feel free to file an issue on GitHub, demonstrating the distribution you have. 

[sumit jain]

Yes, I agree it is plugin's problem, I didn't intend to claim otherwise. I shall create a bug on their repo, right away.