Shard distribution using rabbitmq-sharding
901 views
Skip to first unread message
d...@itsallbroken.com
Feb 12, 2018, 10:27:12 AM2/12/18
to rabbitmq-users
Hi all,
I've been experimenting with the rabbitmq-sharding plugin and hit a bit of an unexpected outcome:
[root@rabbitmq /]# rabbitmqctl -p A set_policy sharding-test "^sharding-test$" '{"shards-per-node": 1}'Setting policy "sharding-test" for pattern "^sharding-test$" to "{\"shards-per-node\": 1}" with priority "0" ...[root@rabbitmq /]# rabbitmqctl list_queues -p A name pid | grep shardsharding: sharding-test - rab...@queue-01.staging.network - 0 <rab...@queue-03.staging.network.2.19070.1111>sharding: sharding-test - rab...@queue-02.staging.network - 0 <rab...@queue-02.staging.network.1.32201.1090>sharding: sharding-test - rab...@queue-03.staging.network - 0 <rab...@queue-03.staging.network.2.31186.1111>
I was expecting the rab...@queue-01.staging.network - 0 queue to be placed on the rabbit@queue-01 node - is this correct? Even when bumping the "shards-per-node" property to 10 there are no shards allocated to queue-01.
Everything appears fine when checking the cluster_status from each of the hosts:
[root@rabbitmq /]# rabbitmqctl cluster_statusCluster status of node 'rab...@queue-01.staging.network' ...[{nodes,[{disc,['rab...@queue-01.staging.network','rab...@queue-02.staging.network','rab...@queue-03.staging.network']}]},{running_nodes,['rab...@queue-03.staging.network','rab...@queue-02.staging.network','rab...@queue-01.staging.network']},{cluster_name,<<"rabbit@rabbitmq">>},{partitions,[]},{alarms,[{'rab...@queue-03.staging.network',[]},{'rab...@queue-02.staging.network',[]},{'rab...@queue-01.staging.network',[]}]}]
The exchange was declared as x-modulus-hash (no other configuration) and has correctly picked up the policy (the queues did not previously exist).
Versions:
[root@rabbitmq /]# rabbitmqctl statusStatus of node 'rab...@queue-01.staging.network' ...[{pid,59563},{running_applications,[{rabbitmq_sharding,"RabbitMQ Sharding Plugin","0.1.0"},{rabbitmq_management,"RabbitMQ Management Console","3.6.6"},{rabbitmq_management_agent,"RabbitMQ Management Agent","3.6.6"},{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.6.6"},{rabbit,"RabbitMQ","3.6.6"},{amqp_client,"RabbitMQ AMQP Client","3.6.6"},{webmachine,"webmachine","1.10.3"},{mochiweb,"MochiMedia Web Server","2.13.1"},{rabbit_common,[],"3.6.6"},{xmerl,"XML parser","1.3.14"},{mnesia,"MNESIA CXC 138 12","4.14.3"},{ssl,"Erlang/OTP SSL application","8.1.3"},{public_key,"Public key infrastructure","1.4"},{crypto,"CRYPTO","3.7.4"},{os_mon,"CPO CXC 138 46","2.4.2"},{compiler,"ERTS CXC 138 10","7.0.4"},{inets,"INETS CXC 138 49","6.3.9"},{ranch,"Socket acceptor pool for TCP protocols.","1.2.1"},{asn1,"The Erlang ASN1 compiler version 4.0.4","4.0.4"},{syntax_tools,"Syntax tools","2.1.1"},{sasl,"SASL CXC 138 11","3.0.3"},{stdlib,"ERTS CXC 138 10","3.3"},{kernel,"ERTS CXC 138 10","5.2"}]},{os,{unix,freebsd}},{erlang_version,"Erlang/OTP 19 [erts-8.3.5] [source] [64-bit] [smp:40:40] [async-threads:640] [hipe] [kernel-poll:true] [dtrace]\n"},
Thanks in advance,
Dom
Michael Klishin
Feb 12, 2018, 11:07:53 AM2/12/18
to rabbitm...@googlegroups.com
You have 3 shards in your output, one on rab...@queue-03.staging.network.1
and two on rab...@queue-03.staging.network.2.
The README isnt' clear enough on this but the setting
means "up to so many shards per sharded queue per node" IIRC.
and two on rab...@queue-03.staging.network.2.
The README isnt' clear enough on this but the setting
means "up to so many shards per sharded queue per node" IIRC.
On Mon, Feb 12, 2018 at 3:27 PM, dom via rabbitmq-users <rabbitm...@googlegroups.com> wrote:
Hi all,I've been experimenting with the rabbitmq-sharding plugin and hit a bit of an unexpected outcome:[root@rabbitmq /]# rabbitmqctl -p A set_policy sharding-test "^sharding-test$" '{"shards-per-node": 1}'Setting policy "sharding-test" for pattern "^sharding-test$" to "{\"shards-per-node\": 1}" with priority "0" ...[root@rabbitmq /]# rabbitmqctl list_queues -p A name pid | grep shardsharding: sharding-test - rab...@queue-01.staging.network - 0 <rab...@queue-03.staging.network.2.19070.1111>sharding: sharding-test - rab...@queue-02.staging.network - 0 <rab...@queue-02.staging.network.1.32201.1090>sharding: sharding-test - rab...@queue-03.staging.network - 0 <rab...@queue-03.staging.network.2.31186.1111>I was expecting the rab...@queue-01.staging.network - 0 queue to be placed on the rabbit@queue-01 node - is this correct? Even when bumping the "shards-per-node" property to 10 there are no shards allocated to queue-01.Everything appears fine when checking the cluster_status from each of the hosts:
[root@rabbitmq /]# rabbitmqctl cluster_statusCluster status of node 'rab...@queue-01.staging.network' ...
[{nodes,[{disc,['rabbit@queue-01.staging.network','rab...@queue-02.staging.network','rab...@queue-03.staging.network']}]},{running_nodes,['rabbit@queue-03.staging.network',
'rab...@queue-02.staging.network','rab...@queue-01.staging.network']},{cluster_name,<<"rabbit@rabbitmq">>},{partitions,[]},
{alarms,[{'rabbit@queue-03.staging.network',[]},
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
Dom Dwyer
Feb 12, 2018, 11:25:19 AM2/12/18
to rabbitm...@googlegroups.com
Hi Michael - thanks for the speedy reply.
I tried setting the policy to {"shards-per-node": 200} in a fresh vhost, but all queues with queue-01 in the name name were placed on queue-02 this time.
I'm assuming I can explicitly get the "1 shard per node, over N nodes" behaviour by creating the queues manually with x-queue-master-locator set and using the consistent hash exchange to distribute across the queues?
Thanks again,
Dom
On Mon, Feb 12, 2018 at 4:07 PM, Michael Klishin <mkli...@pivotal.io> wrote:
You have 3 shards in your output, one on rab...@queue-03.staging.network.1
and two on rab...@queue-03.staging.network.2.
The README isnt' clear enough on this but the setting
means "up to so many shards per sharded queue per node" IIRC.
On Mon, Feb 12, 2018 at 3:27 PM, dom via rabbitmq-users <rabbitmq-users@googlegroups.com> wrote:
Hi all,I've been experimenting with the rabbitmq-sharding plugin and hit a bit of an unexpected outcome:[root@rabbitmq /]# rabbitmqctl -p A set_policy sharding-test "^sharding-test$" '{"shards-per-node": 1}'Setting policy "sharding-test" for pattern "^sharding-test$" to "{\"shards-per-node\": 1}" with priority "0" ...[root@rabbitmq /]# rabbitmqctl list_queues -p A name pid | grep shardsharding: sharding-test - rab...@queue-01.staging.network - 0 <rab...@queue-03.staging.network.2.19070.1111>sharding: sharding-test - rab...@queue-02.staging.network - 0 <rab...@queue-02.staging.network.1.32201.1090>sharding: sharding-test - rab...@queue-03.staging.network - 0 <rab...@queue-03.staging.network.2.31186.1111>I was expecting the rab...@queue-01.staging.network - 0 queue to be placed on the rabbit@queue-01 node - is this correct? Even when bumping the "shards-per-node" property to 10 there are no shards allocated to queue-01.Everything appears fine when checking the cluster_status from each of the hosts:
[root@rabbitmq /]# rabbitmqctl cluster_statusCluster status of node 'rab...@queue-01.staging.network' ...[{nodes,[{disc,['rabbit@queue-01.staging.network','rab...@queue-02.staging.network','rab...@queue-03.staging.network']}]},{running_nodes,['rabbit@queue-03.staging.network','rab...@queue-02.staging.network','rab...@queue-01.staging.network']},{cluster_name,<<"rabbit@rabbitmq">>},{partitions,[]},
{alarms,[{'rab...@queue-03.staging.network',[]},
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--MKStaff Software Engineer, Pivotal/RabbitMQ
--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/6krHUpVUm0U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
Michael Klishin
Feb 12, 2018, 12:34:47 PM2/12/18
to rabbitm...@googlegroups.com
Queues have shards, and the plugin by design wants to have at least 1 per node.
So it is perfectly possible to have 1 shard *of a particular queue* per node but not 1 shard
globally.
If you want the latter I'd reevaluate why you want to use rabbitmq-sharding at all.
Perhaps one of the options documented in http://www.rabbitmq.com/ha.html#master-migration-data-locality
would be enough.
Dom Dwyer
Feb 13, 2018, 8:05:02 AM2/13/18
to rabbitm...@googlegroups.com
Hi Michael
So it is perfectly possible to have 1 shard *of a particular queue* per node
This is the behaviour I am after - I'm attempting to spread the load over multiple nodes (3 in this case).
I have 1 queue (lets call it incoming) and I bind it to an exchange (incoming-exchange) that is configured as a x-modulus-hash exchange. I then apply the sharding policy with {"shards-per-node": 2} to the exchange. After reading the docs, I was expecting:
- Queue incoming.1 => rabbit@queue-01
- Queue incoming.2 => rabbit@queue-02
- Queue incoming.3 => rabbit@queue-03
- Queue incoming.4 => rabbit@queue-01
- Queue incoming.5 => rabbit@queue-02
- Queue incoming.6 => rabbit@queue-03
Thanks for the help!
Dom
On Mon, Feb 12, 2018 at 5:34 PM, Michael Klishin <mkli...@pivotal.io> wrote:
Queues have shards, and the plugin by design wants to have at least 1 per node.So it is perfectly possible to have 1 shard *of a particular queue* per node but not 1 shardglobally.If you want the latter I'd reevaluate why you want to use rabbitmq-sharding at all.Perhaps one of the options documented in http://www.rabbitmq.com/ha.html#master-migration-data-localitywould be enough.
Reply all
Reply to author
Forward
0 new messages