x-consistent-hash exchange: All messages going to single queue

[Noonan, Tom]

I'm attempting to shard an exchange to multiple queues using the x-consistent-hash exchange type.  (I'm also evaluating the sharding plugin, but at the moment x-consistent-hash seems like a better fit for my use case.)  I've bound 4 queues to my test exchange, each with routing key "10". All messages are coming into the exchange with the same routing key, it's not easy to make the key unique across producers in my setup.  Since the same routing key will always go to the same queue per the docs, I've set the hash-property argument on the exchange to hash off a message property field.  However, I'm noticing all the messages tend to only go to one of the queues.  With {"hash-property": "timestamp"} all the messages went to queue 0. As an experiment I changed to {"hash-property": "message_id"} and now all messages are going to queue 1.

I set up the exchange with the following commands (obviously erasing the exchange and queues between these commands):

rabbitmqadmin --config /etc/rabbitmqadmin.conf -V /es declare exchange 'name=es' type=x-consistent-hash auto_delete=false internal=false durable=true arguments='{"hash-property": "timestamp"}'

rabbitmqadmin --config /etc/rabbitmqadmin.conf -V /es declare exchange 'name=es' type=x-consistent-hash auto_delete=false internal=false durable=true arguments='{"hash-property": "message_id"}'

Have I missed a required configuration option?

Is there a better option to round-robin messages with identical routing keys between queues with this exchange type?

Thanks!

Tom Noonan II

DevOps Engineer

TNo...@shoretel.com

720.966.8896

www.shoretel.com

---

CONFIDENTIALITY NOTICE: The information contained in this e-mail message and any attachments is for the sole use of the intended recipients and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient and have received this communication in error, please contact the sender by reply e-mail and destroy all copies of the original message. Thank you.

  ­­  

[Michael Klishin]

Was the timestamp property explicitly set by your clients? It's not automatically filled by the broker

(there is a plugin that does it, though).

You say that routing keys are naturally non-evenly distributed in your system. Why do you want to use

consistent hashing routing in such case?

Also, rabbitmq-sharding is orthogonal to the consistent hashing exchange and is most commonly used with that

specific exchange type.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Noonan, Tom]

> Was the timestamp property explicitly set by your clients? It's not automatically filled by the broker (there is a plugin that does it, though).

I'll have to check, but then probably not.  However, the message_id should still be unique, yes?

> You say that routing keys are naturally non-evenly distributed in your system. Why do you want to use consistent hashing routing in such case?

> Also, rabbitmq-sharding is orthogonal to the consistent hashing exchange and is most commonly used with that specific exchange type.

I'm trying to spread out load from LogStash which is overloading a single HA queue.  The logstash producer is fairly simple, and while I could set a unique key per host I'd prefer not to as single producer can peg out the queue so I'd like to break the messages up on something else.  Setting a truly random key isn't an option without patching the logstash client.  As for the sharding plugin, the answer is in the other thread I have going today.  Logstash assumes it needs to create the queue and per your last message in that thread it prevents the sharding plugin from working correctly.  This is why I'm looking into the x-consistent-hash exchange, as (If I can get the hashing working) I can allow multiple pools of Logstash workers to create queues as they expect while still round-robining messages between them.  So my end goal isn't to route based off unique hashes itself, it's to get messages coming in to a single exchange with identical routing keys to go to multiple queues without duplication so I can distribute load.  This course of action is based on Simon's comment on this page: https://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/

Tom Noonan II

DevOps Engineer

TNo...@shoretel.com

720.966.8896

www.shoretel.com

[Michael Klishin]

Message ID isn't set by RabbitMQ either.

[Noonan, Tom]

Okay, well then the behavior I'm seeing makes sense.  Thanks for the help, I know know where I need to look.