ConsistentHash exchange changes distribution after server restart

[Marcin Stefaniuk]

I have following scenario - consistent hash exchange is bound to X queues with binding key "1" (the same weight to each of them). Messages are hashed basing on routing key but after RabbitMQ server restart the same routing key A is landing in different queue than before restart. After several restarts there is complete mess in the queues (without any binding change).

This behavior is making this exchange useless because each restart causes consumers to have different distribution and thus proper order of the messages cannot be preserved.

Is this behavior by design or it is a bug?

Kind regards
Marcin

[Michael Klishin]

Most likely neither (or both): a scenario not considered by the authors.

It would be a lot easier to have a constructive conversation if a way to reproduce

was provided and fewer bold claims and blanket statements were used.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

A quick look suggests that the consistent hashing exchange plugin relies on 3 values

when setting up hash buckets:

 * binding source (an exchange)

 * binding destination (typically a queue)

 * routing key

all of which should be stable as long as both source and destination are durable:

https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/blob/stable/src/rabbit_exchange_type_consistent_hash.erl#L131

It further uses a stable hashing function (or the docs don't suggest otherwise) with a constant range:

http://erlang.org/doc/man/erlang.html#phash2-1

The only volatile values I see are message timestamp, ID and correlation ID which are used if present:

https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/blob/stable/src/rabbit_exchange_type_consistent_hash.erl#L196

On Tue, Jun 27, 2017 at 7:26 PM, Michael Klishin <mkli...@pivotal.io> wrote:

Most likely neither (or both): a scenario not considered by the authors.

It would be a lot easier to have a constructive conversation if a way to reproduce

was provided and fewer bold claims and blanket statements were used.

On Tue, Jun 27, 2017 at 4:29 PM, Marcin Stefaniuk <mar...@stefaniuk.info> wrote:

I have following scenario - consistent hash exchange is bound to X queues with binding key "1" (the same weight to each of them). Messages are hashed basing on routing key but after RabbitMQ server restart the same routing key A is landing in different queue than before restart. After several restarts there is complete mess in the queues (without any binding change).

This behavior is making this exchange useless because each restart causes consumers to have different distribution and thus proper order of the messages cannot be preserved.

Is this behavior by design or it is a bug?

Kind regards
Marcin

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Marcin Stefaniuk]

Michael,

Sorry for big words used. My steps to reproduce are simple:

* I’m using RabbitMQ 3.6.6 but we noticed it on other versions from line 3.6

* I’m creating 50 queues (c1 to c50) - all durable

* I’m creating one durable consistent hash exchange (without any additional parameter so routing key should be used)

* I’m creating binding between exchange and all queues in determined order (from c1 to c50) with key „1"

* I’m sending 10 messages with only one routing key (e.g. 0 or 1 but real solution uses 20 digit numbers)

* Between several runs all messages land in single queue (expected behavior - only one routing key used)

* After server restart no setup is run (old exchange, queues and bindings)

* Complete set of messages from run after restart lands in different queue (c19 instead of c25) and messages of following runs are routed using new way

I assume that there can be different process when binding is applied to CH exchange and quite different when it is recovered from Mnesia after restart. Order of application may impact on assigned parts from the hashing space.

I can provide setup and publishing script written in Node.js (but usually we are configuring this kind of binding using management plugin directly).

Kind regards

Marcin

Wiadomość napisana przez Michael Klishin <mkli...@pivotal.io> w dniu 27.06.2017, o godz. 18:41:

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/qHHptwZoxj8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

These steps should be enough, we'll try to reproduce.

Kind regards

Marcin

On Tue, Jun 27, 2017 at 4:29 PM, Marcin Stefaniuk <marcin@stefaniuk.info> wrote:

I have following scenario - consistent hash exchange is bound to X queues with binding key "1" (the same weight to each of them). Messages are hashed basing on routing key but after RabbitMQ server restart the same routing key A is landing in different queue than before restart. After several restarts there is complete mess in the queues (without any binding change).

This behavior is making this exchange useless because each restart causes consumers to have different distribution and thus proper order of the messages cannot be preserved.

Is this behavior by design or it is a bug?

Kind regards
Marcin

-- 
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 

MK

Staff Software Engineer, Pivotal/RabbitMQ

-- 

MK

Staff Software Engineer, Pivotal/RabbitMQ

-- 

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/qHHptwZoxj8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Arnaud Cogoluègnes]

I managed to reproduce and created a GitHub issue: https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/issues/32

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Fixed in https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/pull/33.

[Marcin Stefaniuk]

May I ask for beta build with this change to enforce feedback loop?

Kind regards

Marcin Stefaniuk

Wiadomość napisana przez Michael Klishin <mkli...@pivotal.io> w dniu 05.07.2017, o godz. 21:49:

Fixed in https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/pull/33.

On Friday, June 30, 2017 at 11:20:38 AM UTC+3, Arnaud Cogoluègnes wrote:

I managed to reproduce and created a GitHub issue: https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/issues/32

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/qHHptwZoxj8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

I *think* it is included in [1][2] but just in case attaching a build of the plugin specifically. Replace the .ez in your local installation and give it a try.

1. https://bintray.com/rabbitmq/debian-dev/rabbitmq-server/3.6.11~alpha.11-1

2. https://bintray.com/rabbitmq/rpm-dev/rabbitmq-server/3.6.11~alpha.11-1

On Wed, Jul 5, 2017 at 11:21 PM, Marcin Stefaniuk <mar...@stefaniuk.info> wrote:

May I ask for beta build with this change to enforce feedback loop?

Kind regards

Marcin Stefaniuk

Wiadomość napisana przez Michael Klishin <mkli...@pivotal.io> w dniu 05.07.2017, o godz. 21:49:

Fixed in https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/pull/33.

On Friday, June 30, 2017 at 11:20:38 AM UTC+3, Arnaud Cogoluègnes wrote:

I managed to reproduce and created a GitHub issue: https://github.com/rabbitmq/rabbitmq-consistent-hash-exchange/issues/32

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/qHHptwZoxj8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Marcin Stefaniuk]

I had originally overlooked your attachment. I've just checked this plugin version and it works perfectly. When can we expect final version of RabbitMQ 3.6.11?

Kind regards
Marcin

[Michael Klishin]

In the next few weeks.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcin Stefaniuk]

Maybe it is possible to provide consistent hash plugin apart from RabbitMQ server? It can makes sense because I presume this fixed version of plugin can be applied to all 3.6.8 and above versions of a server.

Kind regards
Marcin

[Michael Klishin]

Hi Marcin,

We can post an .ez of the plugin off the tip of stable here but we won't be distributing (or versioning) plugins

that ship with RabbitMQ independently.

Let me know if that would be helpful to you.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcin Stefaniuk]

Michael,

You've already posted one *.ez file in this thread so it already satisfy our needs (workaround). I was not aware that you treat differently plugins shipped from the others. I thought that you can provide CH plugin similarly as http auth plugin is provided (version 3.6.8 via bintray). In addition autonomous repository suggested similar approach. But I understand you position now.

Thanks

Marcin

2017-07-18 12:54 GMT+02:00 Michael Klishin <mkli...@pivotal.io>:

Hi Marcin,

We can post an .ez of the plugin off the tip of stable here but we won't be distributing (or versioning) plugins

that ship with RabbitMQ independently.

Let me know if that would be helpful to you.

On Tue, Jul 18, 2017 at 11:12 AM, Marcin Stefaniuk <mar...@stefaniuk.info> wrote:

Maybe it is possible to provide consistent hash plugin apart from RabbitMQ server? It can makes sense because I presume this fixed version of plugin can be applied to all 3.6.8 and above versions of a server.

Kind regards
Marcin

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

--

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/qHHptwZoxj8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcin Stefaniuk
mar...@stefaniuk.info