RabbitMQ message deduplication plugin

[noxdafox]

Hi everybody,

I often found myself in need of de-duplicating RabbitMQ messages and I usually ended up relying on producer/consumer implementations based on third party data storages.

To learn a bit more about RabbitMQ and Elixir, I decided to build a simple message de-duplication plugin.

It is pretty simple and I didn't manage to test it at scale yet but so far it looks promising.

I shared it on my git account:
https://github.com/noxdafox/rabbitmq-message-deduplication

Contributions and comments are more than welcome :)

Matteo.

[Michael Klishin]

Thank you!

It's interesting that you decided to deduplicate at the exchange level but I can see how

this is much easier than any other alternative :)

It would be nice if you could

1) Document what RabbitMQ versions are supported

2) Produce and distribute binary builds via GitHub releases

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[noxdafox]

On Thursday, 1 March 2018 20:15:20 UTC+2, Michael Klishin wrote:

Thank you!

It's interesting that you decided to deduplicate at the exchange level but I can see how

this is much easier than any other alternative :)

As this was my first approach to RabbitMQ plugin development, I chose the easiest alternative. Plugin development process is not well documented and if you're new to Erlang it gets overwhelming at first.
There were quite few exchange-based examples on the Web so I could at least follow some reference.

Moreover, what I needed was a way to filter duplicates as soon as possible. With this approach I can build several routes without the need of worrying about duplicates on each route.

Later on, I will add queue-level deduplication as well.
 

It would be nice if you could

1) Document what RabbitMQ versions are supported

2) Produce and distribute binary builds via GitHub releases

Do you have any reference examples on how RabbitMQ community distributes binaries via GitHub?
 

On Thu, Mar 1, 2018 at 8:52 PM, noxdafox <noxd...@gmail.com> wrote:

Hi everybody,

I often found myself in need of de-duplicating RabbitMQ messages and I usually ended up relying on producer/consumer implementations based on third party data storages.

To learn a bit more about RabbitMQ and Elixir, I decided to build a simple message de-duplication plugin.

It is pretty simple and I didn't manage to test it at scale yet but so far it looks promising.

I shared it on my git account:
https://github.com/noxdafox/rabbitmq-message-deduplication

Contributions and comments are more than welcome :)

Matteo.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Just produce a release (an .ez), tag it and create a GitHub release with it.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[noxdafox]

Done as requested.

Built binary release and attached to git tag.

Added installation instructions with supported RabbitMQ versions.

[noxdafox]

UPDATE: I added support for queue de duplication

Since version 0.2.0 of the plugin, it is possible to de duplicate messages both at the exchange and at the queue sides.

As a general guideline: queue level de duplication is to be used to ensure no duplicate messages are stored within a given queue at the same time.
Exchange level de duplication instead, allows to de duplicate messages over time windows. In other words allows to prevent the same message to be routed more than once in a given period of time.

[Michael Klishin]

Interesting. How does it work for queues, implementation-wise?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

And thank you for your work on this!

On Thu, May 10, 2018 at 11:32 AM, Michael Klishin <mkli...@pivotal.io> wrote:

Interesting. How does it work for queues, implementation-wise?

On Thu, May 10, 2018 at 11:07 AM, noxdafox <noxd...@gmail.com> wrote:

UPDATE: I added support for queue de duplication

Since version 0.2.0 of the plugin, it is possible to de duplicate messages both at the exchange and at the queue sides.

As a general guideline: queue level de duplication is to be used to ensure no duplicate messages are stored within a given queue at the same time.
Exchange level de duplication instead, allows to de duplicate messages over time windows. In other words allows to prevent the same message to be routed more than once in a given period of time.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[noxdafox]

np, I hope it will be useful to somebody.

Both queues and exchanges rely on dedicated caches to check if a message is a duplicate. Every time a message carrying a `x-message-deduplication` header is published, its value is looked up within the exchange/queue cache. A cache hit results in the message being dropped, otherwise its deduplication header is added to the cache and the message continue its route.

The use of caches simplifies significantly the implementation. For queues in particular, this strategy results in a very thin layer which can be added on top of any queue implementation (variable queue, priority queue etc..). I implemented the `rabbit_backing_queue` behaviour and most of the logic simply delegates the operations to the underlying backing queue.

The cache itself is built on top of Mnesia.
 

[Michael Klishin]

I see, so it is a `rabbit_backing_queue` implementation that passes through most operations. Cool!

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[noxdafox]

Yep! The goal was to be the least intrusive possible.

The cache makes it simple and fast by trading off some memory or disk space.

[Akshaya]

Hi,

Could you please provide one sample command using rabbitmqadmin while declaring queue or exchange?How to specifiy this as header -x-message-deduplication?

Also it is possible to edit the existing messages and queues?

- Akshaya

[Michael Klishin]

Should be no different from any other header, which has certainly been discussed before on this list and teh Internet.

The right thing to do in the vast majority of cases is to use a policy [1].

1. http://www.rabbitmq.com/parameters.html#policies

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Akshaya]

Hi,

Thank you. I am new to rabbitMQ. Could you please let me know is this correct?

        properties = pika.BasicProperties(headers = {'x-deduplication-header': 'true'})
        channel.basic_publish(exchange='test_exch',
                              routing_key='',
                              properties=properties,
                              body=topublish)

But still could find the duplicate messages in test_queue.

How to fix this?

- Akshaya

On Friday, July 6, 2018 at 2:55:47 PM UTC+5:30, Michael Klishin wrote:

Should be no different from any other header, which has certainly been discussed before on this list and teh Internet.

The right thing to do in the vast majority of cases is to use a policy [1].

1. http://www.rabbitmq.com/parameters.html#policies

On Fri, Jul 6, 2018 at 12:03 PM, Akshaya <nach...@gmail.com> wrote:

Hi,

Could you please provide one sample command using rabbitmqadmin while declaring queue or exchange?How to specifiy this as header -x-message-deduplication?

Also it is possible to edit the existing messages and queues?

- Akshaya

On Thursday, March 1, 2018 at 11:22:59 PM UTC+5:30, noxdafox wrote:

Hi everybody,

I often found myself in need of de-duplicating RabbitMQ messages and I usually ended up relying on producer/consumer implementations based on third party data storages.

To learn a bit more about RabbitMQ and Elixir, I decided to build a simple message de-duplication plugin.

It is pretty simple and I didn't manage to test it at scale yet but so far it looks promising.

I shared it on my git account:
https://github.com/noxdafox/rabbitmq-message-deduplication

Contributions and comments are more than welcome :)

Matteo.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[noxdafox]

You can check this gist for a reference example on how to deduplicate messages at the exchange.
https://gist.github.com/noxdafox/ad1fb4c3769e06a888c3a542fc08c544

The `x-deduplication-header` must contain a value which uniquely identifies the message itself. The plugin will drop all other messages with the same header.
If the body of the message is what you want to ensure to be unique, then you can use its hash (MD5, SHA1, ...) as a `x-deduplication-header` value.