How to design a message replay service

[YuvalP]

Hi, 

I am trying to design a replay mechanism that will enable users to replay messages from the queues.

The best design I have come up for an exchange that contains multiple queues and multiple consumers is:

1. Create a recorder service that will:

• Create a queue and bind all routing keys to it. 
  
• Consume all messages from the exchange. 
  
• Save all messages to the DB.

     2. Subscriber request for replay.

• Each subscriber creates a new exchange and subscribes a queue to it with same bindings as its regular queue. 
  
• Subscriber sends a rest requests to a web server to start replay with a filter ( startdate, etc). Request contains its replay exchange name.
• Web server pulls data from the DB and publishes it to the specific exchange
• We can add refinements like RequestId and echo it back. 

Have an enlightening picture.

Questions: 

1. Was I making sense here?
2. Am I inventing the wheel? Is there a rabbit inherent solution? plugin?
3. Does creating multiple exchanges considered a good practice? 
   In this solutionan exchange for each queue is created in order to publish the same message.

Another solution: 

Create an additional queue "ReplayQueue" for each queue. set a TTL (let's say a month).

Each time a user requests a replay let him replay from its own ReplayQueue without acking.

This solution is a bit problematic because. 

1. In order to replay last day, consumers will have to fetch all earlier 29 days and filter them out. 
2. This solution scales up - Queues will get larger (unlike db storage that can scale out).

[Jean-Sébastien Pédron]

On 15.04.2015 10:48, YuvalP wrote:
> Hi,

Hi!

> I am trying to design a replay mechanism that will enable users to
> replay messages from the queues.
> The best design I have come up for an exchange that contains multiple
> queues and multiple consumers is:
>

> 1. Create a recorder service that will:
>
> * Create a queue and bind all routing keys to it.
> * Consume all messages from the exchange.
> * Save all messages to the DB.

Maybe you can leverage on the firehose feature:
https://www.rabbitmq.com/firehose.html

> 1. Was I making sense here?

Yes

> 2. Am I inventing the wheel? Is there a rabbit inherent solution? plugin?

I am not aware of any plugins doing this.

> 3. Does creating multiple exchanges considered a good practice?

Yes, there is no problem with multiple exchanges. Internally, they take
a bit of memory, but they are just "routing rules" which are eval'd in
the context of the channel process (the one talking to your client).
There is no dedicated process(es) for them.

--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Michael Klishin]

On 15 April 2015 at 11:48:10, YuvalP (yuval...@gmail.com) wrote:
> I am trying to design a replay mechanism that will enable users
> to replay messages from the queues.

There's a 3rd party tool that does that, please take a look :
http://qdb.io/

You'd need some sort of database for this, as messages in RabbitMQ queues are gone as
soon as they are consumed and acknowledged.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Simrandeep Singh]

Hi MK

Can you comment on the performance impact with qdb?

In other words I am mainly interested in the publish rate with qdb. How hard is this value affected?

I had skimmed through their website but failed to find answers for these questions.

Is it advisable to use qdb in production with medium to high reliability on rabbitmq?

[Michael Klishin]

 On 13 Oct 2015 at 16:36:24, Simrandeep Singh (simrande...@payu.in) wrote:
> Can you comment on the performance impact with qdb?
> In other words I am mainly interested in the publish rate with
> qdb. How hard is this value affected?

Sorry, we don’t have hands-on experience with it on our team.

[Simrandeep Singh]

We currently persist messages to DB but this has very prominent impact on our publish rates(decreased by almost 100 times).

Can you suggest any better way of ensuring complete reliability?  

[Michael Klishin]

On 13 Oct 2015 at 17:03:09, Simrandeep Singh (simrande...@payu.in) wrote:
> We currently persist messages to DB but this has very prominent
> impact on our publish rates(decreased by almost 100 times).
> Can you suggest any better way of ensuring complete reliability?

Complete reliability is a myth. However, "99.9% reliability” is possible and requires having
a write-ahead log of messages you want to publish in every client. This will still reduce your
overall throughput but it can be a substantially smaller penalty because writing to a local
disk is better than to a remote one (via a data store or any other option).

So the only two suggestions I have:

 * Write to local storage
 * Batch writes (and “logical” messages published to RabbitMQ)

if possible