Pub/Sub reliability - no message lost

[Ori D]

Hi,

I'd like to implement a pub/sub mechanism for updating an in-memory cache, and I cannot afford to lose any update message.

I have several subscriber instances (microservice, Java SpringBoot), each creates its own queue on startup and binds the queue to a "fanout" exchange, Thus, every update message I publish to the exchange is handled by all of the subscribers.

Now, how can I guarantee reliability for the subscribers to receive all published messages once message is successfully published to the exchange? 

(handling of failed published messages is already performed and is out of the scope of this question)

Can I consider RabbitMQ bullet-proof or that I must build some compensation mechanism on subscribers to validate against my "source of truth" that I haven't missed any update?

Thanks,

Ori.

[aviv salem]

Hey ori...

There's a specific configuration if you want to absolutely not lose any messages. 

1. Subscribers must consume with noack=false (and ack messages upon completion of processing)

2. Messages should be published with persistent mode, and mandatory flag. (and make sure that you are waiting for publisher confirms) 

3. Queues should be declared as durable 

4. HA is recommended. If you use one, there's a whole set of policy settings that applies (let me know, and I'll share, cause it's a bit long.)

5. Even if you don't have HA but have a cluster, you should set the partition policy to pause_minority. 

I recommend the following link for a read in the subject.... 

https://jack-vanlightly.com/blog/2018/9/10/how-to-lose-messages-on-a-rabbitmq-cluster

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/73a29751-8ee7-4f73-8b93-05fe1f7491ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ori D]

Hi Aviv,

Thank you very much for the prompt response (and sorry for my late reply - summer vacation...).

1. I'd like to here more about HA and the policy settings you mentioned. 

2. In my application, when the subscriber is starting, it is ok for it to start with zero knowledge and only get new updates from the queue. But once it is started, it must not miss any update.

    Therefore, I think I should declare my subscriber queue with a random name , exclusive (true) and autodelete (true). Is that correct?

3. We do have a cluster of 3 nodes and queues are replicated between nodes. Is that what you meant by having HA ?

  

Regards,

Ori.

On Sunday, July 14, 2019 at 5:17:36 PM UTC+3, aviv salem wrote:

Hey ori...

There's a specific configuration if you want to absolutely not lose any messages. 

1. Subscribers must consume with noack=false (and ack messages upon completion of processing)

2. Messages should be published with persistent mode, and mandatory flag. (and make sure that you are waiting for publisher confirms) 

3. Queues should be declared as durable 

4. HA is recommended. If you use one, there's a whole set of policy settings that applies (let me know, and I'll share, cause it's a bit long.)

5. Even if you don't have HA but have a cluster, you should set the partition policy to pause_minority. 

I recommend the following link for a read in the subject.... 

https://jack-vanlightly.com/blog/2018/9/10/how-to-lose-messages-on-a-rabbitmq-cluster

On Sun, Jul 14, 2019, 15:03 Ori D <ori...@gmail.com> wrote:

Hi,

I'd like to implement a pub/sub mechanism for updating an in-memory cache, and I cannot afford to lose any update message.

I have several subscriber instances (microservice, Java SpringBoot), each creates its own queue on startup and binds the queue to a "fanout" exchange, Thus, every update message I publish to the exchange is handled by all of the subscribers.

Now, how can I guarantee reliability for the subscribers to receive all published messages once message is successfully published to the exchange? 

(handling of failed published messages is already performed and is out of the scope of this question)

Can I consider RabbitMQ bullet-proof or that I must build some compensation mechanism on subscribers to validate against my "source of truth" that I haven't missed any update?

Thanks,

Ori.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.