Duplicate messages when using HA queues

[Alexey Zimarev]

We configured RabbitMq cluster with three servers and now we see that services that perfectly work in single RabbitMq server environment start to get lots of duplicate messages when working in the HA environment where the default queue policy is HA.

I understand this might be not exactly the MT topic but may be there is a common gotcha we do not know about, although I cannot find anything about this particular issue...

If this is normal, is there any way to easily dedup messages with MT without changing consumers to be idempotant?

[Chris Patterson]

You shouldn't get duplicates at all if your queues are setup correctly and your services are reading from the same queue -- regardless of which server the node is connected.

Are you sure the queues are setup in HA, and were created after the HA policy was applied to the broker?

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstransit-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/08964565-a871-4f11-b646-feafd9c6694b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexey Zimarev]

Yes, I am sure, I had set up a clean cluster and applied the policy, I can see all queues to have the HA+2 sign with all three servers involved. We are deploying to dev and this one is test, in dev we have single server environment and everything works just fine, the same code runs on HA cluster gives many duplicates. Not sure what the reason is. We have one receiving endpoint per service and one queue per receiving endpoint so this is fine.

You received this message because you are subscribed to a topic in the Google Groups "masstransit-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/masstransit-discuss/m8f_AfCFMYg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYdUN1o%3D2jcxu2hgbNwa8CNtQvRyW6SJ6ErKmbdXwgiXnA%40mail.gmail.com.

[Alexey Zimarev]

I think I know what it is. These are not duplicates. I believe when saga state changes, it cannot be persisted properly in the join due to parallel processing and one of the updates fails with deadlock, however all Send and Publish are already done. Then we get to a retry for the When and these Send and Publish fire again, then saga persists properly and everything works. That is happens in HA environment is a coincidence.

Currently I think the only way to avoid this is to set concurrent processes number to one.

[Chris Patterson]

x.ReceiveEndpoint(host, "my_queue", e =>

{

    e.UseInMemoryOutbox();

});

That will enable the outbox so that messages aren't published until the saga is committed.

On Fri, Aug 12, 2016 at 2:17 PM, Alexey Zimarev <azim...@gmail.com> wrote:

I think I know what it is. These are not duplicates. I believe when saga state changes, it cannot be persisted properly in the join due to parallel processing and one of the updates fails with deadlock, however all Send and Publish are already done. Then we get to a retry for the When and these Send and Publish fire again, then saga persists properly and everything works. That is happens in HA environment is a coincidence.

Currently I think the only way to avoid this is to set concurrent processes number to one.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstransit-discuss@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/1114284f-0cd8-4784-8263-c2c976ad6670%40googlegroups.com.

[Alexey Zimarev]

Definitely, the Outbox helps. Thank you Chris.