Question on atomicity guarantees?

[edal...@gmail.com]

I was reading the Semantics of Tx article and there is something I would like to validate:

> AMQP guarantees atomicity only when transactions involve a single queue, i.e. all the publishes inside the tx get routed to a single queue and all acks relate to messages consumed from the same queue. When multiple queues are involved it is possible that in the event of a broker failure during tx.commit the effects of the transaction are  only visible in some of the queues. Furthermore, RabbitMQ provides no atomicity guarantees even in case of transactions involving just a single queue, e.g. a fault during tx.commit can result in a sub-set of the transaction's publishes appearing in the queue after a broker restart.

Oh boy, those are pretty lousy transaction guarantees :-)

Anyways, focusing on the part that talks about multiple queues in transactions: if I am understanding this correctly, this implies that any publishing of messages to fanout or topic exchanges with more than one target queue cannot guarantee the atomicity of the transaction as per the explanation above.

The thing is that one way to interpret the statement above could be simply about writing a transaction that explicitly publishes to different queues directly like

ch.txSelect();

ch.basicPublish("", QUEUE_NAME1, MessageProperties.PERSISTENT_BASIC, "nop".getBytes());
ch.basicPublish("", QUEUE_NAME2, MessageProperties.PERSISTENT_BASIC, "nop".getBytes());
ch.txCommit();

But I suppose this would be pretty much the same thing as to publish to a fanout or topic exchange that would end up sending messages to those two queues as well.

channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes());

So, I just would like to validate that the interpretation of statement above means both things, and not just the first, because at least in my experience the first case might be more unusual within a transaction, but the second one is very, very common, and so I just want to understand the implications of a failure like the one described above.

Am I interpreting this correctly?

[Michael Klishin]

Yes. Transactions are also scoped per channel.

Besides those limitations transactions are not worth the overhead for most users, that's why

publisher confirms were introduced: http://www.rabbitmq.com/confirms.html.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[edal...@gmail.com]

Thanks for the suggestions Michael

I spent some time yesterday trying to understand how to use publisher confirms to ensure delivery of my messages. 

I need to use this feature in the context of a database transaction. So, what I do now is that I first persist my database transaction, and have a database poller that looks for unpublished records in my database and send them to RabbitMQ. Using the publishing confirms callback, when I get the confirmation back from Rabbit I mark my record in the database as published. This was relatively easy to implement with Spring Integration, but I'm struggling with other problems this solution introduces.

Since I will most likely scale my services to multiple instances, now I have the problem that I either need to ensure only a single poller exists, or to partition the data somehow such that multiple, distributed/concurrent pollers do not step into each others toes. I will have to device a way to identify ever publisher uniquely and make sure it has a corresponding poller serving it.

Maybe I'm just making this more complicated than it needs to be. Do you have any advise for me on this?

Thanks in advance for your feedback. 

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Why not make your publishers take care of "acknowledging" to the database what was acknowledged
to them by RabbitMQ?

That's basically how a client-side write-ahead log would work. I can't think of a significant reason
to have a separate process/service/thread that would poll.

> >> *> AMQP guarantees atomicity only when transactions involve a single

> >> queue, i.e. all the publishes inside the tx get routed to a single queue
> >> and all acks relate to messages consumed from the same queue. When multiple
> >> queues are involved it is possible that in the event of a broker failure
> >> during tx.commit the effects of the transaction are only visible in some
> >> of the queues. Furthermore, RabbitMQ provides no atomicity guarantees even
> >> in case of transactions involving just a single queue, e.g. a fault
> >> during tx.commit can result in a sub-set of the transaction's publishes

> >> appearing in the queue after a broker restart.*

> To post to this group, send an email to rabbitm...@googlegroups.com.

[edal...@gmail.com]

And where do you deal with the republishing of messages that were not acknowledged?

So I was thinking, if any message fails to be published to Rabbit I don't get the confirmation and therefore I am supposed to republish it, right? Now, the only evidence I have left of a message not being successfully published would be my database record, and that's why I thought a database poller was needed here. Of course I could keep track in memory of which of my published messages got a confirmation and which did not, but that memory structure would survive a system shutdown or system crash.

Do you have any suggestion to deal with that?

[Michael Klishin]

OK, a separate process makes sense for re-publishing.

In some domains you can republish safely, in others you can't. Consumers should be
prepared to handle duplicates since they can happen even without re-publishing
(at least with manual acknowledgements and realistic networks in mind).

[edal...@gmail.com]

But then, as I was saying, this additional polling process is tricky to do right, because if I have multiple instances of my application, I must make sure only one of them runs the poller, or make sure that all of them run the poller but operating on a partition of the data.

I realize now that doing publisher confirms the right way is not that simple. Something tells me this is not simplest way to do it. I will have to ponder about this a few more hours. I have a few other ideas to try, but as I was saying, it is not immediately obvious how this is supposed to work. I think there is some lack of documentation in the community in general regarding how to work with publisher confirms. At least that's the feeling I get now that I face the problem for the first time.

[Michael Klishin]

Documentation is open source and you are welcome to contribute.

Distributed systems are hard, as is coordination between multiple instances of an application.
Library authors such as our team have no idea what the semantics
of the publisher are.

We'd like to introduce a disk-based write-ahead log for published messages but that is
both tricky to get right and is frowned upon in PaaS-like deployment scenarios. And then it has
to be supported by every popular client, of which there are more than 10.

Just like with connection recovery a few years ago, before one client library gets it right and a year or so
passes, introducing new abstractions around publisher confirms is very risky.