Reliable publishing from a Rails controller

[Jon Zeppieri]

Simple description of the problem:

I want to accept data via HTTP, save it somewhere, then stick the id of the data on a queue for further processing. If I can't confirm that the message has been persisted on the queue, I want to return an error to the HTTP client, indicating that the operation failed and will need to be retried at some point in the future. This pretty much requires a synchronous operation, even if it's built on top of asynchronous primitives.

From what I can tell, there are three options for reliable publishing (and I don't claim to understand all of the detail of each):

- transactions

- publisher confirms (handling acks and nacks)

- mandatory messages (handling returns)

Given the requirement that I need to return a response to the HTTP client, is there any good reason not to use transactions? My understanding (possibly incorrect) is that the commit is a synchronous operation. And, although I understand that transactions come with a high performance penalty, my other options involve blocking the HTTP request until I get back some kind of acknowledgement, and it's not clear to me that there is any time limit on how long I could be waiting for this, unless I impose one myself.

[Michael Klishin]

The reasons against transactions are covered in the blog post that introduced

Publisher confirms [1].

Mandatory publishing only covers routing and is orthogonal to publisher confirms.

The fundamental issue here is that you want to return a response to an HTTP client early but

a confirmation and any error handling you might want to do can happen after the request has completed.

It makes most sense to me to hand off publishing to a thread not involved in HTTP response handling

and always available. Note that if there's more than one such thread you will have concurrent publishing

which is a no-no on a shared channel and even with separate channels will affect message ordering.

The most common approach to this is putting an object to a local Queue data structure and having

a thread that continuously checks if its empty, and if not, dequeues the context object and does something,

e.g. publishes to RabbitMQ and handles errors. That can be done asynchronously (as publisher confirms inherently are)

or using Bunny::Channel#wait_for_confirms with a timeout. You can also do batching with a timeout.

Obviously error handling in such thread is very important.

More advanced techniques involve inserting a "WAL record" somewhere like a data store or local storage (if available) so

that if the request cannot complete or publishing fails or the Rails process is shut down before the message can go out,

those messages can be republished. After receiving a confirm for a message it has to be removed from "the WAL".

Actors-based solutions can be used as well but I'm not up-to-date with what's the most popular one in Ruby

these days and internally they are not significantly more sophisticated than the "a Queue and thread" example above

but might be easier to use if concurrent programming is not something you have a lot of experience with.

1. https://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/

--
Bunny: http://rubybunny.info
March Hare: http://rubymarchhare.info
 
IRC: #rabbitmq on irc.freenode.net
 
Post to the group: ruby...@googlegroups.com | unsubscribe: ruby-amqp+...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "Ruby RabbitMQ libraries" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ruby-amqp+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jon Zeppieri]

On Fri, Jan 18, 2019 at 9:57 AM Michael Klishin <mkli...@pivotal.io> wrote:

The reasons against transactions are covered in the blog post that introduced

Publisher confirms [1].

The only reason I see given on that page is that they "can be unacceptably slow." Are there other problems with them?

 

Mandatory publishing only covers routing and is orthogonal to publisher confirms.

The fundamental issue here is that you want to return a response to an HTTP client early [...]

I suppose that depends on what you mean by "early." I want the HTTP response to indicate success only if the message is safely in Rabbit. I don't mind waiting *some* amount of time for that information -- especially if the alternative is to build my own local queue with a write-ahead log and retry capability, which exists for the sole purpose of getting messages onto another queue.

In the end, my basic position is: I need a  to use a *synchronous* API during the HTTP request. It looks like I have my choice of two: transactions or `wait_for_confirms`.[1] 

If using either of these in an HTTP request is a non-starter, that would be good to know. In that case, I'll use something other than Rabbit. 

[1] It seems like it shouldn't be difficult to provide an API that would wait for confirmation of the message that was just sent, instead of waiting for confirmation of all of them, given the sequence number of the message in question.

- Jon

[Michael Klishin]

No other problems if you really understand what they cover.