Routing DLX message back to originating publisher

[Ive]

Note: Original question on SO -- perhaps it's easier to read there: http://stackoverflow.com/q/26881761/520050

I have multiple web-service server processes receiving requests from web-service clients, performing some processing (what I want to use Rabbit & a task queue for), and then returning the result back to to the respective client. 

However I want to be able to wait for a consumer of this queue to pick up a published request within a timeout. If this timeout expires, then I want the original publisher to receive a notification of this so it can feed back to the client.

Simplistically:

• I have multiple publishers publishing messages to the queue
  
• There are multiple consumers of this queue
• I want to notify the publisher if a consumer doesn't consume a message within a certain time.

Essentially I'm trying to implement a sort of 'Publish TTL', whereby the originating publisher (only) is notified if a message is dead-lettered. (I want the publisher to know if a message is not consumed within a certain timeout).

I know this is similar to various conversations about 'Unbounded queues' (https://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/), however my question is more about notifying the publisher

I have a solution, (which is working) as follows:

• When publishing messages, each publisher uses a routing key containing a unique (per publisher) Process-ID, e.g. 'thekey.pid3' for publisher 3
• Consumers each consume from the same queue, that is bound to the exchange using the routing key 'thekey.#'
• I'm not explicitly specifying any routing-key for the DeadLetter Exchange (DLX) when publishers declare the queue -- only the 'x-dead-letter-exchange'. This means that if a message is deadlettered, then the original routing-key (i.e. 'thekey.pid3') will carry through to the DLX.
• Each publisher binds it's own new consumer queue to the DLX using it's specific routing key, e.g. 'thekey.pid3'

This, however, seems a little convoluted...and inefficient, as the wildcard routing occurs for ALL messages (except those dead-lettered -- which shouldn't _normally_ be happening). I'd rather be using a Direct Exchange.

This must be a common requirement -- Is there a far better, more efficient (simpler?) solution? Or does this look viable...

I've attached a diagram of my attempt -- hopefully that clarifies things

Many thanks

Ive

[Michael Klishin]

 On 12 November 2014 at 17:20:11, Ive (ive.c...@gmail.com) wrote:
> Essentially I'm trying to implement a sort of 'Publish TTL',
> whereby the originating publisher (only) is notified if a message
> is dead-lettered. (I want the publisher to know if a message is
> not consumed within a certain timeout).

Ive,

Do you mean "consumed" or "processed" (whatever that may mean in your case)?
If the latter, you have a pretty common request/reply (RPC) case where consumers
can notify publishers.

The former case is indeed trickier.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ive]

Thanks Michael 

Yes, I do mean 'consumed'. I've already implemented an RPC-type mechanism to get processed responses back to the client, but I want to notify ONLY the publisher of the message if that message is not consumed with a certain time. A DLX looked like the right mechanism to do this, but getting the deadlettered message back to only the client that published it seems to be a little tricky.

Basically I want to handle the case where consumers either haven't actually started up yet, or if they are all too busy to pop messages off the queue; so that I can notify the client within some reasonable timeout.

Ive

[Michael Klishin]

On 12 November 2014 at 17:33:58, Ive (ive.c...@gmail.com) wrote:
> Yes, I do mean 'consumed'. I've already implemented an RPC-type
> mechanism to get processed responses back to the client, but
> I want to notify ONLY the publisher of the message if that message
> is not consumed with a certain time. A DLX looked like the right
> mechanism to do this, but getting the deadlettered message back
> to only the client that published it seems to be a little tricky.
>
> Basically I want to handle the case where consumers either haven't
> actually started up yet, or if they are all too busy to pop messages
> off the queue; so that I can notify the client within some reasonable
> timeout.

Right, so combining a TTL on messages and a DLX is the way to go. There previously
was a feature in RabbitMQ that handled this specific case but also carried a massive
performance penalty for clusters that didn't use it (immediate publishing).

If your publisher declares a new uniquely named queue to collect responses, can you
set its x-dead-letter-routing-key argument in such a way that it only receives
dead lettering notifications for that specific message?

Then your publisher would get 2 types of responses in a single queue, which is
quite manageable. Use the "type" property on messages to make it easy to tell
timeouts from results and propagate that notification to a Web client.

That doesn't sound very complicated. Perhaps I'm missing some limitation of
how x-dead-letter-routing-key can be set up in your specific  case?

[Ivor Chalton]

OK, so it sounds like I'm on the right track.

>If your publisher declares a new uniquely named queue to collect responses, can you
>set its x-dead-letter-routing-key argument in such a way that it only receives
>dead lettering notifications for that specific message?

Apologies Michael, I'm not with you here. 

- Where would these responses be coming from? 

- Is the x-dead-letter-routing-key not specified on a 'normal' queue to tell Rabbit what to replace the routing key with when dead-lettering a message? Or have I got that totally wrong?!

Ive

[Michael Klishin]

 On 12 November 2014 at 17:57:22, Ivor Chalton (ive.c...@gmail.com) wrote:
> Apologies Michael, I'm not with you here.
> - Where would these responses be coming from?

The DLX. "Notification" is a loosely used term here, I really meant a message.
Which also means the type won't vary (my bad) but exchange name should
be set to the DLX.

> - Is the x-dead-letter-routing-key not specified on a 'normal'
> queue to tell Rabbit what to replace the routing key with when
> dead-lettering a message? Or have I got that totally wrong?!

It can be specified on *any* queue. I've suggested using response queues
because they are already used by a single producer.

[Ive]

No worries - I read 'message'; and understand that you can declare the DLX on any queue. 

Back to this then:

>If your publisher declares a new uniquely named queue to collect responses, can you 
>set its x-dead-letter-routing-key argument in such a way that it only receives 
>dead lettering notifications for that specific message? 

So, you're saying set the x-dead-letter-routing key on the response-queue already defined by the publisher to consume its RPC responses? Wouldn't that just attempt to deadletter (already successfully consumed, processed) messages published to that queue?

[Michael Klishin]

On 12 November 2014 at 18:43:54, Ive (ive.c...@gmail.com) wrote:
> So, you're saying set the x-dead-letter-routing key on the
> response-queue already defined by the publisher to consume
> its RPC responses? Wouldn't that just attempt to deadletter
> (already successfully consumed, processed) messages published
> to that queue?

No, as responses won't have the TTL (which can be set per-message). 

[Ive]

I'm wasting your time here - I'm so sorry. I'll try one last time before hitting the documentation again:

A message is published to the 'common' queue 'A' (common to all publishers), and can be deadlettered if it's TTL expires.

Non-deadlettered messages get consumed and _eventually_ (via a number of other exchanges)  get published to a uniquely-defined (response) queue, B -- that the originating publisher (alone) consumes from.

You're saying to set the x-dead-letter-routing key on 'B', but how does that result in getting a deadletter message from 'A' back to the originating publisher? Wouldn't that only deadletter 'response'-messages published to 'B' (which have a TTL set)?

[Michael Klishin]

On 12 November 2014 at 19:01:38, Ive (ive.c...@gmail.com) wrote:
> You're saying to set the x-dead-letter-routing key on 'B',
> but how does that result in getting a deadletter message from
> 'A' back to the originating publisher? Wouldn't that only deadletter
> 'response'-messages published to 'B' (which have a TTL set)?

A message M will expire and be dead lettered to a DLX that should route M to B.

Every publisher can even have its own DLX of type fanout so that you don't have to think
much about how to make things route. Just bind B to it. DLX can be declared once per publisher
on application boot.

[Ive]

...and the penny drops. 

So I'm binding my Response queue (B) to more than one exchange. I'd read that was possible, but hadn't considered a use for it...well there ya go.

One point about the DLX being a fanout: Because all publishers publish to the same queue, I have to specify the same DLX on that queue when each publisher application starts (don't I?). If the DLX is a fanout, then every publisher will get every dead-lettered message (even from other publishers) deposited into it's response queue! 

[Michael Klishin]

On 12 November 2014 at 19:45:18, Ive (ive.c...@gmail.com) wrote:
> So I'm binding my Response queue (B) to more than one exchange.
> I'd read that was possible, but hadn't considered a use for it...well
> there ya go.

Sure, you can have hundreds or thousands of bindings for a single queue. Not that
you need that many often but technically you can.

> One point about the DLX being a fanout: Because all publishers
> publish to the same queue, I have to specify the same DLX on that
> queue when each publisher application starts (don't I?). If
> the DLX is a fanout, then every publisher will get every dead-lettered
> message (even from other publishers) deposited into it's response
> queue!

Note that I suggested that every publisher has its own DLX. It then can use the simplest
routing logic possible, which is fanout if you ask me.

[Ive]

Right. One queue, lots of bindings to multiple exchanges - gotcha.

Regarding the fanout: Because each publisher is publishing to the same queue, then each publisher declaring the queue to specify a different DLX doesn't make sense, does it?! (There can be only one "x-dead-letter-exchange" associated with a queue's arguments).

I've just tested this, and even though it seems I'm allowed to re-declare the publish-queue with a different x-dead-letter-exchange, only the first one to be declared is used when deadlettering. 

What am I missing?

[Michael Klishin]

 On 13 November 2014 at 10:23:56, Ive (ive.c...@gmail.com) wrote:
> Regarding the fanout: Because each publisher is publishing
> to the same queue, then each publisher declaring the queue to
> specify a different DLX doesn't make sense, does it?! (There
> can be only one "x-dead-letter-exchange" associated with a
> queue's arguments).

Each publisher is publishing to the same exchange. And perhaps that exchange routes
all messages to the same queue. Fine.

What I've been suggesting all this time is

 * You need a separate queue per publisher
 * You already have a unique queue per publisher used to collect responses
 * You decide what extra arguments that queue has.

The key thing I've been trying to communicate is that you need a _unique queue_
and a _unique DLX_ which also means that the DLX can be of type fanout for simplicity.

[Ive]

Thanks Michael - I appreciate the patience.

I see - so a unique DLX AND unique publisher-queue per publisher is the pattern you're suggesting. With that, then I need to figure out how to implement the consuming (and load-sharing) side of these queues with consumers not knowing the (dynamically generated) publisher-queue name(s).

I'll do a bit more research and have a think - thanks again.

Ive

[Michael Klishin]

On 13 November 2014 at 12:28:02, Ive (ive.c...@gmail.com) wrote:
> I see - so a unique DLX AND unique publisher-queue per publisher
> is the pattern you're suggesting. With that, then I need to figure
> out how to implement the consuming (and load-sharing) side of
> these queues with consumers not knowing the (dynamically generated)
> publisher-queue name(s).

No, the unique queues in RPC (request/response) are used to collect replies. Consumers
can use a single shared queue if you want.

Tutorial 6: http://www.rabbitmq.com/getstarted.html.

[Michael Klishin]

On 13 November 2014 at 12:29:30, Michael Klishin (mkli...@pivotal.io) wrote:
> No, the unique queues in RPC (request/response) are used to
> collect replies. Consumers
> can use a single shared queue if you want.

…of course, you can use a separate one entirely (e.g. every publisher will declare
publisherN.replies and publisherN.timeouts), that may simplify telling timeouts from
legit responses.

But the point is still the same, these are not the queues that consumers will use to get requests, only to publish response (or, as described above, some are used by consumers and some are by DLXes) .

[Ive]

Apologies for the delay in responding here, Michael

Yes, I understand that each publisher needs its own unique queue for receiving it's own timed-out messages. And sure - we can use the RPC-response queue for this. 

At the risk of repeating myself, I lost you here:

>If your publisher declares a new uniquely named queue to collect responses, can you
>set its x-dead-letter-routing-key argument in such a way that it only receives
>dead lettering notifications for that specific message?

and

>The key thing I've been trying to communicate is that you need a _unique queue_ 
>and a _unique DLX_ which also means that the DLX can be of type fanout for simplicity. 

You're suggesting a unique (fanout) DLX defined on each publisher's response queue; I'm unclear how that relates to my goal of dead-lettering requests published to the request queue? 

Perhaps I wasn't clear originally: I'm looking to dead-letter messages sent to the queue you mention here:

>Each publisher is publishing to the same exchange. And perhaps that exchange routes 
>all messages to the same queue. Fine. 

Many thanks,

Ive

[Michael Klishin]

 On 14 November 2014 at 17:36:07, Ive (ive.c...@gmail.com) wrote:
> You're suggesting a unique (fanout) DLX defined on each publisher's
> response queue; I'm unclear how that relates to my goal of dead-lettering
> requests published to the request queue?

The response queue can also receive timed out requests, or you can have a separate
queue (per publisher) to bind to the DLX altogether. The latter would potentially require one extra
round trip per request but also make your consumer code more straightforward.

[Ive]

So to be clear: Just one one DLX now, set as a parameter on the request queue, with the publisher-response queues bound to this?

[Michael Klishin]

On 14 November 2014 at 18:09:41, Ive (ive.c...@gmail.com) wrote:
> So to be clear: Just one one DLX now, set as a parameter on the request
> queue, with the publisher-response queues bound to this?

Sounds right.

P ---> RQ ----> DLX ----> TOQ

P = Publisher
RQ = Request Queue 
TOQ = Timed out requests Queue

[Ive]

This is exactly the solution I had. Please have a look at the diagram I attached to my first post (I've attached it here again). 

What I'm worried about is the routing mechanism:

>This, however, seems a little convoluted...and inefficient, as the wildcard routing 

>occurs for ALL messages (except those dead-lettered -- which shouldn't _normally_ be happening). 

>I'd rather be using a Direct Exchange.

>This must be a common requirement -- Is there a far better, more efficient (simpler?) solution? 

>Or does this look viable...

Are my fears unfounded? Are you happy with a topic exchange and every client (worker) using the one-and-only request queue with a bindingkey of 'thekey.#' ? It really just didn't seem correct to me - although it works. 

Thanks!

[Michael Klishin]

On 14 November 2014 at 18:29:44, Ive (ive.c...@gmail.com) wrote:
> Are my fears unfounded? Are you happy with a topic exchange and
> every client (worker) using the one-and-only request queue
> with a bindingkey of 'thekey.#' ? It really just didn't seem correct
> to me - although it works.

Topic exchange implementation is not naive, so I doubt it will noticeably affect
overall latency:

http://www.rabbitmq.com/blog/2010/09/14/very-fast-and-scalable-topic-routing-part-1/
http://www.rabbitmq.com/blog/2011/03/28/very-fast-and-scalable-topic-routing-part-2/

So, I'd start with what you have  and worry about routing if it actually becomes an issue.

[Ive]

Great, thanks Michael - will do.

I certainly learnt a lot trying to figure out what I mistakenly thought your proposed solution was!

Ive