x-first-death-queue & headers exchange

[Brett Janer]

Hey rabbitmq-users,

I'm currently trying to set up a simple retry system by shoveling from a deadletter queue to a headers exchange, and then routing based on the x-first-death-queue header.  It seems like binding for the x-first-death-queue header is being ignored.  Is this design?

RabbitMQ v3.7.12

Thanks,

Brett

[Brett Janer]

I guess I should have checked github before posting.  It seems like the 'x-first-death-queue' header was introduced with intentions to be used with the headers exchange (Consider adding separate headers for original dead-lettering event ("x-death") #1332).  In the issue, sylvainhubsch points out that "all headers beginning with "x-" can't be matched".  This seems to have been missed when implementing.  Does this sound worthy of opening a new github issue?

Thanks,

Brett 

[Luke Bakken]

Hi Brett,

I put together a script to set up two queues and dead-lettering in the same manner that I think you expect. The message lives in the first queue for 10 seconds, then is dead-lettered to the amq.headers exchange, where it is routed to the second queue. You can use any headers exchange in your environment - I just used the default.

https://gist.github.com/lukebakken/16314770aecbc492bb4c434b84221ad6

You need to have the rabbitmq_management plugin enabled (you do already) and python and rabbitmqadmin in your PATH to run it. It can be downloaded here: https://github.com/rabbitmq/rabbitmq-management/blob/master/bin/rabbitmqadmin

I tested this with RabbitMQ 3.7.12

Thanks,

Luke

[Brett Janer]

Hey Luke,

Thank you for putting some time into this.  Yes your script does work, but it's because the amq.headers exchange is acting more like a fanout exchange and not actually applying any filtering.  Add a third queue to your script and bind it to the amq.headers exchange with a different x-first-death-queue argument value to see what I mean.

[Luke Bakken]

Hi Brett,

Thanks for clarifying that.

Yep, this does seem like a bug. I updated the script as you described as well as tested with and without an x-match binding argument and it didn't make a difference.

I'll update this thread with more info as I find it out.

Luke

[Michael Klishin]

IIRC dead lettering only uses a fixed routing key specified in the x-arguments at queue declaration time.

Luke Bakken:

[Luke Bakken]

Brett -

Now that I know what I'm looking for, it is documented: https://www.rabbitmq.com/dlx.html#routing

When you declare your original queues, you can set the x-dead-letter-routing-key queue argument to route the message the way you'd like, if that routing differs from the routing key of the message.

Thanks -

Luke

[Luke Bakken]

Hi again Brett -

I have updated my gist to use routing keys and the amq.direct exchange, and I see the expected behavior -

https://gist.github.com/lukebakken/16314770aecbc492bb4c434b84221ad6

Have a good weekend -

Luke

[Brett Janer]

Thanks Luke! Really appreciate the help.  I'll change my approach to use routing keys as suggested.  With your script as a guide, I'm pretty sure I can get there.  On a side note, what really tripped me up for a bit was the fact that the headers exchange doesn't take into account headers that are prefixed with 'x-'.  If there is documentation on this, I couldn't find any.  If there's not, do you think it'd be worth while to add?

[Luke Bakken]

Hey Brett,

The headers exchange does work with headers that begin with x- ... I added another example to the gist that demonstrates: https://gist.github.com/lukebakken/16314770aecbc492bb4c434b84221ad6#file-headers-exch-sh

It's just the interaction between dead-lettering and the headers exchange doesn't work as you (or I, initially) think it should. But, at least there's a workaround using routing keys.

Thanks!

[Brett Janer]

Are you sure it's not acting like a fanout?

[Luke Bakken]

I sure stand corrected! Gist is updated.

I'll make sure this is clearly documented.

[sylvain...@gmail.com]

Brett Janer : you are right.

I'm sorry but once again, it is just a fact that all argument's key of any binding in a headers exchange starting by "x-" other than "x-match" is just NOT taken into account, point.

It is definitively coded that way :

<<

% Delete bindings starting with x- headers_match([{<<"x-", _/binary>>, _PT, _PV} | PRest], Data, AllMatch, AnyMatch, MatchKind) -> headers_match(PRest, Data, AllMatch, AnyMatch, MatchKind);

>>

(I know that, I worked on it !)

If you are not convinced yet, please do a real test by yourself :

* take the latest RabbitMQ's version on Docker (3.7.14 now)

* create a queue "q1"

* in some headers exchange, create a simple binding with only 1 argument's key "x-foo" with value "bar" linked to q1 (x-match will default to "all" type)

* in that same exchange, just publish a message with an argument's key "x-foo" with value "NOTBAR at all but alors pas du tout !"

* and please look at the simple fact that your message is.. routed

But this is not a "bug", this comes from AMQP specs dealing with argument's keys starting by "x-" which are "reserved for future use" : it is an historic implementation, which I can totally understand !

So yes, as I said before when I wrote that it << can't be matched >>, it just cannot be :)

By the way, there is a new exchange type with which you can do things like that : https://github.com/sylvain-hh/rabbitmq-open-exchange (but that's another story).

SylvainHH.