Federated Exchanges :: Do queues in downstream federated exchange hosts need to be pre-declared?

[frith]

I'm setting up a Federated Exchange relationship between two hosts (upstream & downstream)

Everything's working well.. so far... I have an upstream relationship set up and a link as well.. log files seem to indicate this relationship is working properly....

However, in my downstream host, I see that there are no queues defined and bound to the federated exchange.

Do the federated operations automatically create queues if they don't exist, or, do I need to predeclare all the queues in the downstream host(s)?

The exchange we want to federate has a bunch of dynamically named queues we also want to federate downstream.

We're trying to figure out if we need to write an application that constantly polls downstream hosts and creates queues, or, if federation takes care of this?

Thanks in advance.

[Michael Klishin]

Federation will declare the queues it needs to perform replication. The rest is up to you. Those exchanges

should probably be bound to something in order to be useful. This can be true for the upstream cluster as well.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[frith]

Thanks for all your help today, it's much appreciated.

Help me understand you last reply...

Here's what I have:

Upstream Server (What i want federated/replicated) :

host = rabbit.foo.bar

vhost = /app

exchange =  app.exchange.topic  (there are a ton of queues bound to this exchange)

Downstream server:

rabbitmqctl set_parameter -p '/app' federation-upstream fedex-app '{ "uri":"amqp://user:pa...@rabbit.foo.bar/%2fapp" , "prefetch-count":1000, "reconnect-delay":1, "ack-mode":"on-confirm", "trust-user-id":false, "exchange":"app.exchange.topic", "max-hops":10, "message-ttl":86400000 }'

rabbitmqctl set_policy federate-app -p '/app' '^app\.' '{"federation-upstream":"fedex-app"}'


Once I run those two commands, i see federation appears to be setup correctly, what i dont see in the downstream host are the queues that are present upstream.

What am I missing?

Thanks again

[Michael Klishin]

How do you observe that "federation appears to be set up correctly"?

What makes you believe that exchange federation will replicate your queues?

It will replicate an exchange (of special type, x-federated) and upstream exchange's bindings there.

It will not declare any queues beyond what it needs to operate.

There will be a federation link per federated exchange.  '^app\.' should match all exchanges

that begin with "app.".

Queues are created by queue federation — to make all matching queues appear on/in all federated nodes/clusters [1].

Exchange federation, as the name suggests, federates exchanges.

1. https://www.rabbitmq.com/federated-queues.html

[frith]

Ok, I see what you're saying.. let's leave queues out of the equation for the moment.

Another question/concern i have is i see no "activity" for the federated exchange.  

let me tackle that first.  thanks again for all your help here...

>> How do you observe that "federation appears to be set up correctly"?

1.  in my downstream, im seeing a federated link in a running state:

2.  in my upstream, i see something that i would assume suggests this is setup/working?  in this image below is the exchange i want to replicate to my downstream, and, what appears to be a properly configured federation?:

3.  in reference to the image in 2 above, should i expect to see message rates > 0 since the exchange is federated, right?   Is the federated exchange in fact, not working properly?

Omitting queues from the conversation altogether, do you see anything above that suggests federation of the exchange is not setup correctly?  

Thank you

On Friday, February 15, 2019 at 3:15:55 AM UTC-5, Michael Klishin wrote:

How do you observe that "federation appears to be set up correctly"?

What makes you believe that exchange federation will replicate your queues?

It will replicate an exchange (of special type, x-federated) and upstream exchange's bindings there.

It will not declare any queues beyond what it needs to operate.

There will be a federation link per federated exchange.  '^app\.' should match all exchanges

that begin with "app.".

Queues are created by queue federation — to make all matching queues appear on/in all federated nodes/clusters [1].

Exchange federation, as the name suggests, federates exchanges.

1. https://www.rabbitmq.com/federated-queues.html

On Thu, Feb 14, 2019 at 9:54 PM frith <erriv...@gmail.com> wrote:

Thanks for all your help today, it's much appreciated.

Help me understand you last reply...

Here's what I have:

Upstream Server (What i want federated/replicated) :

host = rabbit.foo.bar

vhost = /app

exchange =  app.exchange.topic  (there are a ton of queues bound to this exchange)

Downstream server:

rabbitmqctl set_parameter -p '/app' federation-upstream fedex-app '{ "uri":"amqp://user:pass@rabbit.foo.bar/%2fapp" , "prefetch-count":1000, "reconnect-delay":1, "ack-mode":"on-confirm", "trust-user-id":false, "exchange":"app.exchange.topic", "max-hops":10, "message-ttl":86400000 }'

[frith]

I'm able to see messages now in my downstream exchange.

I had to add a binding to the federated queue in the upstream server to the exchange i want to federate and add a routing key of '#'

I guess I had assumed this binding was implicit in federation, or, did i just hack something ugly together by doing that?

[Michael Klishin]

I don't think you want all the gory details but exchange federation does propagate bindings. That's one of the reasons why

there's a special exchange type that you see in the row, x-federation-upstream. The gray row is an exchange

that's federated, so yes, your setup is correct.

Federated exchanges will only receive a stream of messages if the source (the upstream exchange) does.

What I tried to communicate earlier is that exchange federation does not manage downstream exchange's bindings for you.

It cannot know what you want the topology to look like ;) What it does "know" (assumes) is that when a downstream exchange is bound

or unbound, the changes are reflected on the federated upstream side.

And then there is a message looping protection mechanism. That's mostly it.

I re-read the guide on federated exchanges and would agree that it needs love and editing.

rabbitmqctl set_parameter -p '/app' federation-upstream fedex-app '{ "uri":"amqp://user:pa...@rabbit.foo.bar/%2fapp" , "prefetch-count":1000, "reconnect-delay":1, "ack-mode":"on-confirm", "trust-user-id":false, "exchange":"app.exchange.topic", "max-hops":10, "message-ttl":86400000 }'

[frith]

Thank you again Michael,

From start to finish in a few hours for something as complex as federation says lots of good things right off the bat.  Kudos to you and the team.

Add to that, the fact that I was able to post questions and get so much helpful info from you so quickly, I have nothing but praise at this point.

Sure, documentation could use some sprucing up.. I'm sure that's true on my side of the fence, too.  :-)

Thank you again for all the help.

rabbitmqctl set_parameter -p '/app' federation-upstream fedex-app '{ "uri":"amqp://user:pass@rabbit.foo.bar/%2fapp" , "prefetch-count":1000, "reconnect-delay":1, "ack-mode":"on-confirm", "trust-user-id":false, "exchange":"app.exchange.topic", "max-hops":10, "message-ttl":86400000 }'