Naming upstream and downstream federated exchanges

[uvzu...@gmail.com]

I have virtual hosts "A" and "B". In "B" I defined a federation upstream pointing to "A" like so:

amqp://uid:pwd@host:5672/A

and a federation policy like so:

pattern=^X, applies-to=exchanges

Does this mean that I have to have exchanges named "X" in both virtual hosts "A" and "B"? If so, may I name upstream and downstream exchanges differently? I've seen references to the "exchange" parameter that would allow me to name the upstream exchange differently from the downstream exchange, but it's not obvious where to supply it and how. Would you please give an example? 

Thanks in advance!

[Michael Klishin]

On 27 January 2016 at 00:57:23, uvzu...@gmail.com (uvzu...@gmail.com) wrote:
> Does this mean that I have to have exchanges named "X" in both
> virtual hosts "A" and "B"?

You don't but 99% of exchange federation users probably do. This is because
exchange federation will use the name of the upstream exchange (the one in A in your case)
when declaring a downstream one (the one in B).

However, you can customize upstream exchange name (see below).

> If so, may I name upstream and downstream
> exchanges differently? I've seen references to the "exchange"
> parameter that would allow me to name the upstream exchange differently
> from the downstream exchange, but it's not obvious where to supply
> it and how. Would you please give an example?

The "exchange" option lets you specify upstream exchange name.
The easiest way to try it is by enabling the federation_ management plugin and looking
under the Admin tab in management UI.

We have some tests that use it [1] but they are hardly easy to digest examples.

1. https://github.com/rabbitmq/rabbitmq-federation/blob/master/test/src/rabbit_federation_exchange_test.erl
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

+rabbitmq-users 

On 27 January 2016 at 01:53:00, uvzu...@gmail.com (uvzu...@gmail.com) wrote:
> Michael, where is the "exchange" option? When I define the upstream
> or when I define the policy or somewhere else? Could you please
> be specific? Thanks in advance!

It's an upstream option, so provided to `rabbitmqctl set_parameter`.
Policies then "connect" exchanges that should
be federated with an upstream:
http://www.rabbitmq.com/federation-reference.html

(I understand that that page could use some  editing and federation docs weren't written with beginners in mind)

[uvzu...@gmail.com]

Is it not exposed via the management UI? All I see is virtual host, name, uri, expires, ttl, hops, prefetch, delay, ack mode, and trust user id.

[Michael Klishin]

ah, shoot! That seems to be the case. The option was introduced in 2012. You can tell it is
not particularly popular.

F iled https://github.com/rabbitmq/rabbitmq-federation-management/issues/5.

[uvzu...@gmail.com]

That explains it! ;) I suspect we cannot use rabbitmqctl when Rabbit is deployed to PCF since we cannot login the VM, or can we?

[Michael Klishin]

On 27 January 2016 at 02:24:04, uvzu...@gmail.com (uvzu...@gmail.com) wrote:
> I suspect we cannot use rabbitmqctl when Rabbit is deployed
> to PCF since we cannot login the VM, or can we

Technically you can use `rabbitmqctl` with any remote node (see the -n flag) but specifically on Cloud Foundry
there are two significant pain points:

 * Node hostname is generated and is not human-friendly at all
 * You'd still need to obtain the Erlang cookie from the VM

So I suspect we should add the option to federation_management in a future 3.6.x release.

May I ask why you'd like your upstream exchange to be different from the downstream one? 

[uvzu...@gmail.com]

I am researching (experimenting with) a solution for data center failover, i.e. being able to consume messages from cluster B that were published to cluster A if cluster A dies. Since clustering does not work across data centers (across WAN), I envisioned the following that is not perfect but approximates the solution with a few caveats:

When messages are published to cluster A, they end up on the work queue in A via the exchange "work". When messages are published to cluster B, they end up on the work queue in B via the exchange also named "work". Exchanges and queues are named the same in both clusters because, ultimately, it's the same solution that is deployed several times for redundancy.

Then, I would federate "work" exchange from cluster A into "copy" exchange in cluster B. Note that I do not want to federate "work" in A to "work" in B because that would cause the messages processed twice -- once in A and second time in B. The "copy" exchange in B would have the "copy" queue bound to it. It will be a holding area for messages for when cluster A fails, in which case I would either start consumer to drain "copy" or shovel the messages from "copy" into "work". Since "copy" in B is not affected by consuming from "work" in A, it will keep accumulating messages, and, so, it would have high level message ttl set to expire messages that were presumably processed in A. I do understand that "copy" in B may not have all unprocessed messages as some may not have been yet delivered from A, or may have expired in B without being processed in A. "copy" in B may also have messages that were processed in A but not yet expired in B at the time of A dying, i.e. duplicates. I assume consumer is idempotent, so duplicates are ok. Some loss is assumed to be ok too -- better something than nothing in this case.

The same would apply to cluster B, i.e. "work" in B is replicated to "copy" in A via federation.

This is why I would like the upstream ("work") to be named differently from the downstream ("copy"). 

If upstream and downstream are named the same then I would have to have another exchange in A, say "workA", bound to "work" in A, have "workA" in A federated to "workA" in B, have "workB" in B bound to "work" in B, have "workB" in B federated to "workB" in A. Now, imagine I have one more data center... and one more ...

With upstream and downstream names matching each data center would have to have N+1 exchanges ("work", "work1", ... "workN"). With upstream name different from downstream, each data center would have exactly 2 exchanges: "work", meaning "my work", and "copy", meaning "other's work", and "copy" would be the downstream for multiple "work" upstreams.

I hope this made sense. I attached the picture of the two topologies.