Filtering Downstream Federation Messages based on topic/routing key
104 views
Skip to first unread message
Eric Lalonde
Jun 11, 2015, 8:56:15 PM6/11/15
to rabbitm...@googlegroups.com
Hello,
We appreciate that in the examples it appears possible to federate exchanges based on a regular expression. My impression is that all messages sent to any routing key on that exchange are placed into the queue and sent downstream. Is it possible for downstream to specify a subset of routing keys to further filter which messages are sent downstream? This would be valuable in the context of topic exchanges, where only a subset of the topics on the exchange are interesting to downstream.
Since I cannot see a way to currently achieve this when downstream creates the federation policy, we end up filtering these messages based on the name of the exchange. Those messages that are not interesting to downstream A are sent to a different exchange that might be federated by a different cluster, even though the messages are of the same content. This is certainly feasible, but I find myself wishing for topic-based filtration in the context of federation, especially when we need to separate messages into different federated exchanges based on network topology, since this can get cumbersome as things scale.
Am I correct in my assessment that topic-based filtering (as performed by upstream) is not currently possible with exchange federation? If so, would it be possible to add support such that when downstream declares its policy, it can additionally express a topic-based (routing key) filtration of federated messages (e.g. statistics.$HOSTNAME.*)? The goal here is to avoid putting messages into the queue (and ultimately over the wire) in situations where there does not exist a downstream that has expressed interest in the topic or routing key associated with the message. This also avoids the unwieldy task of arbitrarily splitting messages onto different exchanges just to avoid sending messages to downstream clusters that are going to drop them on the receiving end.
Thanks,
- Eric
We appreciate that in the examples it appears possible to federate exchanges based on a regular expression. My impression is that all messages sent to any routing key on that exchange are placed into the queue and sent downstream. Is it possible for downstream to specify a subset of routing keys to further filter which messages are sent downstream? This would be valuable in the context of topic exchanges, where only a subset of the topics on the exchange are interesting to downstream.
Since I cannot see a way to currently achieve this when downstream creates the federation policy, we end up filtering these messages based on the name of the exchange. Those messages that are not interesting to downstream A are sent to a different exchange that might be federated by a different cluster, even though the messages are of the same content. This is certainly feasible, but I find myself wishing for topic-based filtration in the context of federation, especially when we need to separate messages into different federated exchanges based on network topology, since this can get cumbersome as things scale.
Am I correct in my assessment that topic-based filtering (as performed by upstream) is not currently possible with exchange federation? If so, would it be possible to add support such that when downstream declares its policy, it can additionally express a topic-based (routing key) filtration of federated messages (e.g. statistics.$HOSTNAME.*)? The goal here is to avoid putting messages into the queue (and ultimately over the wire) in situations where there does not exist a downstream that has expressed interest in the topic or routing key associated with the message. This also avoids the unwieldy task of arbitrarily splitting messages onto different exchanges just to avoid sending messages to downstream clusters that are going to drop them on the receiving end.
Thanks,
- Eric
Michael Klishin
Jun 12, 2015, 4:05:14 AM6/12/15
to Eric Lalonde, rabbitm...@googlegroups.com
can be useful. Please file an issue.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
Eric Lalonde
Jun 12, 2015, 4:13:52 AM6/12/15
to Michael Klishin, rabbitm...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages