MassTransit/RabbitMQ - how to minimize shovel setup steps when handling multiple message types?

[pasi...@georiot.com]

Our system has regional RabbitMQ servers in each of our hosted data centers (where we run our website/service/dashboard), and then a central RabbitMQ cluster (where we do our backend heavy lifting).  A couple queues currently exist in the regional deployments, each with a shovel defined to move the messages over to our central cluster for processing.  All of the actual message consumption happens in the central region.  Each of these queues that are currently setup with shovels, hold a single message type, and have a matching queue on the central cluster.  We are planning on adding many more message types, which worries me when it comes to setup/runtime overhead for many low volume queues/shovels.

Basically I want to minimize the overhead of adding new message types to the regional deployments, as this is something I see wanting to do frequently as we add new features to our dashboard, etc.  I was thinking of perhaps queuing multiple message types into a single queue (i know this isn't the best thing to do in general, for prioritization of messages, etc), and then having them shovel into an exchange on the backend cluster, where they could be fanned out appropriately.  Is there an established pattern for this type of work (these aren't high throughput messages, more like system configuration/account changes, etc), or should I really just stick with independent queues for each type of operation?  I feel like with the low volume of the messages, putting them all in one queue in each region won't be an issue (at least for some time).

If you have ideas/suggestions please share.  :)

Thanks,
Jesse

[pasi...@georiot.com]

Hi, I never got any feedback on this question, so I wanted to bring up a possible implementation.

To start over...

I have a bunch of events that I publish regionally, to a regional rabbitmq server.  The events aren't high throughput, and have namespaces that follow the form MyNamepace.MassTransit.Events.*.
I have a single queue per region that I want all the events to go into, "ha.masstransit_events"
I have a single shovel to publish move the events from the regional queue to my central processing queue, and publish them into the exchange matching the one "as published" (according to the rabbitmq shovel management page).
I have some apps on the backend, that have consumers looking for different types of messages that fall into the MyNamespace.MassTransit.Events.* types, and if messages of the types are published they receive/process them just fine.

My concern is that each new event type that I add, I'll have to go to each of my regional rabbitmq servers, and tie up a binding between its exchange (like   MyNamespace.MassTransit.Events.MyNewEvent) and the queue that gets shoveled.  While this can be done, it is extra setup logic, that i'd just assume avoid. 

Is there a way to tell masstransit to force a binding to a specific queue so when it creates the fanout exchange in the regional rabbitmq server that it maps it to the queue that is configured to be shoveled?
or
Is there a way to tell rabbitmq to look for all MyNamespace.MassTransit.Events.* exchanges and automatically bind them to the ha.masstransit_events queue?  (i tried using a wild card on that binding which isn't allowed as-is)
or
Do I need to write some code on my publishing side using the RabbitMQ client API to create these exchanges if they don't exist, and tie up the bindings on startup or something?

Any feedback/suggestions would be greatly appreciated.

Thanks!
- Jesse

[Chris Patterson]

That sounds particular to your specific use case for sure, but should be easy enough using the RabbitMQ extensions that we have available. If you look at how the RabbitMQ exchanges are bound when a message is published, you could do something similar but add an additional queue binding for your event prior to the first publish of each message type.

You'd want to add something like is shown here:

https://github.com/MassTransit/MassTransit/blob/develop/src/Transports/MassTransit.Transports.RabbitMq/PublishEndpointInterceptor.cs#L73

But in addition to binding the exchanges for the class inheritance chain for exchanges only, you'd want to add a specific binding for the event type -> shovel queue.

Similar to how the queue is bound here:

https://github.com/MassTransit/MassTransit/blob/develop/src/Transports/MassTransit.Transports.RabbitMq/RabbitMqConsumer.cs#L89

I hope that helps you out, not sure if it can be done by default with RabbitMQ directly (either via the client or the management API).

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/38cba735-64fb-408f-994a-65f79a7347f6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[pasi...@georiot.com]

Chris, thanks for your suggestion.  Is there an easy way to plug in some additional logic there in the MassTransit pipeline, or should I simply mimic what you're doing there in MT off on the side for the message types i know i publish (a bit more work to keep things in sync)?

It looks like if the receiving end of the shovel doesn't have the exchanges already defined (i.e., decide to start publishing an event in our dashboard before the consumer of that event has been deployed) the shovel ends up hanging and must be removed/readded after the missing exchange has been added.  Since there is a manual step to create the exchange on the cluster, or having to make sure that the consuming side has been deployed before the producing side, it might make sense to leave this as a manual step to create the exchange bindings (scripted of course, for easy rebuild), for now.  If the shovel issue is a bug that gets resolved, then I will look more into having the publishing app ensure the proper bindings are in place.

Question I posted about the Shovel behavior for reference:  https://groups.google.com/forum/#!topic/rabbitmq-discuss/v0AVjwlCiOg

Thanks again,
Jesse

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

[Chris Patterson]

There's no plug-in for the plug-in, you would just mimic what is being done in the existing one.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/b5f82518-88f4-4311-82f0-c4f23f06eef6%40googlegroups.com.