Single bus with multiple hosts (MT3)
1,301 views
Skip to first unread message
Tommy Jakobsen
May 6, 2015, 6:39:42 AM5/6/15
to masstrans...@googlegroups.com
Hi,
I've been following MT3 for some time, and I must say that you did a great job! I would like to thank you for that :-)
Looking into upgrading our applications, I've configured MT3 with two hosts:
Bus.Factory.CreateUsingRabbitMq(busCfg =>
{
busCfg.UseLog4Net();
busCfg.UseJsonSerializer();
var domainHost = busCfg.Host(domainHostAddress, hostCfg =>
{
hostCfg.Username("");
hostCfg.Password("");
});
var dmzHost = busCfg.Host(dmzHostAddress, hostCfg =>
{
hostCfg.Username("");
hostCfg.Password("");
});
busCfg.ReceiveEndpoint(dmzHost, "commands", endpointCfg =>
{
endpointCfg.PrefetchCount = 4;
endpointCfg.Consumer<TestConsumer>();
});
busCfg.ReceiveEndpoint(domainHost, "domain", endpointCfg =>
{
endpointCfg.PrefetchCount = 4;
endpointCfg.Consumer<TestConsumer>();
});
});
If I start the bus and do a simple publish like this:
await bus.Publish(new PingMessage { Prop = "Ping test" });
It looks like the message is published using the host that was first registered in the factory. Is this correct?
Is there a way to select what host to use, or should I create two different buses, one with each host?
When publishing messages, a temporary queue seems to be created and automatically deleted. Is it possible to use the same durable queue as the receiving endpoint, or is this new approch more preferable, and why?
I've been following MT3 for some time, and I must say that you did a great job! I would like to thank you for that :-)
Looking into upgrading our applications, I've configured MT3 with two hosts:
Bus.Factory.CreateUsingRabbitMq(busCfg =>
{
busCfg.UseLog4Net();
busCfg.UseJsonSerializer();
var domainHost = busCfg.Host(domainHostAddress, hostCfg =>
{
hostCfg.Username("");
hostCfg.Password("");
});
var dmzHost = busCfg.Host(dmzHostAddress, hostCfg =>
{
hostCfg.Username("");
hostCfg.Password("");
});
busCfg.ReceiveEndpoint(dmzHost, "commands", endpointCfg =>
{
endpointCfg.PrefetchCount = 4;
endpointCfg.Consumer<TestConsumer>();
});
busCfg.ReceiveEndpoint(domainHost, "domain", endpointCfg =>
{
endpointCfg.PrefetchCount = 4;
endpointCfg.Consumer<TestConsumer>();
});
});
If I start the bus and do a simple publish like this:
await bus.Publish(new PingMessage { Prop = "Ping test" });
It looks like the message is published using the host that was first registered in the factory. Is this correct?
Is there a way to select what host to use, or should I create two different buses, one with each host?
When publishing messages, a temporary queue seems to be created and automatically deleted. Is it possible to use the same durable queue as the receiving endpoint, or is this new approch more preferable, and why?
Chris Patterson
May 6, 2015, 10:27:57 AM5/6/15
to masstrans...@googlegroups.com
First, thank you for the feedback on the changes in MT3. It's been a long labor of love (jeez, when did I start?) but it's coming together nicely.
You raise a good point about multiple hosts and publishing events. Publish is confined to a single host (and in this case, the default bus host) for the reason that while it may be useful to have multiple receiving hosts responding to events from multiple virtual hosts, in those cases, the consumers would publish events back into the host from which the message was received (the ConsumeContext interface has all of the publish/send methods).
In the case of just publishing from the IBus, that is indeed tied to the first host that is registered. In this situation, using the IBus to publish would be akin to an application using it as an adapter to publish events into a host, or using the IBus to send requests to an endpoint, and wait the response (using the new RequestClient).
If you indeed want to publish to both (or perhaps all) of the hosts in the bus, I would need to think about how that best makes sense since it's not really part of my original intentions for supporting multiple hosts. Hmm.
Thoughts?
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/e1652057-d504-48e8-89f4-935be09e5f97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Tommy Jakobsen
May 6, 2015, 3:24:52 PM5/6/15
to masstrans...@googlegroups.com
You sure did put a lot of great work into MT3 and for that we're grateful. Our company has donated to open source projects that we use before, and I would like your project to be next, if there's any way of doing so?
Let me try and explain why we have two hosts, and why we need to be able to specify what host we need to publish messages to.
We have a "Worker Role" application utilizing the application domain through application services etc.. Domain Driven Design inspired. The Worker Role subscribes to queue Q1 on server A (the first host) placed in a perimeter network. In the same zone we have a public REST API, that publishes commands to the RabbitMQ exchange on server A, queuing up commands in Q1.
The Worker Role is placed in the intranet zone, with the ability to communicate with the RabbitMQ service on server A. It consumes the commands from Q1 as they tick in. But instead of publishing messages back to server A, they should be published to a different host inside the intranet zone. Let's say a RabbitMQ instance on server B (the second host). The worker role also consumes messages from this instance (Q2), hence it acts like a "coordination service", activating different parts of the domain, sagas etc. Also other applications in the intranet zone subscribes to messages on B. Does it make any sense? It's definitely possible, that these messages will be published to both A and B in the future, but right now it's only to B.
With MT3, we can have a single IBus, consuming messages from both A-Q1 and B-Q2, but we can only publish messages to the RabbitMQ exchange on A. We could create two different busses, one for each host and control what bus to publish to, by injecting the bus into our consumers. We did that with MT2 and it works, but it doesn't feel right with MT3.
I actually like the current way MT3 handles it, because I can't see how one should be able to "select" what bus to publish to. Maybe it would make sense to make a configuration that tells the bus to publish to all hosts. But the consume context repesents a message from a specific host, and publishing using that context, it makes sense that it's only published to that host. The way it works now.
Maybe you should be able to "resolve" named hosts from the consume context if you need to publish to a specific host? It seems clumsy though... Maybe having different busses for this scenario is the best solution?
Let me try and explain why we have two hosts, and why we need to be able to specify what host we need to publish messages to.
We have a "Worker Role" application utilizing the application domain through application services etc.. Domain Driven Design inspired. The Worker Role subscribes to queue Q1 on server A (the first host) placed in a perimeter network. In the same zone we have a public REST API, that publishes commands to the RabbitMQ exchange on server A, queuing up commands in Q1.
The Worker Role is placed in the intranet zone, with the ability to communicate with the RabbitMQ service on server A. It consumes the commands from Q1 as they tick in. But instead of publishing messages back to server A, they should be published to a different host inside the intranet zone. Let's say a RabbitMQ instance on server B (the second host). The worker role also consumes messages from this instance (Q2), hence it acts like a "coordination service", activating different parts of the domain, sagas etc. Also other applications in the intranet zone subscribes to messages on B. Does it make any sense? It's definitely possible, that these messages will be published to both A and B in the future, but right now it's only to B.
With MT3, we can have a single IBus, consuming messages from both A-Q1 and B-Q2, but we can only publish messages to the RabbitMQ exchange on A. We could create two different busses, one for each host and control what bus to publish to, by injecting the bus into our consumers. We did that with MT2 and it works, but it doesn't feel right with MT3.
I actually like the current way MT3 handles it, because I can't see how one should be able to "select" what bus to publish to. Maybe it would make sense to make a configuration that tells the bus to publish to all hosts. But the consume context repesents a message from a specific host, and publishing using that context, it makes sense that it's only published to that host. The way it works now.
Maybe you should be able to "resolve" named hosts from the consume context if you need to publish to a specific host? It seems clumsy though... Maybe having different busses for this scenario is the best solution?
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
Chris Patterson
May 6, 2015, 4:40:18 PM5/6/15
to masstrans...@googlegroups.com
I think in the case you mentioned, having two IBus instances, one for each publish context, makes sense. In your case, to make it understandable, you might want to create some domain-level interfaces that delegate to the appropriate bus instances rather than having multiple IBus interfaces registered in your application and trying to keep them differentiated.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/e1652057-d504-48e8-89f4-935be09e5f97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/15d599db-7a05-402f-b681-0f423423d05d%40googlegroups.com.
Chris Patterson
May 6, 2015, 4:45:55 PM5/6/15
to masstrans...@googlegroups.com
So, if you publish using the ConsumeContext reading from queues on host A, those published events will go back to the source host (A). If you publish using IBus, they will go to the first host registered.
So...
You could specify host B first, and create your receive endpoint on host A. To publish on A, use ConsumeContext, to publish on host B, use IBus.
Just document the hell out of the behavior :)
On Wed, May 6, 2015 at 12:24 PM, Tommy Jakobsen <to...@holmjakobsen.dk> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/e1652057-d504-48e8-89f4-935be09e5f97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/15d599db-7a05-402f-b681-0f423423d05d%40googlegroups.com.
Tommy Jakobsen
May 7, 2015, 1:40:06 AM5/7/15
to masstrans...@googlegroups.com
Something like that yes. Domain level interfaces to differentiate between them is a good idea.
Thanks for your thoughts Chris :-)
My other question regarding MT3 behaviour when publishing messages. A temporary queue seems to be created and automatically deleted. Is it possible to use the same durable queue as the receiving endpoint (problem if there's more than one?), or is this new approch more preferable, and why?
Thanks for your thoughts Chris :-)
My other question regarding MT3 behaviour when publishing messages. A temporary queue seems to be created and automatically deleted. Is it possible to use the same durable queue as the receiving endpoint (problem if there's more than one?), or is this new approch more preferable, and why?
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/e1652057-d504-48e8-89f4-935be09e5f97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
Chris Patterson
May 7, 2015, 10:17:25 AM5/7/15
to masstrans...@googlegroups.com
The reason for the temporary queue (exclusive/auto-delete) is to make it possible for the bus to be agnostic to any reception. It seems like in every service we build, we end up with a "management queue" that is configured with the host name and process name to ensure it's unique, and then it uses that for cache update/configuration events that are published by other applications to which the service needs to reconfigure. So, I just made it officially part of the bus. It's also used for request/response where the response is meaningless if the process dies anyway, so they just go away.
The testing I need to confirm is that a response to a no longer existing queue/exchange doesn't re-create the exchange and/or queue, nor does it choke and die because the exchange isn't present.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/e1652057-d504-48e8-89f4-935be09e5f97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/15d599db-7a05-402f-b681-0f423423d05d%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/0a41f4f2-882f-44e8-94da-4a1967656408%40googlegroups.com.
Tim Thompson
Mar 2, 2016, 2:33:44 AM3/2/16
to masstransit-discuss
G'day Chris,
Sorry for the thread resurrection here but
>I would need to think about how that best makes sense since it's not really part of my original intentions for supporting multiple hosts.
What _was_ your original intention for supporting multiple hosts??
We're considering supporting multiple independent Rabbit brokers as a means of gaining robustness without going down the RabbitMQ-Cluster-NetworkPartitioning path of pain... effectively having redundant consumers on each broker, and building some sort of "publish to this host unless that fails, when you should fallback to this other host" code. Knowing whether to look into your multiple hosts scenraio or just roll with two separate bus instances would be handy.
Cheers,
Tim
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
César
Aug 2, 2018, 6:52:04 PM8/2/18
to masstransit-discuss
Sorry for another resurrection of this thread, but I'm currently facing this exact scenario and would like to know how would it be dealt with.
At least a basic skeleton with what should be put in place.
Thanks.
At least a basic skeleton with what should be put in place.
Thanks.
Eric Piraux
Aug 3, 2018, 1:26:57 AM8/3/18
to masstrans...@googlegroups.com
Hi,
If your're using RabbitMQ as transport, the Shovel plugin is what you need : https://www.rabbitmq.com/shovel.html
The idea is that the responsibility to move messages from one "bus" to another is delegated to the messaging system.
Eric
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/a23c96ac-29d8-47d6-85a9-4e5e34fdc988%40googlegroups.com.
César Demicheli
Aug 3, 2018, 6:06:33 AM8/3/18
to masstrans...@googlegroups.com
My scenario is like this: I consume from an intranet broker messages that are received through a queue, then my consumer, after processing those messages, has to publish the same content plus some extra data to another broker that will be exposed on the DMZ for internet. So it will not be just a plain forwarding, because there will be extra data added to the new mesages.
Will this scenario be supported by shovels as well?
Thanks!
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CALao87HTxRow8cvb7ao1egF7F_PNQ_aXcs78kJhW13XqZbMTNQ%40mail.gmail.com.
Eric Piraux
Aug 3, 2018, 7:37:59 AM8/3/18
to masstrans...@googlegroups.com
Yes.
Your MT process, after enrich extra data, publishes on an exchange on the intranet broker, then shovel moves the message to the broker in the DMW
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAKUeuqJFKkAn78-Rrpj7hM2BuJy1EzPMv%2BO849Tj0WbRB92miQ%40mail.gmail.com.
César Demicheli
Aug 3, 2018, 7:42:18 AM8/3/18
to masstrans...@googlegroups.com
Hmm, that sounds very interesting, I'll try it out!! Thanks a lot!!
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CALao87GziKwoq816_zAjefE8iNnz32inaU5tdzo2mHX%2BuxO65g%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages