RabbitMQ + Saga Implemenation design help.
Shibu Raj
We are trying to model a long running process (multiple individual
transactions) into Saga's using mass transit as the ServiceBus and
MSMQ or RabbitMQ as the middleware. RabbitMQ is the middleware of
choice.
Problem Domain : It is very similar to the Order Processing Saga
defined in the PDf doc given in the Masstransit project web site. Web
Application pullishes the initail message. Once the message is
receievd Transaction Tx1 through Txn needs to start. All these
transactions are sequential in nature one triggering other. If one
transaction say Tx3 didnt complete the system has to retry for set
number of times. If it cannot be succeded with the set number of
retries mark it for manaul review, else continue the processing.
In the above set up each and every Transaction (Tx) will start with a
message and end by publishing a new message corelated by the same
corelation id. For each of these messages there will be a class which
does the processing with in the windows service build using TopShelf.
The following are the design questions I have on this model.
1) Do we need to separate the processing of each of the messages for
Transaction (Tx1 -- Txn) in to separate windows service? I assume that
will be an over kill. So my design choice is to use one windows
service. But I would like to know the best practices around this?
2) What is the best way/choice to build redundancy for this Windows
Service Infrastructure who is processing the Saga? What are the
options available with in MassTransit or in the windows world to
support this?
3) I read a post in the forum that if we are using RabbitMQ then we
dont need to technically use the Distributor/Worker? If that is the
case what are the options for the load balncing the saga processing?
What are the possible options we have with in masstransit
infrastructure to process saga's using multiple boxes that way
providing needed redundancy?
5) After reading the forum; I assume if we are using RabbitMQ then we
dont need to use the Mass Transit Runtime? Is that true?
Shibu Raj
Travis Smith
> Transaction (Tx1 -- Txn) in to separate windows service? I assume that
> will be an over kill. So my design choice is to use one windows
> service. But I would like to know the best practices around this?
You can host as many logical services in a Topshelf physical service
as you want. On top of that, a since instance can subscribe to
multiple messages, making this easy to deal with as well.
However, if you are worried about scaling in the near future, it's
easy to spread out services on different machines if they are already
broken up. Completely up to you if that's a need now. I had like 21
services at one point, but we were to set to processing 40 million
messages a day across 9+ machines.
> 2) What is the best way/choice to build redundancy for this Windows
> Service Infrastructure who is processing the Saga? What are the
> options available with in MassTransit or in the windows world to
> support this?
What redundancy are you looking for? I've built "replay" buttons into
the system so you can re-publish one of the events that already
happened if there was a temporary error.
> 3) I read a post in the forum that if we are using RabbitMQ then we
> dont need to technically use the Distributor/Worker? If that is the
> case what are the options for the load balncing the saga processing?
> What are the possible options we have with in masstransit
> infrastructure to process saga's using multiple boxes that way
> providing needed redundancy?
Sagas might be a bit tricky to do in a load balanced scenario. I'm not
sure I have a good answer to this since every correlated message needs
to go to the right host. That means content based routing, which means
'meh'. And that's prone to failures if something changes. Now if just
the works who don't actually use a saga are load balanced and the
director service is a saga and isn't load balanced you might be fine
there.
RabbitMQ supports competing consumers. I would use that for load
balancing. No point in using the Distributor logic when the transport
will do it for you.
> 5) After reading the forum; I assume if we are using RabbitMQ then we
> dont need to use the Mass Transit Runtime? Is that true?
True. I can't promise the exchange binding is without some quirkiness
at the moment, but it should work pretty well and RabbitMQ has pretty
good tools to see what's going on now.
Shibu Raj
Thanks for the quick response. I guess I agree with the options you
gave for Points 1and 2. I guess i will go by the same route as you
suggested.
For Point 3 and 4 you said "Now if just the works who don't actually
through. I understand from your response above and some other posts in
the forum that content based routing is not a design choice. So I will
refrain from that. But will you elaborate on the above statement I
quoted.
I will look into the RabbitMQ competing consumers option for load
balancing.
For point 5 you mentioned " the exchange binding is without some
quirkiness at the moment." Can you throw some more light here, please.
Also I would like to mention that our first application build on top
masstransit is now in production. We are an insurance provider in
property and casualty area. We are just using the publish subscribe
model for the time being to connect our policy system to the agency
management system. Working well. Thanks for the great infrastructure
and support through this forum. Right now we are using MSMQ for
middleware. Please let me know if you like to know more information.
We would be happy to share our success story.
The new project in design needs to have Saga implementation. Do you
suggest should we use RabbitMQ or MSMQ with MassTransit 2.0 bits. With
the current state of the code base, what you suggest with the
intention to go production with in next 6 months.
Thanks again for the help
Shibu
Travis Smith
> use a saga are load balanced and the director service is a saga and
> isn't load balanced you might be fine there." I didn't get the point
> through. I understand from your response above and some other posts in
> the forum that content based routing is not a design choice. So I will
> refrain from that. But will you elaborate on the above statement I
> quoted.
Having a saga behind a load balancer could be troublesome, not all
messages are guaranteed to same end point. If it's a multi step saga,
this could cause issues. If you always round trip to the DB or
something to do work you might be okay though. This totally depends on
your saga though.
TS
|
TC--T1
| \
T2 T3
TS - Starts Message
TC - Coordinator or director
T1-3 - Workers for different messages
So if T1 - T3 needs a Saga and they don't just respond to Saga
messages from TC, then hiding T1-3 behind load balancing may be a
problem. You can always move T1-3 to different machines though.
Does this clarify?
> I will look into the RabbitMQ competing consumers option for load
> balancing.
>
> For point 5 you mentioned " the exchange binding is without some
> quirkiness at the moment." Can you throw some more light here, please.
I haven't played with the exchange binding much. My demos always have
the exchanges setup by hand and I've yet to take Rabbit to production.
It's more of a untested area than anything else currently. Our tests
seem to work okay but that's not the same as taking something to
production.
> Also I would like to mention that our first application build on top
> masstransit is now in production. We are an insurance provider in
> property and casualty area. We are just using the publish subscribe
> model for the time being to connect our policy system to the agency
> management system. Working well. Thanks for the great infrastructure
> and support through this forum. Right now we are using MSMQ for
> middleware. Please let me know if you like to know more information.
> We would be happy to share our success story.
This is great to hear. I think compiling some of these success stories
would be great for us. It's on the list, but not with a high priority
currently.
> The new project in design needs to have Saga implementation. Do you
> suggest should we use RabbitMQ or MSMQ with MassTransit 2.0 bits. With
> the current state of the code base, what you suggest with the
> intention to go production with in next 6 months.
If you are going through a full test cycle, I'd do RabbitMQ. It's a
better messaging transport hands down. The tooling isn't as advanced,
mostly since it hasn't been around as long, but what's there is pretty
good stuff. You can scale better with Rabbit as well, it has support
for things like federations. And the 2.0 bits are much easier to us
than the older versions of MT ;) There was a lot of clean up.
Do you have a thought around what your message volume is?
-Travis
Shibu Raj
On Aug 8, 10:20 pm, Travis Smith <tra...@legomaster.net> wrote:
> > For Point 3 and 4 you said "Now if just the works who don't actually
> > use a saga are load balanced and the director service is a saga and
> > isn't load balanced you might be fine there." I didn't get the point
> > through. I understand from your response above and some other posts in
> > the forum that content based routing is not a design choice. So I will
> > refrain from that. But will you elaborate on the above statement I
> > quoted.
>
> Having a saga behind a load balancer could be troublesome, not all
> messages are guaranteed to same end point. If it's a multi step saga,
> this could cause issues. If you always round trip to the DB or
> something to do work you might be okay though. This totally depends on
> your saga though.
>
> TS
> |
> TC--T1
> | \
> T2 T3
>
> TS - Starts Message
> TC - Coordinator or director
> T1-3 - Workers for different messages
>
> So if T1 - T3 needs a Saga and they don't just respond to Saga
> messages from TC, then hiding T1-3 behind load balancing may be a
> problem. You can always move T1-3 to different machines though.
>
> Does this clarify?
>
I dont think we need to have a load balanced approach for the problem
we have in hand now. The problem domain we have now is just one Saga
with sequential set of transactions one starting after the other
finishes. The transaction are mostly data crunching tasks as well.
None of the transactions start a new saga as well. So I guess if
required; as per your comment above; we can put the load balancer
approach in place. But it is not required. Also for scalability we can
deploy different message consumers in different machines for better
throughput as well.
> > I will look into the RabbitMQ competing consumers option for load
> > balancing.
>
> > For point 5 you mentioned " the exchange binding is without some
> > quirkiness at the moment." Can you throw some more light here, please.
>
> I haven't played with the exchange binding much. My demos always have
> the exchanges setup by hand and I've yet to take Rabbit to production.
> It's more of a untested area than anything else currently. Our tests
> seem to work okay but that's not the same as taking something to
> production.
>
Runtime act as the Subscription infrastructure which will internally
let other clients know about the messages they are interested in. In
this approach MT Runtime act as a single point of failure. But the
thing is the MT Runtime is required only during the bootstrap phase of
the workers/consumers. So the impact of MT Runtime on overall
infrastructure availability is very small. Also in the MSMQ based
implementation the messages are send/published to the local queue in
the same box. That makes things more durable and available for the
running application (in my opinion). Now taking this infrastructure
design to the RabbitMQ, it seems like RabbitMQ brokers pretty much act
like the Runtime. That's great because we have no more one extra thing
to deploy and configure. The broker can be clustered for more
processing power and we can get high availability using the heartbeat
implementation as described in the Rabbit Docs. But for the worker
(publisher/consumer) processes has to read and write from the remote
Rabbit Queues set up in the broker machines. By remote I mean
different machines. Is that something we should avoid? Having a need
to write to a remote queue residing in the broker machine opens up all
the issues we are trying to avoid using messaging. I am new to Rabbit?
Is there any concept like local queues in Rabbit. I checked the
RabbitQ doc couldn't find any reference to the same. Any pointers in
this area is much appreciated.
> > Also I would like to mention that our first application build on top
> > masstransit is now in production. We are an insurance provider in
> > property and casualty area. We are just using the publish subscribe
> > model for the time being to connect our policy system to the agency
> > management system. Working well. Thanks for the great infrastructure
> > and support through this forum. Right now we are using MSMQ for
> > middleware. Please let me know if you like to know more information.
> > We would be happy to share our success story.
>
> This is great to hear. I think compiling some of these success stories
> would be great for us. It's on the list, but not with a high priority
> currently.
>
> > The new project in design needs to have Saga implementation. Do you
> > suggest should we use RabbitMQ or MSMQ with MassTransit 2.0 bits. With
> > the current state of the code base, what you suggest with the
> > intention to go production with in next 6 months.
>
> If you are going through a full test cycle, I'd do RabbitMQ. It's a
> better messaging transport hands down. The tooling isn't as advanced,
> mostly since it hasn't been around as long, but what's there is pretty
> good stuff. You can scale better with Rabbit as well, it has support
> for things like federations. And the 2.0 bits are much easier to us
> than the older versions of MT ;) There was a lot of clean up.
>
of head winds then we may change to MQMQ. Thats the sole reason I am
asking so many design/infrastructure questions. We have some testing
time. Also I guess using RabbitQ will help us to scale better down the
line. I turn we can also help MT is testing the Rabbit Q / Transport
implementation.
> Do you have a thought around what your message volume is?
Start Saga Messages.
>
I have one another question Travis. I am new to Saga implementation so
bear with me here.
In MSMQ based pub/sub model we can have the MSMQ Multicast messaging
and no need to have the MT Runtime. (In the last project as I
explanined earlier we are using MT Runtime). In case of Sags with MSMQ
do we need MT Runtime to orchestrate the Saga?
Another Question is; If we choose to go via RabbitQ and later decided
to use MSMQ for whatever reason; I assume all we need to change the
transport logic/implementation and test the same. Is that true?
We talked about the RabbitQ based Load Balancing for Saga
Implementation. I am wondering if we use MSMQ, for the above described
sequential Saga can we do Distributor/Worker model with out any of the
issues described above. I am just putting down the question here, so
others who are reading this post will have a good frame of reference
to compare and contrast the two transports.
Thanks
> -Travis
Chris Patterson
--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/masstransit-discuss?hl=en.
Chris Patterson
> -Travis
Chris Patterson
> -Travis
Dru Sellers
On each machine running masstransit code, we install an instance of RabbitMQ on that server (requiring erlang and rabbitmq, both easy to install). This allows us to have local reads and writes (prevents network failures as well as being faster). Then we use RabbitMQ clustering (http://www.rabbitmq.com/clustering.html) to 'connect' all of the instances together.
I hope that added to the convo. :)
-d
Travis Smith
> distributor. It has been engineered specifically to allow this as part of
> the configuration.
Does that still work competing consumers on RabbitMQ? Without our
Distributor infrastructure, I wasn't aware this would work.
If so, I learned something new ;)
-Travis
Chris Patterson
-Travis
Shibu Raj
I have just couple of follow up questions on some info provided by
Chris. For the benefit of others, I am using the same forum question.
Point A : You mentioned "If your sagas are written properly, they can
specifically to allow this as part of the configuration."
1) Make each transaction as granular as possible.
2) Make sure each transaction have a separate message contract.
3) Make sure each message contract is serviced by a separate worker
process/service. Can be implemented under the same topshelf install as
described by Travis.
4) If require load balancing, we can either use Distributor/Worker
model or Competing Consumers model under MSMQ as well as RabbitMQ. As
per Chris, we should only use either one of the approach. Dont not mix
both.
Chris/Travis/Dru - Is there any other design aspects that needs to be
taken care of.
Point B :- Another point Chris mentioned is "You may also consider
behavior of each step behind a separate message contract and use the
saga to orchestrate the calls to those three services. That way, you
can scale the services separately from the actual saga. That can
simplify some load issues."
considerations described above. One point I want to put it to
consideration from the people who are experienced in Saga is "Do we
need to load balance the Saga Orchestrator process." . (Please note
that I am new to Saga and if it is a stupid questions, just let me
know). I understand we can load balance the worker process but not
sure how a Saga Orchestrator works in this scenario. Can we treat the
Saga Orchestrator just like any other worker process/consumer.
Point C:- For load balancing the general idea is to use the Worker/
Distributor or Competing Consumer. Which one is the preferred approach
for a) When we use MSMQ middle ware and b) When use Rabbit MQ?
Point D: You mentioned "The distributor uses sends to queues, instead
of pub/sub, so it should work with either RabbitMQ or MSMQ." Chris I
assume you are talking about instead of use the Bus.Publish() we need
to use the Send command the message. Please accept my ignorance on
Saga and distributor/worker process. I had a project in production
using pub/sub model. Just learning the Distributor/Worker model,
Competing Consumer and Saga.
Thanks Dru, Chris and Travis for such a great infrastructure.
Best
Shibu
Shibu Raj
Shibu