MassTransit Service Bus - Actual Real World Use Cases

[Mark Gillen]

Hi all,

My question is more in general terms but of the folks out there using MassTransit in production, what kinds of services are you using MassTransit for?  I mean the classic example of two web service calls, one using SSL, the other not, is an obvious example.  What I'm interested in is learning about the types of things that experienced users are hosting within a MassTransit environment.

My question is broad on purpose.  I'm not interested in code or design just on commentary about how you use MassTransit and for what kinds of things. 

I ask this since I want to have a better idea about partitioning - that is what kinds of application services do you use MassTransit for.  My own approach has been to factor out all those common services that I'd rather abstract away behind a service bus interface such as SFTP transfers to remote systems and command API's such as are used by SilverPop or Exact Target email systems.

Also, regarding the service bus itself:  How many subscriptions or publishers do you recommend hosting within a single service application?  Take the Starbucks coffee example for instance:  Clearly the Barista app is a good service candidate but would you wrap the cashier app into the same service executable? 


Thanks in advance to anyone that replies.

Mark Gillen

[Travis Smith]

So when I first started using MassTransit, we took a system that was all hosted in one executable and broke it into about 8-10 components. Some of the components were easy to identify how to break up - they connected to third parties or had different load characteristics so we knew we wanted to balance that part differently. Today, the same system has about 30 distinct services on the bus with around 50 message types. We have effectively broken down each possible step in the workflow into it's own service (though not perfectly, there's still a couple we could split). This allows us to monitor and manage each step in the workflow individually. We have ALMOST been able to bring up new parts of the workflow with zero code changes to existing ones (we've just had to add a field or two to some messages, so minimal, but not zero impact). 

How many subscriptions or publishers do you recommend hosting within a single service? As many as you're comfortable managing and deploying in a set. I have some services that are just 1 consumer and 2 publishes (result + operational data in separate messages). There can be multiple consumers and no publishers (for consuming all the operational data). Or any mix that makes sense.

In the Barista app I wouldn't wrap the Cashier app together because I see the Cashier as a different member and you could have multiple Cashiers (1-2x) vs. a different number of Barista (3?) vs. a whole bunch of customers (10). In that topology it's important to have them separate if you need to have differing numbers of them to support your load. 

Does this help? It would be awesome if we could adjust the documentation to answer as much of this as possible - not that I don't want to help on the list but it lower the bar of entry for more people since the mailing list is more effort than a lot of people are willing. So let me know what you are able to distill out of this as the most useful bits and maybe we can get into the docs in a way that helps someone else. 

-Travis

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/e0b7c33c-42ec-4e10-bab8-1412ba9b2507%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Gillen]

Travis,

Thanks it helps a lot.  Can you share with me an example or two of the types of components that you decomposed your application?  Nothing proprietary mind you.  I'm just curious about granularity. 

Where I'm headed is using this in a large data sync operation - I take membership info, and transfer it to a couple of external systems each with a different protocol (one XML based the other a .net service).  So any changes on our side reflect to these external systems.  The fly in the ointment is volume.  I may push a million member records over the wire to these systems, some realtime i.e. request-response, others in batch files that are processed in batch and I get the results.  Now the existing system that does this, is similar in evolution to the system you alluded to, namely a number of workflow functions rolled into ETL like processes (get the recs, create the file, send the file, check the file is there, launch the batch job...etc.etc.) So some of my partitioning is a natural function of what belongs together.  The reason I ask about what other folks are doing is that I like to see if something pops up that challenges my notion of what belongs where.  Sort of trying to get outside of a problem that I've solved before in case my thinking is just stuck in a rut and I don't see it - don't know what I don't know sort of thing.

BTW somewhere I noticed that MassTransit is being used at Pandora?  Is that accurate?  I work for Triton Digital and Pandora uses our streaming and streaming metrics - so naturally it begs the question what does Pandora use MassTransit for.  Again I sense a lot of power behind message driven architectures so I want to lay the foundation properly.

Thanks again, Travis.

--
You received this message because you are subscribed to a topic in the Google Groups "masstransit-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/masstransit-discuss/66UWIuJn1GM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAAqUTbk5X0TgV_%2BSGXovjitpocjGWaoCPMsuhbZbTmMZUYKuyQ%40mail.gmail.com.

[Travis Smith]

So maybe just a quick walk through of the workflow I was describing. It's around insurance eligibility. 

Messages enter via a web service -> validation/clean up service -> workflow owner service, checks to see what components need executed from rules then sends all of them out -> (just care about eligibility right now) -> build the outgoing EDI document -> load balanced connection to third party that responds with another EDI document -> data processor to transform EDI document into object representation -> data to load stuff into reporting tables -> responds to workflow owner service -> waits for all steps to be completed and sends out a message to the return service -> generates a response to the customer. 

A few of these services (connecting to a third party, transforming EDI response) are load balanced. Others have different configuration for threading/etc. All of them report metrics to the operations service. We have additional paths which are similarly executed at the same time from the workflow owner service. 

Now, if you already have an ETL process, splitting it may not make sense. What we've always ended up doing is unrolling batches and having a coordinator keep track of the batch group to report if the batch is complete. This lets everyone use the same infrastructure (we don't ETL) of a single transaction with just a couple different front ends. Owning both a batch and a real time instance of the same code gets old. 

=

So I know nothing about Pandora using MassTransit - but if you saw it on our website, Pandora is the WordPress theme we're using. I do know some banks use it. Novell uses it for some VM/cloud management stuff. Microsoft uses (or maybe used, the one guy involved isn't there any more) it for something internal. I've deployed it a few smaller organizations. It's in a few healthcare organizations. 

-Travis

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAEn6A5M7aEdw3PqiuMQZeO3c3uaFCBtScrMALVqq5jMwSsD22g%40mail.gmail.com.

[Mark Gillen]

Travis,

Perfect!  That's very helpful.  Yea the ETL processes specifically the creation of a file and the transferring the file are wrapped into the same ETL process...meaning each ETL step is in fact a workflow step.  What I've observed over time is that file creation works fine but the external system has an FTP issue hence workflow fails which being an ETL process causes an unnecessary amount of things to unwind just so the database stays nice and tidy and we can step through the process later.

My intention is to separate pieces of that ETL line so file creation uncoupled from file transfer and file transfer is uncoupled from launching the external process.  The motivation is due to the fact that the external process can be, at time, a bit unreliable, they go offline for maintenance (lousy at telling us that this event is about to happen) so fault tolerance and allowing process to thread along more naturally removing bottle necks.  So my create files processing does that and only that and informs subscribers that there are files ready to transfer, etc.

Thanks, again Travis.  I'm on the right track.

Regards,

Mark Gillen

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAAqUTbnzSVoWHXxS1kpXM4727WEdNhL0tYsOA59GOqkc1GYNhg%40mail.gmail.com.

[Mark Gillen]

Travis,

One follow up question:  How much heavy lifting (long running processing) does the service bus actually do?  Do you delegate to a job service or launch console apps or do you do the processing in threads pooled in the service bus itself.  My gut feel is to keep the service bus as responsive as possible so to limit the amount of heavy lifting in terms of processing is that pretty much your experience?  Also are there any best practice guidelines tips etc. around?

Thanks, Travis.

Regards,

Mark Gillen

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAAqUTbnzSVoWHXxS1kpXM4727WEdNhL0tYsOA59GOqkc1GYNhg%40mail.gmail.com.

[Travis Smith]

Mark, 

Great question. So anything that will be doing heavy lifting is isolated. When we communicate with third parties, it can be holding a thread for up to 2 minutes. So those processes are load balances and have a lot more threads assigned to them since they'll be blocked for a long time. The key is really just isolation. Having a consumer do heavy lifting isn't a problem. 

I know some people do video encoding which has be much longer. So if you have using RabbitMQ, you need to configure your ACK timeout to be longer than the longest you'll expect your process to take. If you're using MSMQ with multiminute consumers, do not use transactional queues. Even with extended timeouts you can end up with resource starvation, unless you never expect to process more than a handful at one time. 

-Travis

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAEn6A5Nt%2BfBF%3Dfxz9vU8aVqeiYX2QLUiydAnk6M78zrQEH_DQQ%40mail.gmail.com.

[Mark Gillen]

Travis,

Perfect!  Thanks a bunch.

Regards,

Mark G.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAAqUTbkO7iHNDgzqcpAX4KfgDwL24KLaap%2BZS%2BXNGp5vvFSBgg%40mail.gmail.com.

[Mark Gillen]

Travis,

Do you orchestrate your workflow with sagas or do you use more of pub-sub, request-response to keep processes pieces in lock step?

Thanks,

Mark Gillen

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAAqUTbnzSVoWHXxS1kpXM4727WEdNhL0tYsOA59GOqkc1GYNhg%40mail.gmail.com.

[Chris Patterson]

Mark,

We use MassTransit on RabbitMQ, as well as Automatonymous for sagas (stored using NHibernate on SQL Server), as well as MassTransit-Courier (for dynamic workflow), and MassTransit-Quartz (for message scheduling, typically via Automatonymous state machines).

I created the RapidTransit (github.com/masstransit/rapidtransit) project out of the complexity in setting up and hosting multiple service bus instances within the same application. We have six Windows services hosting between five all the way up to 50 bus instances, all coordinated through something akin to RapidTransit.

In our case, an entire distributed interoperability platform (focused on healthcare connectivity) is built using these components. 

While these building blocks are just ceremonial, and not indicative of the state machine and flows running on top of them, we are processing an extensive amount of traffic with super low latency (typically less than half a second end to end for connectivity flows - that include significant transformation and mapping logic) and guaranteed - fully trackable delivery with auditing down to the accuracy of DateTime. The platform supports full message recovery and retry, automatically, to ensure fault tolerant operation in a 24x7 operation.

Does that help?

On Thu, May 8, 2014 at 10:31 AM, Mark Gillen <markg....@gmail.com> wrote:

--

[Mark Gillen]

Chris,

Yes, that helps tremendously!  As I'm building out my first services I'm keeping an eye out for what makes logical sense and in what service.  I think part of my problem has been thinking about "a service bus" instead of "service buses" which is something I've arrived at independently. 

Thanks for the feedback - this is one great piece of software, Chris. 

Regards,

Mark Gillen

--
You received this message because you are subscribed to a topic in the Google Groups "masstransit-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/masstransit-discuss/66UWIuJn1GM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAHX4WYf1otyQoar3Jv0emTC7upfhm04Dqg3%2BjsVe3uhsr8xa%3DQ%40mail.gmail.com.