Application Design using MassTransit

[Michael Hamrah]

I've been digging into MT and have some questions about service
design. It seems that the fundamental approach of MT is client-
centric. Meaning, the conversation goes:

1) "I'm a client listening for requests at this Queue, let me tell my
subscription manager about me"
2) "I'm a subscription manager- who are my clients and where are their
endpoints."
3) "I'm a publisher/server- I send messages but don't know/care who's
listening. If my subscription manager tells me about a client, I'll
send them a message"

The problem I have with this is I don't want to deal with a
subscription manager (side note- am I wrong for not wanting this? I'm
not really interested in directly linking pub/subs together- I want
distributed worker processing via competing consumer). I just want n
number of systems to publish messages to a common queue, and have x
number of consumers processing those messages. A publisher should
always send a message to a queue, and trust that something, at some
other point in time, will pick up the message and deal with it.

I don't feel I can easily do this with Mass Transit- but am I missing
something? My intention was to create a DummySubscriptionManager that
always publishes messages to a specified queue, but then I realized
the design of the bus was to Listen with queues (endpointToListenOn),
not publish to them directly. I know that DeadLetterQueueing is in
the works, but that's also not really an approach I want to take-
again, I don't care about multiple subscribers with single publishers-
I want more of an "event based" distributed system.

My other idea was to have my publisher directly send messages via an
endpoint, not the bus. Then use the bus/sagas/etc. to process
messages. Thoughts?

Thanks,

Mike

[rg]

I think you made a good point. From the system design perspective,
dynamic subscriptions aren't always necessary and in many scenarios we
could do without the subscription manager and configuration nuances it
brings. If dynamic subscriptions aren't needed we could replace them
with static routing - some configuration file or data structure
telling the message bus where to forward different message types.
Like:
MessageType1 -> Queue1
MessageType2 -> Queue1, Queue2
etc
Each endpoint would have its own 'routing table'. I'd be glad to have
such options, what is more it would allow us to configure many-to-many
Pub/Sub scenarios.
Rafal

[Chris Patterson]

Routing is going to be an option, but the issue is configuration. If you have to configure every single service and the routing options, that is a pain for operations.

The subscription service will soon be back to where it reads a list of "persistent subscriptions" from the database at startup. At this point, any services that come online would also get this directory of services added to their outbound pipeline. This is easier to configure at a single location (the database) rather than touching the configuration of each node in the system.

If you need to send to a single endpoint, you can always just GetEndpoint() from the EndpointFactory and call .Send() yourself. This makes your application a big more tightly coupled, but eliminates the need for the subscription client (and even the bus for that matter if you are just sending and not receiving messages). In this case, you're just getting a nice insulation from MSMQ and the API around it.

The persistent subscriptions should be operational in the trunk (and 0.6 final) within the next week or two if one of us gets time to finish it up and test it all.

-cp

Chris Patterson

chrisfr...@gmail.com

[Dru Sellers]

inline 

On Fri, Apr 17, 2009 at 11:13 AM, Michael Hamrah <mha...@gmail.com> wrote:

I've been digging into MT and have some questions about service
design.  It seems that the fundamental approach of MT is client-
centric.

endpoint centric - really everything is a client

 

 Meaning, the conversation goes:

1) "I'm a client listening for requests at this Queue, let me tell my
subscription manager about me"
2) "I'm a subscription manager- who are my clients and where are their
endpoints."
3) "I'm a publisher/server- I send messages but don't know/care who's
listening.  If my subscription manager tells me about a client, I'll
send them a message"

Yup

 

The problem I have with this is I don't want to deal with a
subscription manager (side note- am I wrong for not wanting this? I'm
not really interested in directly linking pub/subs together- I want
distributed worker processing via competing consumer).

Depends on what your mental view of this is. Are you really linking things together?

bus.Publish(msg);

The linking is done in the infrastructure. behind the scenes.

 

 I just want n
number of systems to publish messages to a common queue, and have x
number of consumers processing those messages.

the 'n' number of systems can call _bus.Publish(msg)

the 'x' number of systems can all subscribe via 'Consumes<MSG>.All'

 

A publisher should
always send a message to a queue, and trust that something, at some
other point in time, will pick up the message and deal with it. 

I don't feel I can easily do this with Mass Transit- but am I missing
something?  My intention was to create a DummySubscriptionManager that
always publishes messages to a specified queue,

Here I would (as chris said) look at IEndpoint

endpointFactory.GetEndpoint(uri).Send(msg)

then still use the Consumes<MSG>.All to wire up the other end.

 

but then I realized
the design of the bus was to Listen with queues (endpointToListenOn),
not publish to them directly.  I know that DeadLetterQueueing is in
the works,

not sure why you would want that outside its normal use

 

but that's also not really an approach I want to take-
again, I don't care about multiple subscribers with single publishers-
I want more of an  "event based" distributed system.

Sure, MT is very much built to be event based. not sure where you are going with the multiple subscribers / single publishers, MT supports multiple|single subscribers to multiple|single publishers.

 

My other idea was to have my publisher directly send messages via an
endpoint, not the bus.  Then use the bus/sagas/etc. to process
messages.  Thoughts?

This is an option as well, and one that works well. But do read Chris's email.

Hope this helps!!!

-d

 

Thanks,

Mike

[Michael Hamrah]

Thanks for all the responses- I keep enjoying MT the more I dig into
it. I have to say, the infrastructure is pretty slick!

I was able to get started with .EndPointFactory/Send() approach on one
side with the Consumes class on the other. At this point, I'm still
trying to understand the problems with distributed programming which
MT solves. I wasn't sure if just using the EndPointFactory/Send() was
the "right" way to do it. I can see how using a subscription service
will make the entire app a lot easier to manage (by consolidating
configuration/monitoring heartbeats/subscription management).

Looking at the samples, it seemed you always needed a consumer to
publish via the bus. This is what I meant by coupling- I wouldn't
want a producer to not send a message if no one was listening.
Producers should always publish messages into a queue (and if it
didn't know which queue, then maybe a dead letter queue). The
behavior of a publisher always sending a message is exactly what I got
with the EndpointFactory/Send(), and what I couldn't figure out how to
do with a subscription service. But I'll look into how
PersistentSubscriptions can solve that problem.

I was also confused on how the subscription service worked- I was
worried that the bus sent a copy of each published message to all
subscribers. Which it does- I was just confusing subscribers with
instances. The copy goes to each type of subscription, not each
running instance (for "competing consumers"). Now I'm seeing how a
subscription service can orchestrate that so well.

Thanks for helping me along!

Mike

On Apr 17, 7:33 pm, Dru Sellers <d...@drusellers.com> wrote:
> inline
>

[Dru Sellers]

On Fri, Apr 17, 2009 at 7:13 PM, Michael Hamrah <mha...@gmail.com> wrote:

Thanks for all the responses- I keep enjoying MT the more I dig into
it.  I have to say, the infrastructure is pretty slick!

I was able to get started with .EndPointFactory/Send() approach on one
side with the Consumes class on the other.  At this point, I'm still
trying to understand the problems with distributed programming which
MT solves.  I wasn't sure if just using the EndPointFactory/Send() was
the "right" way to do it.

It depends on what problem you are trying to solve.

It is certainly more coupled than the pub/sub example.

 

I can see how using a subscription service
will make the entire app a lot easier to manage (by consolidating
configuration/monitoring heartbeats/subscription management).

The subscription service does not consolidate all of this. each is a separate service.

 

Looking at the samples, it seemed you always needed a consumer to
publish via the bus.  This is what I meant by coupling- I wouldn't
want a producer to not send a message if no one was listening.

I wouldn't call that coupling. I would call it a dependancy. 

 

Producers should always publish messages into a queue

I would disagree here. If you do this, you would have just coupled the sender with the receiver. The whole point of pub/sub is to avoid this.

 

(and if it
didn't know which queue, then maybe a dead letter queue).

Sure, but what's the policy? Do you retry these? For how long? How many do you keep?

 

 The
behavior of a publisher always sending a message is exactly what I got
with the EndpointFactory/Send(),

It's not technically publishing, its just sending a message.

 

and what I couldn't figure out how to
do with a subscription service.

two different things as you have learned. :)

 

But I'll look into how
PersistentSubscriptions can solve that problem.

Its not. At least not in my definition of PersistentSubscriptions

 

I was also confused on how the subscription service worked- I was
worried that the bus sent a copy of each published message to all
subscribers.

It does. Each instance of the Consumes gets its own copy of the message. They are delivered one per queue, then the subscribers at that queue each get their very own copy of the message.

 Which it does- I was just confusing subscribers with
instances.  The copy goes to each type of subscription, not each
running instance (for "competing consumers").

In this case the competing consumers share one queue and each consumer in that instance will be delivered a copy of the message.

 

hope this helps, too!! :)

-d

[Dru Sellers]

FYI

http://codebetter.com/blogs/dru.sellers/archive/2009/04/18/masstransit-routing.aspx

-d

[Michael Hamrah]

Damn, thanks Dru! Now that's what I call service. And here I was
going to ask another question on what happens if the Barista goes to
the bathroom and never gets the NewOrder message from the Customer who
already paid the Cashier...

I guess it comes down to a design decision- do you want the caller to
handle a situation when the current sequence of events isn't correct
(no message was published because there was no subscribers) or do you
want the system to trust the message pipe for delivery (pushing a
message and having it sit in a queue)? I think the routing approach
is great- it certainly lowers the entrance barrier and can simplify a
system, and now I'm starting to understand the cost of that
simplicity.

Mike

On Apr 18, 7:30 pm, Dru Sellers <d...@drusellers.com> wrote:
> FYIhttp://codebetter.com/blogs/dru.sellers/archive/2009/04/18/masstransi...

> -d
>
> On Fri, Apr 17, 2009 at 8:29 PM, Dru Sellers <d...@drusellers.com> wrote:
>