Priority Queues

[JakeH]

Has anyone tried to tackle the issue of needing to support different
message priorities within Mass Transit? Ideally this would be achieved
by having one service bus pull from 3 different queues for each
priority. However since the mass transit bus is designed to only have
1 endpoint I'm looking for any advice on how else to achieve this.
Possibly this could be done using some form of indirection or using
RabbitMQ's routing system?

How difficult would it be to create a custom inbound endpoint/
transport that read from multiple queues? Possibly passing in a single
endpoint uri such as "msmq://localhost/foo_[high|normal|low]" where
the [high|normal|low] would be parsed and used to pull from foo_high,
foo_normal and foo_low in order of priority.

Thanks in advance
- Jake

[Dru Sellers]

Before you start doing this I would suggest listening to:

http://www.udidahan.com/2008/01/30/podcast-message-priority-you-arent-gonna-need-it/

and read:

http://tech.groups.yahoo.com/group/nservicebus/message/787

TL;DR:

you aren't going to need priority queues. check out the links for more info.

-d

-d

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/masstransit-discuss?hl=en.

[JakeH]

Thanks for the links Dru. Unfortunately Udi's podcast is no longer
available to download.

I think in my case though it makes sense to use prioritization. I have
one consumer responding to an event that will cause two other command
messages to be sent. Both of these commands cause data to be synced to
another database. One command will execute quickly and is more time
critical. The other will take a bit longer to execute and it doesn't
really matter as much when it happens as long as it eventually does
happen.

I've seen some discussion about using two processes with their own bus
to handle the different priorities. I'm not sure how this helps. Sure
i could have one process dedicated with more hardware resources than
the other but that doesn't really solve anything. It would neither
insure that high priority messages get executed first nor would it be
an efficient use of hardware resources since when there are no high
priority messages waiting to be processed then the bulk of the
hardware resources would be sitting idle.

As how to go about implementing this within mass transit. I'm starting
to think that using the distributor is the way to go. There would be a
IPrioritizedMessage interface that the distributor would look for when
handling messages. Does this sound like the best way to go?

Another option that i am considering (and maybe this is what people
meant about using separate service busses) is to have one service bus
setup as the router. There would be an additional 2 busses, one for
low and the other for high priority. The router would route messages
to the proper bus. Each time it routes a high priority message to the
high priority bus it would send a control message to the low priority
bus to stop processing messages. Once the high priority bus is done
processing all messages it would send a control message back to the
router which would then send a control message to the low priority bus
to start processing messages. This seems like a pain in the ass and
over architected solution but im just throwing ideas out at this
point.

On Sep 10, 5:26 pm, Dru Sellers <d...@drusellers.com> wrote:
> Before you start doing this I would suggest listening to:http://www.udidahan.com/2008/01/30/podcast-message-priority-you-arent...

>
> and read:http://tech.groups.yahoo.com/group/nservicebus/message/787
>
> TL;DR:
> you aren't going to need priority queues. check out the links for more info.
>
> -d
>
> -d
>
>
>
>
>
>
>
> On Saturday, September 10, 2011 at 7:23 PM, JakeH wrote:
> > Has anyone tried to tackle the issue of needing to support different
> > message priorities within Mass Transit? Ideally this would be achieved
> > by having one service bus pull from 3 different queues for each
> > priority. However since the mass transit bus is designed to only have
> > 1 endpoint I'm looking for any advice on how else to achieve this.
> > Possibly this could be done using some form of indirection or using
> > RabbitMQ's routing system?
>
> > How difficult would it be to create a custom inbound endpoint/
> > transport that read from multiple queues? Possibly passing in a single
> > endpoint uri such as "msmq://localhost/foo_[high|normal|low]" where
> > the [high|normal|low] would be parsed and used to pull from foo_high,
> > foo_normal and foo_low in order of priority.
>
> > Thanks in advance
> > - Jake
>
> > --
> > You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

> > To post to this group, send email to masstrans...@googlegroups.com (mailto:masstrans...@googlegroups.com).
> > To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com (mailto:masstransit-dis...@googlegroups.com).

[Dru Sellers]

A full response in the morning.

-d

> To post to this group, send email to masstrans...@googlegroups.com.
> To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.

[Dru Sellers]

so we have two messages (1 that is high priority 'A' and one that is low priority 'B')

you set up one bus instance in one windows service (will call this one 'JohnnyRocket') and have it subscribe to message A.

You set up another bus instance in one windows service (we can call this one 'HerpDerp') and have it subscribe to message B.

The JohnnyRocket instance can listen at 'msmq://localhost/johnnyrocket' and the HerpDerp can listen at 'msmq://localhost/herpderp'

Now assuming that message A is not often occurring as soon as one comes in the process can pick it up and process it with out having to wait for any B messages. So 90% of the time the JohnnyRocket queue will be empty and will just sit there waiting for the next message.

The other service 'HerpDerp' will have a queue that is usually full of B messages and can just sit there and chew on them until the cows come home.

If B messages start to overwhelm HerpDerp then we can easily move that service to its own box with the same or better hardware. Now since it doesn't have to share the load of HerpDerp JohnnyRocket should be able to keep up with the A payloads.


Does that make sense?
Does that help?

-d

To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.

[JakeH]

Dru, Thank you for your response. Your example makes sense but is it
the most efficient use of server resources? If B messages start to
become too great they need to be moved to another box in order to make
sure that the A messages will execute quickly. But when there are no A
messages you just have a server sitting their idle while when it could
be used for B messages.

Perhaps most importantly, is that all of these messages will still be
hitting one SQL server. I would rather the sql server not get
inundated with requests by both A and B messages at the same time,
especially during peak usage hours. I could use a scheduler to only
process B messages during off-peak hours but thats too much of an all
or nothing approach. It would be nice if the B messages were processed
in a timely manor - but only if its not going to bog the SQL server
down.

Thanks,
Jake

[Dru Sellers]

inline

On Mon, Sep 12, 2011 at 11:15 PM, JakeH <jake.h...@gmail.com> wrote:

Dru, Thank you for your response. Your example makes sense but is it
the most efficient use of server resources? If B messages start to
become too great they need to be moved to another box in order to make
sure that the A messages will execute quickly. But when there are no A
messages you just have a server sitting their idle while when it could
be used for B messages.

Is this a real problem or an imaginary one? Likely the business will be willing to tolerate some level of server inefficiency in order to guarantee a certain message SLA.
 

Perhaps most importantly, is that all of these messages will still be
hitting one SQL server. I would rather the sql server not get
inundated with requests by both A and B messages at the same time,
especially during peak usage hours. I could use a scheduler to only
process B messages during off-peak hours but thats too much of an all
or nothing approach. It would be nice if the B messages were processed
in a timely manor - but only if its not going to bog the SQL server
down.

This is more a problem out side of MT, around scalable database design. Of course if both process are going to be locking the same rows that this is going to be an issue. So we just have to design it so that they don't.
:)
 

[JakeH]

I know that MassTransit (and the other ESBs) are designed to be
"Enterprise" Service Buses. However I am not developing something at
an enterprise level. Its just a startup website running on EC2 and I
would rather not pay for more computing power than I need. It seems to
me that all arguments for not utilizing priorities within an ESB
result in additional server resources being needed. Not all
organizations (or individuals) have unlimited resources at their
disposal.

Part of your point seems to be that the need for message priority
indicates poor design. That the lack of message priority within the
ESB is good because it will force you to design things in a better
way. I agree that someone could get lazy and misuse the functionality
but I don't agree that it serves no purpose. Priority messaging is
simply a more efficient and less complicated solution in some cases
and I believe its worth having the option available.

I think we may be at a bit of an impasse on this topic. Thank you for
taking time to give me your input.

- Jake

On Sep 13, 5:41 am, Dru Sellers <d...@drusellers.com> wrote:
> inline
>

[Travis Smith]

Not that I like the solution but you can use the Distributor to do
content based routing. You can send lower priority messages to
queue_low and just have the consumer going with 1x threads. Where as
the high priority queue, queue_high, could have 5x threads, or
whatever is reasonable for your need. The distributor can throttle
messages as well; this might be a better fit, low priority messages
just don't get an endpoint until condition x is met.

Additionally, you could write an Accept<TMessage> method that kept
track of all the messages it saw and only accepted the lower priority
messages when it saw it got back to the start of the queue without
accepting a message. This could get nasty if you have multiple
consuming threads because of locking issues, but it's not
insurmountable at all.

The entire pipeline for MT is pretty plugable. It's not always the
easiest thing to do, but you could inject some magic in there if you'd
like as well.

So we don't have the full context of what you're trying to do, but
having separate consumers does not mean you're throwing more hardware
at it.

-Travis

[Dru Sellers]

Ah! A new wrinkle has been uncovered. You are hoping to leverage an EC2 instance and each hour of uptime == money spent. This information was not yet made available to us.

When working in these systems one of the issues we have run into is that we have to know a lot more about your network and server environment to make all of this happen and understandable. :)

I am not trying to be a dick when I ask questions like "Is this a real problem" (although I often fail at that) but rather I am trying to understand the complex scenario that you are living and working in.

That said, having two instance (two windows services) should let you get the same effects of a prioritized queue. It will let you switch out queue vendors more easily too (they don't all support the concept).

You are correct in interpreting that I think needing message priority is a smell of a bad design.

I am aware that not all organizations or individuals have unlimited resources. But code runs pretty fast now a days and a small server can get quite a bit of work done. At my last job when we went virtual the VMWare tooling shut down over 1/2 of our servers because it didn't need them much to our shock. So I wouldn't worry about the performance issue unless you have actual locking issues or CPU loads killing you. To help save costs you can also help us get MT fully mono complaint so that you can pay even less money and run on *nix. :) [i think we are actually pretty close]

Will the current contributors to MT give you access to Message Priority? Probably not, the two main contributors of the project (at this point) feel that this kind of feature is a bad feature. Chris and I are always up for discussions and welcome a separate thread in the list on message priority - unless someone is going to build it and support it, it most likely won't get done.

Even with our thoughts on message priority being at an impasse, I am still very interested in discussing how we can help you achieve your goals without message priority. Have solid patterns for deploying to EC2 like resources are good things to have and talk about.

-d

[JakeH]

Thanks Dru and Travis. I really appreciate both of your input.

Let me describe my solution a bit so that you have more context. The
type of product I am creating has similar design challenges as Twitter
does. Even though I'm not building a twitter clone - lets just assume
that i am for simplicity. With Twitter when someone posts a tweet it
needs to get propagated to the user's follower's feeds. If they have
20K followers than thats 20K feeds that need to be updated. In my case
I use Mongo DB as the de-normalized db for storing each individual
user's feed.

So if 20K feeds now need to be updated in response to the tweet posted
event then the event consumer will grab the list of followers and send
a "AddTweetToUserFeed" command for each follower. Each command, when
processed will determine if that item should actually be added to the
feed or not (twitter doesn't actually need to do this but my solution
does). So while each of the 20K commands aren't incredibly expensive -
they are still more expensive then just updating a record within
Mongo.

Pretty much everything else would be normal priority and i would want
that to take precedence. I would like the feeds to be updated ASAP but
also don't want them to clog up the system. If i was just to have a
consumer service using only a single thread than that would cause it
to always perform at a slow speed - even when the system isn't being
heavily used.

Travis I like the idea of the Distributor throttling messages. This
would actually be even better because low priority items could still
be continuously processed (at a lower throttled rate) while other
higher priority messages are also being processed. Low priority items
can be throttled up when there is more available bandwidth. Because
this is being built on EC2 there is even the possibility of utilizing
the Amazon API to take notice that certain server instances are being
under/over utilized and could send control messages to the distributor
to throttle up or down - regardless of message priority.

I still need to take a deep look at the distributor. I'm lone wolf on
the product that I building at the moment so much of this is just me
preparing for what may need to happen.

"The distributor can throttle messages as well; this might be a better
fit, low priority messages just don't get an endpoint until condition

x is met. " - How much of this functionality is already in place?

Thanks,
Jake

> ...
>
> read more »

[Travis Smith]

> "The distributor can throttle messages as well; this might be a better
> fit, low priority messages just don't get an endpoint until condition
> x is met. " - How much of this functionality is already in place?

Implement IWorkerSelectionStrategy,
https://github.com/MassTransit/MassTransit/blob/599a4c1357100d84d430dda5c559321333603a72/src/MassTransit/Distributor/IWorkerSelectionStrategy.cs.

Return no end points if the message is to be throttled.

https://github.com/MassTransit/MassTransit/blob/599a4c1357100d84d430dda5c559321333603a72/src/MassTransit/Distributor/DefaultWorkerSelectionStrategy.cs

Is the default one.

There is a distributor sample in the zip download or if you build from
source. We don't include samples in the nuget packages, right?

-Travis

[JakeH]

This should do the trick. Thank You!

On Sep 14, 3:53 pm, Travis Smith <tra...@legomaster.net> wrote:
> > "The distributor can throttle messages as well; this might be a better
> > fit, low priority messages just don't get an endpoint until condition
> > x is met. " - How much of this functionality is already in place?
>

> Implement IWorkerSelectionStrategy,https://github.com/MassTransit/MassTransit/blob/599a4c1357100d84d430d....

>
> Return no end points if the message is to be throttled.
>

> https://github.com/MassTransit/MassTransit/blob/599a4c1357100d84d430d...