live throttling of consumers possible?

[kreiggers]

I have a consumer that has a dependency on an external service that rate-limits calls. If I let MT loose it runs right into this limit and requests start to fail. Is there a way to throttle the number of concurrent consumers with a running bus instance? Is there a way of stopping consumers? (i.e. pause consuming from the queue?)

Thanks

[Stuart Dunkeld]

You can use sbc.SetConcurrentConsumerLimit(limit) in your service bus configuration to restrict the number of concurrent consumers. Setting a limit of one would mean one consumer processing messages sequentially. I believe the default number of concurrent consumers is related to the number of cores on the host server.

That won't solve your problem if one consumer still hits the limit..

Regards

On Thu, Jul 5, 2012 at 11:50 PM, kreiggers <lucky...@gmail.com> wrote:

I have a consumer that has a dependency on an external service that rate-limits calls. If I let MT loose it runs right into this limit and requests start to fail. Is there a way to throttle the number of concurrent consumers with a running bus instance? Is there a way of stopping consumers? (i.e. pause consuming from the queue?)

Thanks

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/RkMCOkIwyIIJ.
To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/masstransit-discuss?hl=en.

[Travis Smith]

If you want to rate limit in a more complex way, take a look at the distributor. This is similar to case for which I wrote. 

-Travis

[Craig Beck]

Yes I can use that but was hoping to find a more dynamic way of doing it. The problem is I don't know how many consumers I can use. The 90% case I can go as fast as I can, (not every message will cause a hit to the external service as we have some caching) but that won't work if there's a lot of hits. 

Thinking that I may have to put another layer of messaging in between these two cases and have the first layer use the max consumers and have a gateway to the external service run with only a couple of consumers. 

----

Craig Beck

[Chris Patterson]

You could really screw up the threading in the bus by doing this, it is best to set the concurrent consumer limit for the bus to the number of concurrent threads you want at one time.

On Sun, Feb 10, 2013 at 1:56 PM, <e.rydg...@gmail.com> wrote:

If all consumers are located inproc then I would use a semaphore to limit the concurrent number of threads. Just request a semaphore before making external call and release it when done. The starting number of available semaphores will be the maximum number of concurrent external callers. Relatively simple and contained inside the consumer without the need of another layer.
Although if you start cloning the consumer out to several nodes then you'll need a distributed semaphore as well, unless each node gets it's own number of available external call slots that is.

- Erik

Den fredagen den 6:e juli 2012 kl. 02:49:01 UTC+2 skrev kreiggers:

Yes I can use that but was hoping to find a more dynamic way of doing it. The problem is I don't know how many consumers I can use. The 90% case I can go as fast as I can, (not every message will cause a hit to the external service as we have some caching) but that won't work if there's a lot of hits. 

Thinking that I may have to put another layer of messaging in between these two cases and have the first layer use the max consumers and have a gateway to the external service run with only a couple of consumers. 

----

Craig Beck

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/gKTRu9LvsC4J.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Guillaume Schuermans]

A bit old, but you can always use Thread.Sleep() in combination with a number of Concurrent Consumers = 1.

[Chris Patterson]

Thread.Sleep is not the best idea, as it can cause the queue to give up and release the message to another consumer. A second or two might not hurt much, but anything too long is not recommended.

I find that creating a nasty stored procedure in SQL and invoking it from the consumer is a great way to slow the overall system performance to a crawl.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/03d3821f-886b-4fc3-8775-27462f5c5c2a%40googlegroups.com.

[Tom]

Hi Chris,

Is releasing a message a big deal if you are handling duplicated and unordered messages well?

BR,
Tomasz

[Travis Smith]

There's a couple of other recent threads on ACK'ing and RMQ. Normally a message will only be released when the either the ACK timeout is hit, or the underlying channel/connection misses heartbeats and is considered dead. Or if you forcibly close the connection, I'm guessing that would do it too. 

The default now should be no ACK timeout, so that's normally not a consideration. What you don't want to do is lockup your consumer so the channel can't respond to heart beats though. A Thread.Sleep() shouldn't do that, but now that Chris has rewrote most of the internals for 3.0 I can't say for sure. It should be easy to test though. 

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/6aa5dc86-ee47-4c24-8c0c-1e8859e5f96a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.