Long-running consumer task vs. background thread

[Jussi Mattila]

I have a long-running external process lasting for about 1 hour that is started from inside a consumer. Should I spawn a new thread for monitoring the task or can I do it inside the Consume method, which is a Task itself? 

That is, can I do this inside a consumer:

while (!job.HasEnded())

{

    await Task.Delay(TimeSpan.FromMinutes(1));

    await job.UpdateAsync();

}

I'm using MassTransit v3

[Travis Smith]

RMQ will consider your message unacked available for redelivery when a) the connection dies, b) the ack timeout is passed. I feel like the default is that there's no ack timeout now, so you only have to worry about a. If I'm wrong on that, you'll just need to disable the ack timeout or extend it beyond any reasonable processing timeframe.

Let's talk about a. for a little bit then. The connection can be considered dead when it receives a close/terminate or when the heartbeat timeout is exceeded. This could happen because the machine is under heavy load and it misses (normally) a couple heart beats, the network has a problem, or there is an actual outage. What do you want to happen if there's a problem and your message is redelivered? Or is redelivery a no-no period? 

Also consider that having unacked messages in RMQ just takes up memory. If you're low volume or low consumer count, then this won't be a problem. 

So you CAN just run the job inside the consumer. It's a question of what's the right behaviour for messages and your system. You might have a need to ack the message and process it outside the consumer. 

-Travis

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/da7866e6-837f-42dc-883d-022712714789%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Patterson]

This is a common question to which there is no great answer. If you're using different queuing systems, sometimes you only have a max of 5 minutes to receive the message. So waiting forever is not a great option. Of course, if the job fails, how do you make sure it actually gets processed?

My general rule is that something that takes that long to process could probably manage to do some sort of job tracking in a database, to keep the history and know when jobs were started that never finished. Use events and commands around that, but keep the state tracking in a database.

If it's a multi-step operation, you can use Courier to coordinate the steps and the events produced by courier to update the state tables.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/CAAqUTbkfLu8E28Esw67MLQS_-N_L-LTScMUuT-K%2BEjctV6j7jQ%40mail.gmail.com.

[Jussi Mattila]

Hi Travis and Chris,

Thanks for the insights, very helpful.

In my current (experimental) setup with very low volume and low consumer count, I'll let the task run within the consumer. I can manage redeliveries easily, i.e. the messages are handled idempotently. Based on your advice, I implemented and tested this over the weekend and it seems to work fine, as expected.

Later, when we start developing the same long-running feature for production environments, I feel that we will want immediate acks and job tracking instead of waiting for indeterminate amount of time. For that, we'll probably use events and a database for state but I'll have a look at the Courier too.

Again, thanks a lot for your responses!

   - Jussi

On Thursday, May 21, 2015 at 5:29:19 PM UTC+3, Chris Patterson wrote:

This is a common question to which there is no great answer. If you're using different queuing systems, sometimes you only have a max of 5 minutes to receive the message. So waiting forever is not a great option. Of course, if the job fails, how do you make sure it actually gets processed?

My general rule is that something that takes that long to process could probably manage to do some sort of job tracking in a database, to keep the history and know when jobs were started that never finished. Use events and commands around that, but keep the state tracking in a database.

If it's a multi-step operation, you can use Courier to coordinate the steps and the events produced by courier to update the state tables.

On Thu, May 21, 2015 at 7:09 AM, Travis Smith <tra...@legomaster.net> wrote:

RMQ will consider your message unacked available for redelivery when a) the connection dies, b) the ack timeout is passed. I feel like the default is that there's no ack timeout now, so you only have to worry about a. If I'm wrong on that, you'll just need to disable the ack timeout or extend it beyond any reasonable processing timeframe.

Let's talk about a. for a little bit then. The connection can be considered dead when it receives a close/terminate or when the heartbeat timeout is exceeded. This could happen because the machine is under heavy load and it misses (normally) a couple heart beats, the network has a problem, or there is an actual outage. What do you want to happen if there's a problem and your message is redelivered? Or is redelivery a no-no period? 

Also consider that having unacked messages in RMQ just takes up memory. If you're low volume or low consumer count, then this won't be a problem. 

So you CAN just run the job inside the consumer. It's a question of what's the right behaviour for messages and your system. You might have a need to ack the message and process it outside the consumer. 

-Travis

On Thu, May 21, 2015 at 9:15 AM, Jussi Mattila <jmatt...@gmail.com> wrote:

I have a long-running external process lasting for about 1 hour that is started from inside a consumer. Should I spawn a new thread for monitoring the task or can I do it inside the Consume method, which is a Task itself? 

That is, can I do this inside a consumer:

while (!job.HasEnded())

{

    await Task.Delay(TimeSpan.FromMinutes(1));

    await job.UpdateAsync();

}

I'm using MassTransit v3

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/da7866e6-837f-42dc-883d-022712714789%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

[Chris Patterson]

Consider using the new Turnout feature in MT3. Long running tasks finally!

[Jussi Mattila]

Thanks, I'll definitely investigate the new feature!