MassTransit stops processing from a queue

[Matt Honeycutt]

I saw another thread that sounded related, but after reading through it, I think this may be a completely separate issue.  I've had MassTransit stop processing incoming messages randomly.  Nothing I've tried, short of deleting the underlying queue, will get MassTransit to start processing again once this happens. 

First, here's some info on my setup:

1) RabbitMQ 3.4.1

2) MassTransit 3.1.2

3) All consumers are simple IConsumer<T>'s, there are no sagas, no request/response, just simple pub/sub.  

I have several TopShelf services that are each listening on different queues via MassTransit.  Things have been working fine 99.95% of the time for the past many months.  

However, I've now had this issue pop up twice, both times with different services.  The affected service suddenly stops processing incoming messages.  It still publishes messages just fine.  The service still starts and stops without issues.  Everything else in the service continues to run fine.  There are no errors from MassTransit (I've got MassTransit.NLog installed and configured). But the queue in RabbitMQ shows 4 unack'd messages, and an ever-increasing number of ready messages.  I've added logging in my app, and I can't see that it is ever even trying to create a consumer.  I also don't see anything in the MassTransit logs that indicates it is trying to process anything (I'm not sure if there should be?)

Again, I've tried stopping the service, purging the queue, then restarting the service, and the same thing happens again immediately.  I've also tried restarting the entire box.  The same thing happens again as soon as the service starts up.  It isn't until I actually delete the queue in RabbitMQ that things start to work correctly again.

Any ideas?  Is there anything else I can enable or tweak to try and troubleshoot this on my end? 

[Chris Patterson]

The fact that there are 4 unacked messages is an indicator. I'm guessing you have a prefetch count of four, and are running four consumer threads. I would hazard a guess that those four consumers are blocked on something, and are unable to complete. 

Are you able to share the code for the consumer on the queue that is blocked? Are you playing nice with the TPL and not using any .Wait() calls or .Result in your consumer?

__
Chris Patterson

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/4d023972-6aff-4222-943a-45f604f2e676%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Honeycutt]

Ugh, yup, you are correct.  I was reading the order of queued messages in RabbitMQ wrong, so I was chasing down the wrong consumer.  There was indeed a deadlock in the actual culprit.  It was pretty obvious once I looked in the right place.  I'm not sure why purge the queue didn't fix things, but deleting it did... must have been pure luck.  Anyway, my bad!

On Monday, February 8, 2016 at 10:33:28 PM UTC-6, Chris Patterson wrote:

The fact that there are 4 unacked messages is an indicator. I'm guessing you have a prefetch count of four, and are running four consumer threads. I would hazard a guess that those four consumers are blocked on something, and are unable to complete. 

Are you able to share the code for the consumer on the queue that is blocked? Are you playing nice with the TPL and not using any .Wait() calls or .Result in your consumer?

__
Chris Patterson

On Mon, Feb 8, 2016 at 8:02 PM -0800, "Matt Honeycutt" <mbhon...@gmail.com> wrote:

I saw another thread that sounded related, but after reading through it, I think this may be a completely separate issue.  I've had MassTransit stop processing incoming messages randomly.  Nothing I've tried, short of deleting the underlying queue, will get MassTransit to start processing again once this happens. 

First, here's some info on my setup:

1) RabbitMQ 3.4.1

2) MassTransit 3.1.2

3) All consumers are simple IConsumer<T>'s, there are no sagas, no request/response, just simple pub/sub.  

I have several TopShelf services that are each listening on different queues via MassTransit.  Things have been working fine 99.95% of the time for the past many months.  

However, I've now had this issue pop up twice, both times with different services.  The affected service suddenly stops processing incoming messages.  It still publishes messages just fine.  The service still starts and stops without issues.  Everything else in the service continues to run fine.  There are no errors from MassTransit (I've got MassTransit.NLog installed and configured). But the queue in RabbitMQ shows 4 unack'd messages, and an ever-increasing number of ready messages.  I've added logging in my app, and I can't see that it is ever even trying to create a consumer.  I also don't see anything in the MassTransit logs that indicates it is trying to process anything (I'm not sure if there should be?)

Again, I've tried stopping the service, purging the queue, then restarting the service, and the same thing happens again immediately.  I've also tried restarting the entire box.  The same thing happens again as soon as the service starts up.  It isn't until I actually delete the queue in RabbitMQ that things start to work correctly again.

Any ideas?  Is there anything else I can enable or tweak to try and troubleshoot this on my end? 

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.