Re: [masstransit-discuss] Unexpected large volume of messages routed to error queue?

[Dru Sellers]

are the logs showing you anything?

-d

On Wed, Jun 6, 2012 at 6:31 PM, Geoff <ge...@geoffsteffens.com> wrote:

We are using MassTransit 2.1.4 and RabbitMQ 2.8.2 to publish the results of consuming a real-time data feed from a remote service, and to keep the cluster of those 'consuming' apps in sync (eg to notify each other of heartbeat, to collaborate on reconnecting to the remote service if it goes down, etc).  To minimize terminology confusion, I'll refer to the cluster of our 'consuming' apps as 'the consumer group' in this post.

Whilst our solution is operating as expected and MT messages do appear to be being correctly routed amongst the consumer group, a large volume (>8000 per 24 hours) are being delivered to RabbitMQ's error queues.  I'm a bit at a loss to explain this because our consumer apps don't appear to be generating any exceptions which would lead to MT moving the message to error queue.  Also, the types of messages in the error queues are a random mixed bag of types.

In a troubleshooting attempt, I simplified our setup into a VM running 1xRabbitMQ and 1xOur Consumer (ie no clustering).  The problem still continues. Below is a copy of the MT introspection log.

Any suggestions on things to try?

host.machine_name: TEST_MACHINE_1

mt.concurrent_receive_threads: 1

mt.control_bus: rabbitmq://localhost/broker_TEST_MACHINE_1

mt.default_serializer: JsonMessageSerializer

mt.max_consumer_threads: 16

mt.network: TEST_MACHINE_1

mt.receive_from: rabbitmq://localhost/broker_TEST_MACHINE_1

mt.receive_timeout: 00:00:03

mt.service_count: 2

mt.transport: [loopback] LoopbackTransportFactory

mt.transport: [rabbitmq] RabbitMqTransportFactory

mt.version: 2.1.410.0

net.version: 4.0.30319.225

os.bits: x64

os.version: Microsoft Windows NT 6.1.7601 Service Pack 1

process.bits: x32

process.id: 8092

zz.mt.inbound_pipeline: Pipeline

Routed (MassTransit.IConsumeContext, MassTransit)

Translated to MassTransit.Diagnostics.Tracing.GetMessageTraceList, MassTransit

Routed (MassTransit.Diagnostics.Tracing.GetMessageTraceList, MassTransit)

Consumed by Instance (MassTransit.Diagnostics.Tracing.GetMessageTraceList, MassTransit)

zz.mt.outbound_pipeline: Pipeline

Interceptor

Routed (MassTransit.ISendContext, MassTransit)

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/UzcRpVy0PK8J.
To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/masstransit-discuss?hl=en.

[Geoff]

Hi Dru,

The logs actually led us to a solution of sorts.. In MT debug level we were intermittently seeing things along the line of:
Skip, skip, skip, retry, retry, too many retries
For the same message

We use the Selected consumers a lot, mainly to filter out loopback messages.

By switching back from Consumes<T>.Selected to .All and manually calling the exact same Accept() code from our Consume method our error count has gone to 0.

So it appears there is some condition in MT whereby Selected causes messages to route tithe error queue if not consumed by Selected. Note that no exceptions were generated by our code.

Very odd but our workaround seems effective

[Geoff]

[Chris Patterson]

If you have multiple bus instances receiving from the same queue, and you are using selected to partition messages, you would certainly see this type of behavior. With a single queue, the messages that are you not receiving (Accept returns false) will eventually end up in the error queue since they were not interesting to you. We have to do this so that the messages don't just sit in the input queue blocking other messages forever.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/nwV-Xx8ZxxcJ.

[Mike Rowley]

We are having this issue as well.  Our event logger is getting spammed with errors when using "SubscribeHandler(action, predicate)" where the predicate returns false.  It appears that the message is skipped 8 times then sent to the error queue.  There is only one IServiceBus instance using this queue.

2012-07-04 15:56:25,570 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27610

2012-07-04 15:56:25,570 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27683

2012-07-04 15:56:28,570 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27610

2012-07-04 15:56:28,571 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27683

2012-07-04 15:56:31,573 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27610

2012-07-04 15:56:31,573 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27683

2012-07-04 15:56:34,573 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27610

2012-07-04 15:56:34,573 - Messages [DEBUG] SKIP:msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27683

2012-07-04 15:56:37,575 - Endpoint [ERROR] Message retry limit exceeded msmq://nova/myApp_entity_change_logging?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27610

2012-07-04 15:56:37,582 - MessageQueueConnection [DEBUG] Creating MessageQueue: FormatName:DIRECT=OS:localhost\private$\myApp_entity_change_logging_error

2012-07-04 15:56:37,600 - Messages [DEBUG] SEND:msmq://nova/myApp_entity_change_logging_error?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27750:

2012-07-04 15:56:37,600 - Messages [INFO] MOVE:msmq://nova/myApp_entity_change_logging?tx=false:msmq://nova/myApp_entity_change_logging_error?tx=false:1cfcae0d-a77e-48ba-8abe-ac386b66a783\27610:

Cheers,

mike

[Chris Patterson]

Selective consumer is really something whose time has come and gone, as it is constantly misused by nearly everyone.

And that's expected, if you return false enough, it will throw the message into the error queue as unwanted.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/eXprtFPk_OcJ.

[Mike Rowley]

I see. I did misunderstand its usage.  

I'll take a look at MassTransit.Reactive and see if I can make use of filtering via Rx.  We are using Rx with SignalR on the client side so it might make sense anyways through our IEventBus abstraction.  

Thanks

To post to this group, send email to masstransit-discuss@googlegroups.com.
To unsubscribe from this group, send email to masstransit-discuss+unsub...@googlegroups.com.

[Dru Sellers]

note that the Rx stuff is functional, but not heavily tested. at least to my knowledge. please share any feedback.

-d

To view this discussion on the web visit https://groups.google.com/d/msg/masstransit-discuss/-/mzleXZDH7BUJ.

To post to this group, send email to masstrans...@googlegroups.com.
To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.

[Greg Hoover]

Thanks, Chris,

Is there a more recent technique you would recommend to accomplish filtering functionality?  Rx or other?

[Dru Sellers]

Hi Greg.

Ok, filtering. What are you trying to do again. Sorry its been a long weekend.

-d

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Geerts]

Hi Chris, Dru

When returning false within the Accept function, after 5 times the message will be moved to the error_queue. Is this behavior that can/will be modified?

I don't see the added value of the Selected interface anymore with this, could you tell me what it is? It seems to be better to just Consume the message and check inside the Consume function if you want to consume the message.

Greetz

T

[Chris Patterson]

Selected was something that should be marked obsolete, it's not adding value and was just part of the API early on.

You're best to consume and act (or discard).

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/ec0260be-92ab-4a3c-afaf-e91f3dce6682%40googlegroups.com.

[Mike Rowley]

Been a while but just wanted to give some feedback regarding Rx.  We used it and it worked perfect for our needs.

-d

Thanks

To post to this group, send email to masstrans...@googlegroups.com.

To unsubscribe from this group, send email to masstransit-discuss+unsubscribe...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/masstransit-discuss?hl=en.