Messages moved to error queue without any logging after parallel publishes to exchange
Gav
Bit of a strange one here. I'm seeing this in production but cannot reproduce on a development environment.
I'm using RabbitMq and Masstransit with the following versions:
<package id="Magnum" version="2.1.2" targetFramework="net45" />
<package id="MassTransit" version="2.9.5" targetFramework="net45" />
<package id="MassTransit.Log4Net" version="2.9.5" targetFramework="net45" />
<package id="MassTransit.RabbitMQ" version="2.9.5" targetFramework="net45" />
<package id="Newtonsoft.Json" version="6.0.8" targetFramework="net45" />
<package id="RabbitMQ.Client" version="3.3.5" targetFramework="net45" />
When messages are published at exactly the same time (as seen from the logs, published at the same ms) then when they arrive at the subscriber one or more of these messages appear in the error queue. If these publishes are spaced out, then they get to the subscriber without issue. With MT logging enabled I can see the message sent successfully to RMQ. With the firehose plugin I can see the message received and delivered to the exchange (5 times) then get moved to the _error queue. But there is nothing in the logs from the subscriber to say that it was even received. I have added Fault consumers also for these messages and these aren't being invoked either.
Now I know its being handled by MT as its re-trying and eventually ends up in the error queue. Is there anything in the MT pipeline that would cause an error not to be logged but the error requeuing to sill work?
Publisher Logs
2017-06-07 10:32:00.128Z DEBUG [51] MassTransit.Messages - SEND:rabbitmq://redacted:ISessionsAvailable:74800000-2923-000c-e646-08d4ad9373fa:MassTransit.Serialization.EncryptedMessageEnvelope, MassTransit
2017-06-07 10:32:00.128Z DEBUG [51] Transports.Endpoint - Received Successfully: dcce0000-290a-000c-f013-08d4ad92efc9
2017-06-07 10:32:00.128Z DEBUG [37] MassTransit.Messages - SEND:rabbitmq://redacted:ISessionsAvailable:74800000-2923-000c-faa5-08d4ad9373fa:MassTransit.Serialization.EncryptedMessageEnvelope, MassTransit
2017-06-07 10:32:00.128Z DEBUG [37] Transports.Endpoint - Received Successfully: dcce0000-290a-000c-dfbc-08d4ad92efcb
Message sent on thread id [51] received by RMQ. Message sent on thread id [37] sent at exactly the same time to the same exchange.
Subscriber Logs
2017-06-07 10:32:00.099Z DEBUG [83] MassTransit.Messages - RECV:rabbitmq://redacted:74800000-2923-000c-faa5-08d4ad9373fa:redacted.ISessionsAvailable, redacted
2017-06-07 10:32:00.099Z DEBUG [83] Context.ServiceBusReceiveContext - Dispatching message on rabbitmq://193.243.28.120/CyaraOmniScheduler.VCCDKTCYARA1 from thread 83
2017-06-07 10:32:00.099Z DEBUG [83] Consumers.SessionsAvailableConsumer - Start consume SessionsAvailable. Ticket:47169f07-5caf-4c17-acb1-674a38cc6568 Count:36 CoordinatorUri:rabbitmq://redacted
Logs indicate that one message was successful. No mention of the other message at all in the logs. The message has been moved to the error queue.
I have attempted to replicate this on my development environment, running multiple threads against the same exchange, however not lucj as yet.
Help!