Hi Chris!
We did a number of unit tests and cannot reproduce the issue. The issue is still present in our staging environment, but only occurs after a few days on a fresh install. At first, everything works as expected, but after a while, messages sometimes do not get delivered to their handlers, they simply end up in the error queue without ever reaching our handler's code.
Looking at the config MassTransit created in RabbitMQ, we see that our target queue is bound to an exchange X as expected.
X receives messages of types E1 and E2 from their respective exchanges, as expected.
X also receives messages of type MassTransit.Diagnostics.Tracing:GetMessageTraceList.
The subscription code is a bit complicated because handler creation (messaging technology independent) is separated from exception handling (messaging technology dependent because of retry-mechanisms). Our MassTransitSubscriber subscribes to MassTransit as follows:
protected override void Subscribe<TMessage, THandlerType>(Action<TMessage, Action<Exception>> handle, int maxRetries)
{
serviceBus.SubscribeHandler<TMessage>(message => handle(message, exc =>
{
IConsumeContext<TMessage> context = serviceBus.MessageContext<TMessage>();
var extProps = new Dictionary<string, object>
{
{"Message ID", context.MessageId},
{"Message Type", context.MessageType},
{"Exception", exc}
};
if (context.RetryCount < maxRetries)
{
logger.WriteWarning("Failed to process message. Message will be retried.", extProps);
context.RetryLater();
}
else
logger.WriteError("Failed to process message (dead message).", extProps);
}));
logger.WriteVerbose(string.Format("MassTransit subscriber {2} subscribed to {0} on {1}.", typeof (TMessage).Name, serviceBus.Endpoint.Address, instanceId));
this.WriteLine("Subscribed to {0} on {1}.", typeof (TMessage).Name, serviceBus.Endpoint.Address);
}
The actual delegate which is subscribed is a 'safe' invocation of the underlying message handler type. Safe means that the handler is wrapped with logging, exception handling and disposal if required. This safe method is defined in the MessageBusSubscriber which is a the base class for the MassTransitSubscriber:
protected void Handle<TMessage, THandlerType>(TMessage message, Action<Exception> handleException)
{
logger.WriteVerbose(string.Format("Receiving message of type {0} on {1}.", message.GetType().Name, this));
IMessageHandler<TMessage> handler = null;
Type handlerType = typeof (THandlerType);
string messageTypeName = message.GetType().Name;
string handlerTypeName = handlerType.Name;
var ep = new Dictionary<string, object> {{"Message", message}};
logger.Log(LogCategories.Technical, LogSeverity.Info, ep, "Handling {0} on {1}...", messageTypeName, handlerTypeName);
try
{
handler = (IMessageHandler<TMessage>) createHandler(handlerType);
handler.Handle(message);
logger.Log(LogCategories.Technical, LogSeverity.Info, "Handling {0} on {1} completed.", messageTypeName, handlerTypeName);
}
catch (Exception exc)
{
logger.Log(LogCategories.Technical, LogSeverity.Warning, "Handling {0} on {1} failed.", messageTypeName, handlerTypeName);
try
{
// handle exception can vary based on the message bus technology (retry mechanisms)
handleException(exc);
}
catch (Exception e)
{
var extProps = new Dictionary<string, object>
{
{"Exception", exc},
{"Error handling exception", e}
};
logger.Log(LogCategories.Technical, LogSeverity.Error, extProps, "Exception while handling message handler exception.");
}
}
finally
{
handler.As<IDisposable>(d => d.Dispose());
}
}
What we see when debugging MassTransit is that MassTransit fails to retrieve an IConsumeContext in MassTransit.Context.ReceiveContext:

T is our expected E2 message type. The result is that the message is skipped until it ends up in the dead message queue. Any clues what could cause this?
The code has gone through a lot of unit tests and does not fail during those unit tests and the first few days on a fresh install. We will
set up a long running IIS based test to see to try to reproduce it.
Curious to find out what makes this happen!
Jochen