Queues fill up, MSMQ freezes

John Reese

unread,

Aug 13, 2010, 1:13:01 PM8/13/10

to masstrans...@googlegroups.com

Hello,

We are having an infrequent but serious problem with Mass Transit. First of all, I should mention that we have adopted the practice of not unsubscribing when subscription hosts are closed…or ever, for that matter. We do this because if the application shuts down unexpectedly (as they did pretty frequently when we first deployed this system) , we don’t want to lose the messages. We want to be able to restart the app and have it pick up any messages that came in when it was down.

I suspect that this practice may have something to do with our problem, which I will now describe:

Every now and then, we will find that our entire system has stopped doing anything because MSMQ is down. We have several monitors set up to check error queues and alert us about fatal errors, but nothing ever goes to an error queue with this problem. Instead, the message is simply not sent. There are errors being logged, but usually we aren’t aware of the problem until we’re informed that customer orders are not processing. This can occur any time, 24/7. If it happens in the middle of the night on a Friday, we’re kind of screwed.

The error that is logged is “There was a problem communicating with the message queue: FormatName:DIRECT=OS(machine name)\private$\(queue name)”, and “Insufficient resources to perform operation.” When we look at the queues, we find that the mt_timeout queue has over 300,000 messages in it.

Purging the queue and restarting Runtime Services helps for a few days, but then the problem recurs. We have found that deleting subscription data gives us a longer-lasting remedy, but a month or two later we experience the same problem.

Looking at our logs, it appears that the failures are at first intermittent, then consistent. When I looked at the messages in the timeout queue, it looked like none of them were more than a couple of hours old, but I could be mistaken about that. It’s difficult to look at the messages when there’s over a quarter of a million of them in there.

In contrast, the errors occurred over about a ten-hour period.

Here is some of the Runtime Services log, which gets a little repetitious after a while:

INFO 12:20:04 Subscription Client Added: msmq://automation2/mt_inventory_list?tx=false [70a3bcc4-c700-43cb-878e-9dce000583a9]

INFO 12:20:05 Subscription Client Removed: msmq://automation2/mt_inventory_list?tx=false [5890e9a6-b407-4a1a-b4c9-9dcd01808848]

INFO 12:20:05 Subscription Removed: MassTransit.Services.HealthMonitoring.Messages.PingEndpoint, MassTransit msmq://automation2/mt_inventory_list?tx=false [18edd481-ad3a-4e96-99f4-9dcd018089f8] []

INFO 12:20:05 Subscription Added: MassTransit.Services.HealthMonitoring.Messages.PingEndpoint, MassTransit msmq://automation2/mt_inventory_list?tx=false [fd6c5da8-c979-4069-9885-9dce00058509] []

ERROR 12:43:55 There was a problem communicating with the message queue: FormatName:DIRECT=OS:automation2\private$\mt_health

System.Messaging.MessageQueueException: Insufficient resources to perform operation.

at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)

at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)

at MassTransit.Transports.Msmq.AbstractMsmqTransport.SendMessage(MessageQueue queue, Message message) in c:\sandbox\MassTransit\src\Transports\MassTransit.Transports.Msmq\AbstractMsmqTransport.cs:line 183