Insufficient resources to perform operation

[WilkoSki]

We are seeing an intermittent problem where MassTransit crashes due to
an MSMQ resource issue. I realise there has been posts about this
before, however we are not seeing a build up of messages in queues
like the previous posts. We have the service configured to restart,
but it takes several minutes for this to happen. We are using publish
and subscribe and all services are enabled in the MassTransit Runtime
Services.

The snippet of the MassTransit log file is attached below.

Any suggestions ?

Thanks

Steve


ERROR 11:58:46 There was a problem communicating with the message
queue: FormatName:DIRECT=OS:hc1-iis-002\private$\mt_accounts_client
System.Messaging.MessageQueueException: Insufficient resources to
perform operation.
at System.Messaging.MessageQueue.SendInternal(Object obj,
MessageQueueTransaction internalTransaction,
MessageQueueTransactionType transactionType)
at
MassTransit.Transports.Msmq.AbstractMsmqTransport.SendMessage(MessageQueue
queue, Message message)
at MassTransit.Transports.Msmq.AbstractMsmqTransport.Send(Action`1
sender)
ERROR 12:10:19 'System.Action`1[[System.Object, mscorlib,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]'
threw an exception consuming message
'MassTransit.Services.Subscriptions.Messages.AddSubscription'
System.Reflection.TargetInvocationException: Exception has been thrown
by the target of an invocation. ---> System.NullReferenceException:
Object reference not set to an instance of an object.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue
value, Boolean add)
at System.Collections.Generic.Dictionary`2.set_Item(TKey key,
TValue value)
at Magnum.CollectionExtensions.DictionaryExt.Retrieve[TKey,TValue]
(IDictionary`2 dictionary, TKey key, Func`1 valueProvider)
at Magnum.Reflection.ReflectiveVisitorBase`1.GetInvoker(Int32 key,
Func`1 getMethodInfo)
at Magnum.Reflection.ReflectiveVisitorBase`1.DispatchVisit(Object
obj)
at Magnum.Reflection.ReflectiveVisitorBase`1.Visit(Object obj,
Func`1 action)
at
MassTransit.Pipeline.Inspectors.PipelineInspectorBase`1.Inspect(Object
sink, Func`1 inspectChildSinks)
at MassTransit.Pipeline.MessagePipeline.Inspect(IPipelineInspector
inspector)
at
MassTransit.Pipeline.Configuration.MessageRouterConfigurator.FindOrCreate[TMessage]
()
at
MassTransit.Pipeline.Configuration.ExtensionsForPipelineConfiguration.ConnectToRouter[T]
(IMessagePipeline pipeline, IPipelineSink`1 sink)
at
MassTransit.Pipeline.MessagePipelineExtensions.Subscribe[TMessage]
(IMessagePipeline pipeline, IEndpoint endpoint)
at
MassTransit.Services.Subscriptions.SubscriptionConsumer.SubscribedTo[T]
(Uri endpointUri)
at
MassTransit.Services.Subscriptions.Client.SubscriptionClient.<>c__DisplayClassa`1.<AddToClients>b__9(IEndpointSubscriptionEvent
x)
at System.Collections.Generic.List`1.ForEach(Action`1 action)
at
MassTransit.Util.RegistrationList`1.<>c__DisplayClass8.<Each>b__7(List`1
x)
at
Magnum.Threading.ReaderWriterLockedObject`1.<>c__DisplayClass1.<ReadLock>b__0(ReaderWriterLockContext
x)
at Magnum.Threading.ReaderWriterLockContext.ReadLock(Action`1
action)
at Magnum.Threading.ReaderWriterLockedObject`1.ReadLock(Action`1
action)
at MassTransit.Util.RegistrationList`1.Each(Action`1 action)
at
MassTransit.Services.Subscriptions.Client.SubscriptionClient.AddToClients[T]
(Uri endpointUri)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target,
Object[] arguments, SignatureStruct& sig, MethodAttributes
methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj,
BindingFlags invokeAttr, Binder binder, Object[] parameters,
CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj,
BindingFlags invokeAttr, Binder binder, Object[] parameters,
CultureInfo culture)
at
MassTransit.Services.Subscriptions.Client.SubscriptionClient.Add(SubscriptionInformation
sub)
at
MassTransit.Pipeline.Sinks.MessageTranslator`2.<>c__DisplayClass1.<Enumerate>b__0(TInput
x)
at
MassTransit.Internal.ServiceBusReceiveContext.DeliverMessageToConsumers(Object
message)
ERROR 12:10:19 An exception was thrown by a message consumer
MassTransit.Exceptions.MessageException: At least one consumer threw
an exception ---> System.Reflection.TargetInvocationException:
Exception has been thrown by the target of an invocation. --->
System.NullReferenceException: Object reference not set to an instance
of an object.
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue
value, Boolean add)
at System.Collections.Generic.Dictionary`2.set_Item(TKey key,
TValue value)
at Magnum.CollectionExtensions.DictionaryExt.Retrieve[TKey,TValue]
(IDictionary`2 dictionary, TKey key, Func`1 valueProvider)
at Magnum.Reflection.ReflectiveVisitorBase`1.GetInvoker(Int32 key,
Func`1 getMethodInfo)
at Magnum.Reflection.ReflectiveVisitorBase`1.DispatchVisit(Object
obj)
at Magnum.Reflection.ReflectiveVisitorBase`1.Visit(Object obj,
Func`1 action)
at
MassTransit.Pipeline.Inspectors.PipelineInspectorBase`1.Inspect(Object
sink, Func`1 inspectChildSinks)
at MassTransit.Pipeline.MessagePipeline.Inspect(IPipelineInspector
inspector)
at
MassTransit.Pipeline.Configuration.MessageRouterConfigurator.FindOrCreate[TMessage]
()
at
MassTransit.Pipeline.Configuration.ExtensionsForPipelineConfiguration.ConnectToRouter[T]
(IMessagePipeline pipeline, IPipelineSink`1 sink)
at
MassTransit.Pipeline.MessagePipelineExtensions.Subscribe[TMessage]
(IMessagePipeline pipeline, IEndpoint endpoint)
at
MassTransit.Services.Subscriptions.SubscriptionConsumer.SubscribedTo[T]
(Uri endpointUri)
at
MassTransit.Services.Subscriptions.Client.SubscriptionClient.<>c__DisplayClassa`1.<AddToClients>b__9(IEndpointSubscriptionEvent
x)
at System.Collections.Generic.List`1.ForEach(Action`1 action)
at
MassTransit.Util.RegistrationList`1.<>c__DisplayClass8.<Each>b__7(List`1
x)
at
Magnum.Threading.ReaderWriterLockedObject`1.<>c__DisplayClass1.<ReadLock>b__0(ReaderWriterLockContext
x)
at Magnum.Threading.ReaderWriterLockContext.ReadLock(Action`1
action)
at Magnum.Threading.ReaderWriterLockedObject`1.ReadLock(Action`1
action)
at MassTransit.Util.RegistrationList`1.Each(Action`1 action)
at
MassTransit.Services.Subscriptions.Client.SubscriptionClient.AddToClients[T]
(Uri endpointUri)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target,
Object[] arguments, SignatureStruct& sig, MethodAttributes
methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj,
BindingFlags invokeAttr, Binder binder, Object[] parameters,
CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj,
BindingFlags invokeAttr, Binder binder, Object[] parameters,
CultureInfo culture)
at
MassTransit.Services.Subscriptions.Client.SubscriptionClient.Add(SubscriptionInformation
sub)
at
MassTransit.Pipeline.Sinks.MessageTranslator`2.<>c__DisplayClass1.<Enumerate>b__0(TInput
x)
at
MassTransit.Internal.ServiceBusReceiveContext.DeliverMessageToConsumers(Object
message)
--- End of inner exception stack trace ---
at
MassTransit.Internal.ServiceBusReceiveContext.DeliverMessageToConsumers(Object
message)
at
MassTransit.Transports.Msmq.MsmqEndpoint.<>c__DisplayClass5.<>c__DisplayClass7.<ReceiveFromTransport>b__4(Message
m)

[Travis Smith]

Do you have any perfmon metrics from MSMQ or your service as it
crashes? Maybe some idea of how much memory it's using (and measured
against total memory usage) or disk space on the drive the queues are
physically located? Are you using transactional or non-transactional
queues?

The only times I've seen MSMQ take minutes to start is related to it
having a lot of messages in a queue. It's very possible that there are
other ways to make this happen though.

-Travis

> --
> You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
> To post to this group, send email to masstrans...@googlegroups.com.
> To unsubscribe from this group, send email to masstransit-dis...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/masstransit-discuss?hl=en.
>
>

[Dru Sellers]

Agreed.
Please look at hard disk and RAM metrics when it crashes.
Also, what does your message volume look like?

-d

[WilkoSki]

I don't have any metrics from it currently. I'll make sure they are
checked when it goes down again as it does appear to be fairly
frequently.

We aren't specifically enabling or disabling transactional queues, we
are just using default settings (non-transactional ?).

We are probably seeing somewhere in the region of 10,000 messages per
hour.

Steve

On May 11, 1:52 pm, Dru Sellers <d...@drusellers.com> wrote:
> Agreed.
> Please look at hard disk and RAM metrics when it crashes.
> Also, what does your message volume look like?
>
> -d
>
>
>
>
>
>
>
> On Wed, May 11, 2011 at 6:49 AM, Travis Smith <tra...@legomaster.net> wrote:
> > Do you have any perfmon metrics from MSMQ or your service as it
> > crashes? Maybe some idea of how much memory it's using (and measured
> > against total memory usage) or disk space on the drive the queues are
> > physically located? Are you using transactional or non-transactional
> > queues?
>
> > The only times I've seen MSMQ take minutes to start is related to it
> > having a lot of messages in a queue. It's very possible that there are
> > other ways to make this happen though.
>
> > -Travis
>

> > On Wed, May 11, 2011 at 7:38 AM, WilkoSki <stevewilkin...@hotmail.co.uk>

> > MassTransit.Pipeline.Configuration.ExtensionsForPipelineConfiguration.Conne ctToRouter[T]

> > > (IMessagePipeline pipeline, IPipelineSink`1 sink)
> > >   at
> > > MassTransit.Pipeline.MessagePipelineExtensions.Subscribe[TMessage]
> > > (IMessagePipeline pipeline, IEndpoint endpoint)
> > >   at
> > > MassTransit.Services.Subscriptions.SubscriptionConsumer.SubscribedTo[T]
> > > (Uri endpointUri)
> > >   at
>

> > MassTransit.Services.Subscriptions.Client.SubscriptionClient.<>c__DisplayCl assa`1.<AddToClients>b__9(IEndpointSubscriptionEvent

> > MassTransit.Pipeline.Configuration.ExtensionsForPipelineConfiguration.Conne ctToRouter[T]

> > > (IMessagePipeline pipeline, IPipelineSink`1 sink)
> > >   at
> > > MassTransit.Pipeline.MessagePipelineExtensions.Subscribe[TMessage]
> > > (IMessagePipeline pipeline, IEndpoint endpoint)
> > >   at
> > > MassTransit.Services.Subscriptions.SubscriptionConsumer.SubscribedTo[T]
> > > (Uri endpointUri)
> > >   at
>

> > MassTransit.Services.Subscriptions.Client.SubscriptionClient.<>c__DisplayCl assa`1.<AddToClients>b__9(IEndpointSubscriptionEvent

> > > message)
> > >   --- End of inner exception stack trace ---
> > >   at
>
> > MassTransit.Internal.ServiceBusReceiveContext.DeliverMessageToConsumers(Obj ect
> > > message)
> > >   at
>

> > MassTransit.Transports.Msmq.MsmqEndpoint.<>c__DisplayClass5.<>c__DisplayCla ss7.<ReceiveFromTransport>b__4(Message

[Dru Sellers]

How big are your messages?
Where are your MSMQ messages stored (MSMQ Service > Properties) by default they are on the C drive.
And the default amount of storage for MSMQ is only 1GB

-d

[Nikita Govorov]

We have the same issue.  Nothing helps(even server restarting) except stopping all services and removing all data(subscriptions are the most important part here, I think) from Runtime Services’s tables. We don’t use timeout and health monitoring services, but actively use request/response way of communication. When it happens Runtime Services process uses about 1.5 GB of RAM and 1500 threads. We use MT from master(dated 2010-08-25).

 

 

Yours sincerely, Nikita Govorov

[Travis Smith]

Runtime Services using 1.5 gig of RAM makes me thing we have an issue
there. How many endpoints are involved in your bus? How often do your
endpoint join/leave the network? I've had to run months with just
using ~50 megs of ram.

-Travis

[WilkoSki]

Dru,

The messages aren't very big, probably 1K at the most.

MSMQ messages are stored on the C drive in the default location
although the allocated storage space has been increased.

When it crashes we don't see any build up of messages in the queues
therefore storage shouldn't be an issue. We have seen a large number
of records in the SubscriptionsSaga and Subscriptions table though,
should this have any affect ? If all subscribers and publishers were
active then we should only be seeing around 10 end points.

The Masstransit database really needs some automatic maintenance to
keep the number of records in check.

Steve

On May 11, 2:21 pm, Dru Sellers <d...@drusellers.com> wrote:
> How big are your messages?
> Where are your MSMQ messages stored (MSMQ Service > Properties) by default
> they are on the C drive.
> And the default amount of storage for MSMQ is only 1GB
>
> -d
>

> ...
>
> read more »

[Travis Smith]

> The Masstransit database really needs some automatic maintenance to
> keep the number of records in check.

It's on our list! You are correct.

-Travis

[Dru Sellers]

Ok, good to know on the message size.
Good to know you have increased your MSMQ storage.

Agreed on the subscription service clean up. Please feel free to log an issue at:
https://github.com/masstransit/masstransit/issues

@Travis: I didn't see one in there yet.

@Steve: Do you know what your peek throughput is on the messages?

-d

--

[Travis Smith]

I guess it's not actually an issue, I'll add it. We have been talking
about it for some time though. At least I've been complaining to Dru
and Chris about it from time to time. It just hasn't been high
priority.

-Travis

[Dru Sellers]

I tag you Travis. ;)

-d