Threading issue when using Unity container and satellites

1,122 views
Skip to first unread message

Björn Bailleul

unread,
Feb 26, 2014, 5:16:59 AM2/26/14
to particula...@googlegroups.com
It seems there's an issues when using a Unity 2.x container and starting satellites. We use a lot of injection in our handlers and in our production environment we see that the parallel tasks for starting satellites poses an issue on startup. In special the second level retry processor satellite. The results in the second level retry processor not starting correctly and not handling its queue.

Any ideas on how to solve this?

Helpful information to include
Product name: NServiceBus
Version: 4.0.4
Stacktrace:

2014-02-25 16:25:25,726 [21] DEBUG NServiceBus.Satellites.SatelliteLauncher [(null)] - Starting 1/3 'NServiceBus.SecondLevelRetries.SecondLevelRetriesProcessor, NServiceBus.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c' satellite
2014-02-25 16:25:25,726 [22] DEBUG NServiceBus.Unicast.UnicastBus [(null)] - Started NServiceBus.Unicast.Subscriptions.MessageDrivenSubscriptions.StorageInitializer, NServiceBus.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c.
2014-02-25 16:25:25,742 [8] DEBUG Pms.Framework.Unity.UnityServiceHostFactory [(null)] - Creating service host for server AdminService with Uri's http://----.com/services/application/AdminService.svc, 
2014-02-25 16:25:25,742 [21] ERROR NServiceBus.Unicast.UnicastBus [(null)] - System.AggregateException: One or more errors occurred. ---> Microsoft.Practices.Unity.ResolutionFailedException: Resolution of the dependency failed, type = "NServiceBus.Unicast.Transport.TransportReceiver", name = "(none)".
Exception occurred while: while resolving.
Exception is: InvalidOperationException - Collection was modified; enumeration operation may not execute.
-----------------------------------------------
At the time of the exception, the container was:

  Resolving NServiceBus.Unicast.Transport.TransportReceiver,(none)
 ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()
   at Microsoft.Practices.Unity.UnityContainer.FillTypeRegistrationDictionary(IDictionary`2 typeRegistrations)
   at Microsoft.Practices.Unity.UnityContainer.get_Registrations()
   at Microsoft.Practices.Unity.UnityContainerExtensions.IsRegistered(IUnityContainer container, Type typeToCheck, String nameToCheck)
   at Microsoft.Practices.Unity.UnityContainerExtensions.IsRegistered(IUnityContainer container, Type typeToCheck)
   at NServiceBus.ObjectBuilder.Unity.PropertyInjectionBuilderStrategy.PreBuildUp(IBuilderContext context)
   at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)
   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)
   --- End of inner exception stack trace ---
   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)
   at Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name, ResolverOverride[] resolverOverrides)
   at Microsoft.Practices.Unity.UnityContainerExtensions.Resolve(IUnityContainer container, Type t, ResolverOverride[] overrides)
   at NServiceBus.ObjectBuilder.Unity.UnityObjectBuilder.Build(Type typeToBuild)
   at NServiceBus.ObjectBuilder.Common.CommonObjectBuilder.NServiceBus.ObjectBuilder.IBuilder.Build[T]()
   at NServiceBus.Satellites.SatelliteLauncher.<>c__DisplayClass4.<Start>b__2(Int32 index)
   at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.For(Int32 fromInclusive, Int32 toExclusive, Action`1 body)
   at NServiceBus.Satellites.SatelliteLauncher.Start()
   at NServiceBus.Unicast.UnicastBus.<>c__DisplayClass18.<Start>b__16()
---> (Inner Exception #0) Microsoft.Practices.Unity.ResolutionFailedException: Resolution of the dependency failed, type = "NServiceBus.Unicast.Transport.TransportReceiver", name = "(none)".
Exception occurred while: while resolving.
Exception is: InvalidOperationException - Collection was modified; enumeration operation may not execute.
-----------------------------------------------
At the time of the exception, the container was:

  Resolving NServiceBus.Unicast.Transport.TransportReceiver,(none)
 ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()
   at Microsoft.Practices.Unity.UnityContainer.FillTypeRegistrationDictionary(IDictionary`2 typeRegistrations)
   at Microsoft.Practices.Unity.UnityContainer.get_Registrations()
   at Microsoft.Practices.Unity.UnityContainerExtensions.IsRegistered(IUnityContainer container, Type typeToCheck, String nameToCheck)
   at Microsoft.Practices.Unity.UnityContainerExtensions.IsRegistered(IUnityContainer container, Type typeToCheck)
   at NServiceBus.ObjectBuilder.Unity.PropertyInjectionBuilderStrategy.PreBuildUp(IBuilderContext context)
   at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)
   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)
   --- End of inner exception stack trace ---
   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)
   at Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name, ResolverOverride[] resolverOverrides)
   at Microsoft.Practices.Unity.UnityContainerExtensions.Resolve(IUnityContainer container, Type t, ResolverOverride[] overrides)
   at NServiceBus.ObjectBuilder.Unity.UnityObjectBuilder.Build(Type typeToBuild)
   at NServiceBus.ObjectBuilder.Common.CommonObjectBuilder.NServiceBus.ObjectBuilder.IBuilder.Build[T]()
   at NServiceBus.Satellites.SatelliteLauncher.<>c__DisplayClass4.<Start>b__2(Int32 index)
   at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)<---
 could not be started.

John Simons

unread,
Feb 27, 2014, 1:12:24 AM2/27/14
to particula...@googlegroups.com
Hi Björn,

This is going to be quite hard to replicate without knowing what you have registered in the container.
Is there any chance of sending us a small repro of the issue?

Cheers
John


--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/0ad8576b-c310-4621-aec0-67211138d8e0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Björn Bailleul

unread,
Feb 28, 2014, 4:19:16 AM2/28/14
to particula...@googlegroups.com
I'd love to post some more information, but the code is under NDA :(

It seems these 3 guys are racing for the container, filling and consuming the underlying collection at the same time:

2014-02-27 09:48:49,111 [11] DEBUG NServiceBus.Satellites.SatelliteLauncher [(null)] - Starting 1/3 'NServiceBus.SecondLevelRetries.SecondLevelRetriesProcessor, NServiceBus.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c' satellite
2014-02-27 09:48:49,111 [13] DEBUG NServiceBus.Satellites.SatelliteLauncher [(null)] - Starting 2/3 'NServiceBus.Timeout.Hosting.Windows.TimeoutDispatcherProcessor, NServiceBus.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c' satellite
2014-02-27 09:48:49,126 [12] DEBUG NServiceBus.Satellites.SatelliteLauncher [(null)] - Starting 3/3 'NServiceBus.Timeout.Hosting.Windows.TimeoutMessageProcessor, NServiceBus.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c' satellite

The TimeoutMessageProcessor get's started, the other 2 (1 and 2) fail due to race conditions in the container. I understand why, I also understand that Unity has issues in this manner. 

Op donderdag 27 februari 2014 07:12:24 UTC+1 schreef John Simons:
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.

John Simons

unread,
Feb 28, 2014, 4:55:38 AM2/28/14
to particula...@googlegroups.com
Is it possible to test using the latest version of NServiceBus?

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/0ad8576b-c310-4621-aec0-67211138d8e0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.
To view this discussion on the web visit https://groups.google.com/d/msgid/particularsoftware/afafaf19-bc7f-491e-92f8-a36d235d9c0e%40googlegroups.com.

Björn Bailleul

unread,
Feb 28, 2014, 4:57:48 AM2/28/14
to particula...@googlegroups.com
We're doing it as I'm typing this message. For our next release we'll be using the latest NServiceBus release. The above problem is one we saw in our current release.
I'll keep you posted on the progress.

Op vrijdag 28 februari 2014 10:55:38 UTC+1 schreef John Simons:
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.

John Simons

unread,
Mar 2, 2014, 7:37:35 PM3/2/14
to particula...@googlegroups.com
Hi Björn,

Is this issue random? or does it happen all the time?

Cheers
John



To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

Kashyap Shah

unread,
Mar 14, 2014, 9:12:46 PM3/14/14
to particula...@googlegroups.com
Hi,

I am currently experiencing this same issue and the issue for me seems to be when there is a message in MSMQ from last run of the service. This happens more in development environment as we are debugging code. Also, feel it has occurred in more often after we upgraded from v3.4.3 to v4.4.0.

Let me know if this helps narrowing down issue.

Using NServiceBus 4.4.0

Here is the stack trace..

[ERROR] Failed raising 'started message processing' event.

Microsoft.Practices.Unity.ResolutionFailedException: Resolution of the dependency failed, type = "NServiceBus.Pipeline.PipelineExecutor", name = "(none)".

Exception occurred while: while resolving.

Exception is: InvalidOperationException - Collection was modified; enumeration operation may not execute.

-----------------------------------------------

At the time of the exception, the container was:

 

  Resolving NServiceBus.Pipeline.PipelineExecutor,(none)

  Resolving parameter "pipelineBuilder" of constructor NServiceBus.Pipeline.PipelineExecutor(NServiceBus.ObjectBuilder.IBuilder builder, NServiceBus.Pipeline.PipelineBuilder pipelineBuilder)

    Resolving NServiceBus.Pipeline.PipelineBuilder,(none)

 ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.

   at System.Collections.Generic.Dictionary`2.KeyCollection.Enumerator.MoveNext()

   at Microsoft.Practices.Unity.UnityContainer.FillTypeRegistrationDictionary(IDictionary`2 typeRegistrations)

   at Microsoft.Practices.Unity.UnityContainer.get_Registrations()

   at Microsoft.Practices.Unity.UnityContainerExtensions.IsRegistered(IUnityContainer container, Type typeToCheck, String nameToCheck)

   at Microsoft.Practices.Unity.UnityContainerExtensions.IsRegistered(IUnityContainer container, Type typeToCheck)

   at NServiceBus.ObjectBuilder.Unity.PropertyInjectionBuilderStrategy.PreBuildUp(IBuilderContext context) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\ObjectBuilder.Unity\PropertyInjectionBuilderStrategy.cs:line 43

   at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)

   at Microsoft.Practices.ObjectBuilder2.BuilderContext.NewBuildUp(NamedTypeBuildKey newBuildKey)

   at Microsoft.Practices.Unity.ObjectBuilder.NamedTypeDependencyResolverPolicy.Resolve(IBuilderContext context)

   at BuildUp_NServiceBus.Pipeline.PipelineExecutor(IBuilderContext )

   at Microsoft.Practices.ObjectBuilder2.DynamicMethodBuildPlan.BuildUp(IBuilderContext context)

   at Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context)

   at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)

   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)

   --- End of inner exception stack trace ---

   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)

   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, String name, IEnumerable`1 resolverOverrides)

   at Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name, ResolverOverride[] resolverOverrides)

   at Microsoft.Practices.Unity.UnityContainerExtensions.Resolve(IUnityContainer container, Type t, ResolverOverride[] overrides)

   at NServiceBus.ObjectBuilder.Unity.UnityObjectBuilder.Build(Type typeToBuild) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\ObjectBuilder.Unity\UnityObjectBuilder.cs:line 57

   at NServiceBus.ObjectBuilder.Common.CommonObjectBuilder.Build[T]() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\ObjectBuilder\Common\CommonObjectBuilder.cs:line 159

   at NServiceBus.Unicast.UnicastBus.get_PipelineFactory() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 1181

   at NServiceBus.Unicast.UnicastBus.TransportStartedMessageProcessing(Object sender, StartedMessageProcessingEventArgs e) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 1052

   at NServiceBus.Unicast.Transport.TransportReceiver.OnStartedMessageProcessing(TransportMessage msg) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 417    at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, Object existing, String name, IEnumerable`1 resolverOverrides)

   at Microsoft.Practices.Unity.UnityContainer.DoBuildUp(Type t, String name, IEnumerable`1 resolverOverrides)

   at Microsoft.Practices.Unity.UnityContainer.Resolve(Type t, String name, ResolverOverride[] resolverOverrides)

   at Microsoft.Practices.Unity.UnityContainerExtensions.Resolve(IUnityContainer container, Type t, ResolverOverride[] overrides)

   at NServiceBus.ObjectBuilder.Unity.UnityObjectBuilder.Build(Type typeToBuild) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\ObjectBuilder.Unity\UnityObjectBuilder.cs:line 57

   at NServiceBus.ObjectBuilder.Common.CommonObjectBuilder.Build[T]() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\ObjectBuilder\Common\CommonObjectBuilder.cs:line 159

   at NServiceBus.Unicast.UnicastBus.get_PipelineFactory() in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 1181

   at NServiceBus.Unicast.UnicastBus.TransportStartedMessageProcessing(Object sender, StartedMessageProcessingEventArgs e) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\UnicastBus.cs:line 1052

   at NServiceBus.Unicast.Transport.TransportReceiver.OnStartedMessageProcessing(TransportMessage msg) in c:\BuildAgent\work\31f8c64a6e8a2d7c\src\NServiceBus.Core\Unicast\Transport\TransportReceiver.cs:line 417

--
You received this message because you are subscribed to the Google Groups "Particular Software" group.
To unsubscribe from this group and stop receiving emails from it, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

Joe Ratzloff

unread,
Mar 30, 2014, 5:22:03 PM3/30/14
to particula...@googlegroups.com
Bjorn,
Did you ever get to the bottom of this issue? 
Thanks.
Joe.


Kashyap Shah

unread,
Mar 30, 2014, 5:25:10 PM3/30/14
to particula...@googlegroups.com

I have not spent tremendous amount  of time on this. I would have to truly isolate NServiceBus from our code to replicate.

--
You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/CrV31LAPZhc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to particularsoftw...@googlegroups.com.

To post to this group, send email to particula...@googlegroups.com.
Visit this group at http://groups.google.com/group/particularsoftware.

John Simons

unread,
Mar 31, 2014, 5:36:52 AM3/31/14
to particula...@googlegroups.com
We have just released a new version that should address this issue, see https://github.com/Particular/NServiceBus.Unity/releases/tag/4.4.3

Regards
John


On Monday, 31 March 2014 07:25:10 UTC+10, Kashyap Shah wrote:

I have not spent tremendous amount  of time on this. I would have to truly isolate NServiceBus from our code to replicate.

On Mar 30, 2014 2:22 PM, "Joe Ratzloff" <joe.ra...@nservicebus.com> wrote:
Bjorn,
Did you ever get to the bottom of this issue? 
Thanks.
Joe.


--
You received this message because you are subscribed to a topic in the Google Groups "Particular Software" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/particularsoftware/CrV31LAPZhc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to particularsoftware+unsub...@googlegroups.com.
To post to this group, send email to particularsoftware@googlegroups.com.

Björn Bailleul

unread,
Mar 31, 2014, 5:43:30 AM3/31/14
to particula...@googlegroups.com
I managed to boil this down to a Unity multithreading issue that exists in the NamedTypesRegistry class and the enclosed dictionary which is not thread safe when consumed by multiple satellites while loading NServiceBus.

Op woensdag 26 februari 2014 11:16:59 UTC+1 schreef Björn Bailleul:
Reply all
Reply to author
Forward
0 new messages