Azure Worker Role Crashes in Ninject
Ravi Krishna
Hi Everyone,
We have worker role which executes lots of incoming messages on different threads. We are seeing lots of individual thread crashes in Ninject component.
We are using older version 3.2.2.0.
I appreciate any suggestion or information to fix this. Thanks.
Symptom:
Hundreds of app crashes happening every day. We can see crash dumps on the VM.
Cause:
Null Ref Exception from Ninject.Activation.Caching.GarbageCollectionCachePruner.PruneCacheIfGarbageCollectorHasRun(System.Object) function
Assessment :
Issue is happening due to an unhandled exception on a custom thread, which is throwing a Null Ref Exception from Ninject.Activation.Caching.GarbageCollectionCachePruner.PruneCacheIfGarbageCollectorHasRun(System.Object) function. See details of investigation below.
DETAILS:
Dump name: WER9A0A.tmp.mdmp
Debug session time: Sun Dec 31 18:55:09.000 2023 (UTC + 0:00)
System Uptime: not available
Process Uptime: 0 days 20:52:32.000
Kernel time: 0 days 0:08:01.000
User time: 0 days 10:10:11.000
Unhandled exception on thread #26, causing crash à
Last Exception:
Address: 0x000001b29dae9368
HResult: 0x80004003
Type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
Stack Trace:
SP IP Function
000000030dafe8e0 00007ffbe49677e8 Ninject.Activation.Caching.GarbageCollectionCachePruner.PruneCacheIfGarbageCollectorHasRun(System.Object)
000000030dafe990 00007ffc3cf94c62 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000030dafea60 00007ffc3cf94af3 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000030dafea90 00007ffc3cfca269 System.Threading.TimerQueueTimer.CallCallback()
000000030dafeaf0 00007ffc3cfca066 System.Threading.TimerQueueTimer.Fire()
000000030dafeb60 00007ffc3cfd4cbf System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
000000030dafeba0 00007ffc3cfd3fd1 System.Threading.ThreadPoolWorkQueue.Dispatch()
This is happening on a background custom thread here à
DbgId ThreadId Apartment Kind CLR GC Mode GC Suspending?
26 6ac MTA Worker v4.7.4051.00 Preemptive no
Failed to evaluate rule BlockingRule: Object reference not set to an instance of an object.
SP IP Function
000000030dafe3e0 0000000000000000 FaultingExceptionFrame
000000030dafe8e0 00007ffbe49677e9 Ninject.Activation.Caching.GarbageCollectionCachePruner.PruneCacheIfGarbageCollectorHasRun(System.Object)
000000030dafe930 00007ffbe49677a7 Ninject.Activation.Caching.GarbageCollectionCachePruner.PruneCacheIfGarbageCollectorHasRun(System.Object)
000000030dafe990 00007ffc3cf94c63 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000030dafea60 00007ffc3cf94af4 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000030dafea90 00007ffc3cfca26a System.Threading.TimerQueueTimer.CallCallback()
000000030dafeaf0 00007ffc3cfca067 System.Threading.TimerQueueTimer.Fire()
000000030dafeb60 00007ffc3cfd4cc0 System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
000000030dafeba0 00007ffc3cfd3fd2 System.Threading.ThreadPoolWorkQueue.Dispatch()
000000030daff038 0000000000000000 DebuggerU2MCatchHandlerFrame
000000030daff1c8 0000000000000000 ContextTransitionFrame
000000030daff3f8 0000000000000000 DebuggerU2MCatchHandlerFrame
This might somehow be related to the other thread, which is again Ninject, trying to unload the app domain
18 Id: 5c0.ef8 Suspend: 0 Teb: 00000003`0001a000 Unfrozen
# Child-SP RetAddr Call Site
00 00000003`08efd8a8 00007ffc`505ee9c3 ntdll!ZwWaitForAlertByThreadId+0x14
01 00000003`08efd8b0 00007ffc`505ee8b6 ntdll!RtlpWaitOnAddressWithTimeout+0x43
02 00000003`08efd8e0 00007ffc`505ee72d ntdll!RtlpWaitOnAddress+0xae
03 00000003`08efd950 00007ffc`505a449c ntdll!RtlpWaitOnCriticalSection+0xd9
04 00000003`08efd9c0 00007ffc`505a43b0 ntdll!RtlpEnterCriticalSectionContended+0xdc
05 00000003`08efd9f0 00007ffc`4152c02b ntdll!RtlEnterCriticalSection+0x40
06 (Inline Function) --------`-------- clr!UnsafeEnterCriticalSection+0x9
07 00000003`08efda20 00007ffc`4152bf48 clr!CrstBase::SpinEnter+0xac
08 00000003`08efda50 00007ffc`41525d33 clr!CrstBase::Enter+0x111
09 (Inline Function) --------`-------- clr!CrstBase::AcquireLock+0x8
0a 00000003`08efda90 00007ffc`415aefa2 clr!CrstBase::CrstHolder::CrstHolder+0x14
0b 00000003`08efdac0 00007ffc`418715e8 clr!SystemDomain::LockHolder::LockHolder+0x22
0c 00000003`08efdb00 00007ffc`419e4204 clr!AppDomain::UnloadById+0xb0
0d 00000003`08efdb70 00007ffc`3d7c5f60 clr!AppDomainNative::Unload+0xb4
0e 00000003`08efdcd0 00007ffb`e2443b47 mscorlib_ni!System.AppDomain.Unload(System.AppDomain)+0x40
0f 00000003`08efdd20 00007ffb`e2443b17 Ninject!Ninject.Modules.AssemblyNameRetriever.GetAssemblyNames(System.Collections.Generic.IEnumerable`1<System.String>, System.Predicate`1<System.Reflection.Assembly>)+0x107
10 00000003`08efdd80 00007ffb`e2443980 Ninject!Ninject.Modules.AssemblyNameRetriever.GetAssemblyNames(System.Collections.Generic.IEnumerable`1<System.String>, System.Predicate`1<System.Reflection.Assembly>)+0xd7
11 00000003`08efddf0 00007ffb`e243f1b4 Ninject!Ninject.Modules.CompiledModuleLoaderPlugin.LoadModules(System.Collections.Generic.IEnumerable`1<System.String>)+0x90
12 00000003`08efde40 00007ffb`e2437a90 Ninject!Ninject.Modules.ModuleLoader.LoadModules(System.Collections.Generic.IEnumerable`1<System.String>)+0x254
13 00000003`08efdeb0 00007ffb`e4a59ffa Ninject!Ninject.KernelBase..ctor(Ninject.Components.IComponentContainer, Ninject.INinjectSettings, Ninject.Modules.INinjectModule[])+0x210
14 00000003`08efdf10 00007ffb`e4223ac5 xyz_Services!Unknown+0x16a
15 00000003`08efe290 00007ffb`e420e1ba Quartz!Unknown+0x175..
Disclaimer: The information contained in this e-mail is private & confidential and may also be legally privileged. The contents of this mail and any of its attachments are subject to change without notice. If you are not the intended recipient of this mail, please notify us, preferably by e-mail; and do not read, copy or disclose the contents of this message to anyone. Whilst we have taken reasonable precautions to ensure that any attachment to this e-mail has been swept for viruses, e-mail communications cannot be guaranteed to be secure or error-free, as information can be corrupted, intercepted, lost or contain viruses. We do not accept liability for such matter or their consequences.