GRPC Crash

[Dalip Choudhary]

Hi Team

We have an application which is using GRPC. Sometimes application working fine and sometimes whole system got stuck and memory consumption is about 95% and around 1340 threads active there.

When we see Dump file it is pointintg some grpc things. I am not able to findout what is wrong. It would be great help if you guys can help me.

In MCDeplSvr.exe.348.dmp the assembly instruction at ntdll!RtlpWaitOnCriticalSection+97 in C:\Windows\System32\ntdll.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to write to memory location 0x00000024 on thread 283
Please follow up with the vendor Microsoft Corporation for C:\Windows\System32\ntdll.dll
Detected possible blocking or leaked critical section at 0x004602a0 owned by thread 688 in MCDeplSvr.exe.348.dmp

Thread 283 - System ID 5496

Entry point  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+272eefCreate time  2021-05-27 11:28:34 AMTime spent in user mode  0 Days 00:00:01.859Time spent in kernel mode  0 Days 00:00:02.984

This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.


Call Stack

  Sourcentdll!NtWaitForMultipleObjects+14  KERNELBASE!WaitForMultipleObjectsEx+ef  KERNELBASE!WaitForMultipleObjects+e  kernel32!WerpReportFaultInternal+4b0  kernel32!WerpReportFault+73  KERNELBASE!UnhandledExceptionFilter+23b  ntdll!RtlUserThreadStart$filt$0+38  ntdll!_C_specific_handler+96  ntdll!RtlpExecuteHandlerForException+d  ntdll!RtlDispatchException+373  ntdll!KiUserExceptionDispatch+3a  ntdll!RtlpWaitOnCriticalSection+97  ntdll!RtlpEnterCriticalSectionContended+cc  ntdll!RtlEnterCriticalSection+40  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+273c6e  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+16f77a  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+171a82  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+de464  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+d61ef  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+d7543  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+71272  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+7104b  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+8bbd7  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+8c0c8  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+8c1c1  grpc_csharp_ext_x64!grpcsharp_auth_context_property_iterator+272fa8  kernel32!BaseThreadInitThunk+14  ntdll!RtlUserThreadStart+21

Impact of this lock

0.08% of threads blocked

(Threads 1008)

The following functions are trying to enter this critical section
ntdll!RtlpReAllocateHeap+3fe

The following module(s) are involved with this critical section
C:\Windows\System32\ntdll.dll from Microsoft Corporation
One or more of the blocking functions is a heap function. In most cases this implies leaked critical section caused by heap corruption. Please follow the steps outlined in the following Knowledge Base article:
300966 Howto debug heap corruption issues in Internet Information Services (IIS) 

Dump file section i am attaching here. please let me nknow why it is happening ? any help would be great on same.

[Jan Tattermusch]

Hi,

since this looks like a potential bug, please consider filing one under https://github.com/grpc/grpc/issues/new/choose (issues like this are best solved when there a github issue to reference)

From the stack trace I can't really tell what's wrong (but the function referenced from the call stack is here: https://github.com/grpc/grpc/blob/da10b795e17758ab19f75678d83933c800515a94/src/csharp/ext/grpc_csharp_ext.c#L1194)

If you could provide a way to reproduce the problem (e.g. a ready-to-run project as a github repo), it would be very helpful.