C# Grpc.Core - how to debug lock in grpc communication

[Marius H]

Hi, we are currently using Grpc.Core in our product for interprocess communication of two programs running on the same system. However, one of our customers reports occasional hangs of one of the processes on multiple of their systems. It is specific to this customer, we haven't had any problems reported from others.

We have received the dump files, and indeed the main thread is stuck in a synchronous grpc request. There are no grpc requests pending on other threads.

I'm lost on how to debug this issue, can someone please give me any pointers on how to proceed?

This is the callstack:

     ntdll!ZwWaitForAlertByThreadID+0xc
    KERNELBASE!SleepConditionVariableCS+0x20
    grpc_csharp_ext_x86!grpcsharp_insecure_channel_create+0x16939c
    grpc_csharp_ext_x86!grpcsharp_insecure_channel_create+0x13b7b0
    grpc_csharp_ext_x86!grpcsharp_insecure_channel_create+0x61e94
    grpc_csharp_ext_x86!grpcsharp_insecure_channel_create+0x33c18
    Grpc.Core.dll!Grpc.Core.Internal.CompletionQueueSafeHandle.Pluck(System.IntPtr tag)    Unknown
     Grpc.Core.dll!Grpc.Core.Internal.AsyncCall<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>.UnaryCall(###.Grpc.Contracts.### msg)    Unknown
     Grpc.Core.dll!Grpc.Core.Calls.BlockingUnaryCall<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(Grpc.Core.CallInvocationDetails<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> call, ###.Grpc.Contracts.### req)    Unknown
     Grpc.Core.dll!Grpc.Core.DefaultCallInvoker.BlockingUnaryCall<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(Grpc.Core.Method<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> method, string host, Grpc.Core.CallOptions options, ###.Grpc.Contracts.### request)    Unknown
     Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall.AnonymousMethod__3_0<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(###.Grpc.Contracts.### req, Grpc.Core.Interceptors.ClientInterceptorContext<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> ctx)    Unknown
     Grpc.Core.Api.dll!Grpc.Core.Interceptors.CallInvokerExtensions.MetadataInterceptor.BlockingUnaryCall<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(###.Grpc.Contracts.### request, Grpc.Core.Interceptors.ClientInterceptorContext<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> context, Grpc.Core.Interceptors.Interceptor.BlockingUnaryCallContinuation<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> continuation)    Unknown
     Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(Grpc.Core.Method<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> method, string host, Grpc.Core.CallOptions options, ###.Grpc.Contracts.### request)    Unknown
     Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall.AnonymousMethod__3_0<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(###.Grpc.Contracts.### req, Grpc.Core.Interceptors.ClientInterceptorContext<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> ctx)    Unknown
     Grpc.Core.Api.dll!Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(###.Grpc.Contracts.### request, Grpc.Core.Interceptors.ClientInterceptorContext<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> context, Grpc.Core.Interceptors.Interceptor.BlockingUnaryCallContinuation<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> continuation)    Unknown
     Grpc.Core.Api.dll!Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult>(Grpc.Core.Method<###.Grpc.Contracts.###, ###.Grpc.Contracts.MCEventResult> method, string host, Grpc.Core.CallOptions options, ###.Grpc.Contracts.### request)    Unknown

[Jan Tattermusch]

It is hard to give a universal guidance on "how do I debug XYZ". But here are some general tips:

- make sure you have the pdb symbol files from https://www.nuget.org/packages/Grpc.Core.NativeDebug that match the exactly the gRPC version you're using. This will help you see more useful info in the visual studio debugger.

- make sure this isn't an issue already filed by someone else (https://github.com/grpc/grpc/issues)

- make sure you're using the latest version of Grpc.Core (as of now that's 2.46.5), since you might be experiencing a problem that was already fixed by someone

- consider using the grpc-dotnet implementation since Grpc.Core is now in maintenance mode.