Grpc Bidirection streaming - IServerStreamWriter::WriteAsync is unable to send anything and throws error: InvalidFlags

[Farhan]

Hi Grpc experts,

I have a Bidirectional streaming application where sometimes server stream is unable to send any messages to any clients. The error message that I get is below. I have tried closing server's ReadStream in order to make client re-establish the connection, but it doesn't help with the error and the issue remains. The IServerStreamWriter::WriteAsync API doesn't take any flags from the caller so I have no idea which flag is invalid.

(Inner Exception #0) System.InvalidOperationException: Call error: InvalidFlags at Grpc.Core.Internal.CallErrorExtensions.CheckOk(CallError callError) at Grpc.Core.Internal.CallSafeHandle.StartSendMessage(ISendCompletionCallback callback, Byte[] payload, WriteFlags writeFlags, Boolean sendEmptyInitialMetadata) at Grpc.Core.Internal.AsyncCallBase`2.SendMessageInternalAsync(TWrite msg, WriteFlags writeFlags) at Grpc.Core.Internal.ServerResponseStream`2.WriteAsync(TResponse message)

I'm hoping to get some pointers on what might be happening?

Thanks,

Farhan

[Farhan]

Some more info on the issue. I looked at Grpc library code in order to get some clarity on the issue. We are using Grpc v2.23 whose commit id is 2b5f41f0be193d97467c3ec5280e269a49d75f8c.

[1] is the code that throws the exception. It looks at flags and checks it against expected values [2]. The code [3] that provides the flag to the low level code sets it to default (0) if none is provided by the caller. We have tried both cases i.e. (1) setting the value to one of expected values or (2) not setting at all. Both cases cause the same exception. I'm unable to reason as to why would library throw an exception

[1] https://github.com/grpc/grpc/blob/2b5f41f0be193d97467c3ec5280e269a49d75f8c/src/core/lib/surface/call.cc  (line: 1644)

[2] https://github.com/grpc/grpc/blob/2b5f41f0be193d97467c3ec5280e269a49d75f8c/include/grpc/impl/codegen/grpc_types.h (line: 445)

[3] https://github.com/grpc/grpc/blob/2b5f41f0be193d97467c3ec5280e269a49d75f8c/src/csharp/Grpc.Core/Internal/ServerResponseStream.cs (line: 64)

We are using .Net v4.5.2.

Instances where we see the exception are running Windows2019 vs instances where we don't the issue are running Windows2016.

Hoping to get some pointers from the grpc experts community.

Thanks,

Farhan

[Jan Tattermusch]

Hi, 

>  I have tried closing server's ReadStream in order to make client re-establish the connection, but it doesn't help with the error and the issue remains

- Once a call is created, it will always be on the same connection and things like "closing server's ReadStream" would do nothing about it. To retry the call you need to cancel the existing one and start a new call.

- would GRPC_VERBOSITY=debug and potentially GRPC_TRACE=see docs give you any more insigths? (see https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md)

- gRPC C# 2.23 is quite old, I'd recommend upgrading to a newer version

- If you had a reproduction (runnable minimal repro in form of e.g. a github repository), it would be helpful (since like this isn't unclear how exactly you configure your calls).

The "InvalidFlags" error comes from the native C-core layer:

https://github.com/grpc/grpc/blob/df3ce2091354762571bff832558110a40af69e2f/include/grpc/impl/codegen/grpc_types.h#L484

it indicates that something is quite wrong about how the calls are invoked. Did you use some special configuration settings for your calls/writes?