Assertion failure with C++ in perform_stream_op_locked

[michi....@crowdstrike.com]

I have a simple C++ grpc client that talks to a grpc echo server written in Go via a bidir stream. The messages that are sent back and forth are binary blobs around 200 bytes in size.

On the C++ client side, I have a reader thread and a writer thread. The reader sits in a loop and calls a blocking Read() on the stream from the server and consumes messages; the writer waits for some data to arrive from an external source and writes it to the stream before waiting for some more data to arrive. The channel and stream are immutable and created before the two threads are created. The only concurrent calls into the grpc runtime are the calls to Read() and Write() on the stream. I'm not calling Read() concurrently with another Read(), or Write() concurrently with another Write(). At any one time, there is at most one blocking Read() and one blocking Write() call in progress, each made from the same single reader or writer thread each time.

All this works fine. I can move around 51 k messages per second, and exchange millions of messages without any issue. The whole process is basically in steady state. No connection is lost or the like, the channel and stream are never re-initialized, etc. Everything stable and perfect.

Then, after 15 or 20 minutes of this, I get the assertion failure below. I am certain that this isn't something stupid, such as memory corruption or the like. The code runs clean with address sanitizer and doesn't leak.

All this is with grpc and my own code compiled with gcc-7.3.0 on Ubuntu 16.04.

I have two questions:

1) Does the stack trace below ring any bells? All other threads are where I would expect them to be, such as in a poll waiting for data to arrive and similar.

2) I compiled grpc and my own code with thread-sanitizer. When I run the code, tsan reports a bunch of race conditions. Are these possibly related to the crash?

Thanks,

Michi.

Stack trace: https://www.dropbox.com/s/n2494rjnhj0hq56/stack-trace.txt?dl=0

Tsan output: https://www.dropbox.com/s/7fkxdu3rmcji6eo/tsan.txt?dl=0

[yas...@google.com]

This could be a potential bug. Could you file an issue on github about this? Please also provide your reproduction code.

[michi....@crowdstrike.com]

Issue opened here: https://github.com/grpc/grpc/issues/16876