Strange bidi streaming behavior

[solomon lifshits]

Occasionally observing following issue:

0. server request a regular(non-streaming) async rpc call

1. client performs a regular(non-streaming) async rpc call corresponding to server's request from step 0,

2. server sends back the response and requests another async bidi streaming rpc call, sees ok on CQ (new stream object(serverasyncreaderwriter) is created prior the request).

3. as a result of  server's response client makes  async bidi streaming rpc call corresponding to server's request from step 2.  Sees ok true on CQ.

4. both client and server start reading from the stream. client also successfully writes  to the stream few times sequentially(no concurrent writes, 1 outstanding write at given moment).

5. On server side read fails instantly(ok=false) , on client side read is pending.

6. Server calls 'Finish' on the stream object and destroys it upon seeing the tag on CQ(ok=false again). Client side has no impact. 

7. Client performs the same step as in 1.

8. Server performs the same step as in 2 including creating new stream object and making rpc request.

9. Client is not making new streaming call as it has existing stream which is still valid from client's prospective.

10. Server reads from the new stream and sees the data, client sent in step 4. Seems that its last request(step 8) was matched to client's call from step 3.

As mentioned this behavior is observed occasionally once in few days of continuous run. Usually server's request from step 2 is successfully matched to client's rpc call from step 3 and server's read from step 4 succeeds. 

Could anyone explain why I am facing this behavior?

Thank you in advance.