Stream flow control on C++ sync API?

[tobias.krueger]

Hi, 

we have encountered some strange behaviors using the Write method of the sync API.

I have two effects that might be different, but perhaps some understanding about the sync API might help.

Both are related to slow / aborting connections.

1. I am trying send several hundred packets (each ~1MB) over a stream from a client to a server.
   The network is using a custom radio module with a very low bandwidth like Wi-Fi direct.
   Every call to  stream.Write(packet) returns immediately.
   Even the stream.WriteLast returns within some seconds.
   Watching with Wireshark I can see, that not even the first packet has left the sending computer. 
   That means all packets are buffered in my system.
   
   Is there any way using the sync API to control/limit the buffering?
   
   
2. In the opposite we encounter sometimes a blocking of the server's Write method when streaming very small packets to the client.
   This can be reproduced when I pull the client's network cable.
   Then the server's Write call is blocked for ~8 seconds

On the on hand I can flood the sync API and get no backpressure at all, on the other hand the Write call gets blocked for several seconds.

Any ideas to get a better understanding of the behavior?

Anything I can do without switching to the async API?

Thanks

Tobias

[apo...@google.com]

A few clarifications that might help to clarify here:

> I am trying send several hundred packets (each ~1MB) over a stream from a client to a server.

Do you know the exact number of packets, and the exact size of each packet?

> Every call to  stream.Write(packet) returns immediately.

Can we measure the amount of time it takes for stream.Write calls to complete in terms of microseconds or milliseconds?

 > Watching with Wireshark I can see, that not even the first packet has left the sending computer. 

This seems unlikely, as we should at least have seen the initial TCP handshake packets when the RPC was started. Are you sure that you're sniffing the correct network interface, and looking at the correct TCP connection?

[tobias.krueger]

I have to apologize for me vague first post.

I should have watched my logs more precisely - the buffering is not as big as I thought.

Every packet contains 3 integer values and 1 MByte (1024*1024) of bytes (==blob)

The timing looks as follows:

Write(Chunk#0)  : ~     1 msec : OK

Write(Chunk#1)  : ~     1 msec : OK

Write(Chunk#2)  : ~     1 msec : OK

Write(Chunk#3)  : ~   900 msec : OK

Write(Chunk#4)  : ~     1 msec : OK

Write(Chunk#5)  : ~20.000 msec : fails

Write(Chunk#6)  :     < 1 msec : fails

Write(Chunk#7)  :     < 1 msec : fails

            :

            :

Write(Chunk#n)  :     < 1 msec : fails

Watching on Wireshark the sending starts as soon as the first chunk is written. 

The 6th call sending chunk #5 actually timeouts after 20 seconds, while the first chunk is still transmitting.

The transmission of the first chunk takes a total time of 70 seconds (seen in Wireshark).

Currently I see no way to slow down my sending client.

From Sync API I see no way to recognize that I should slow down to avoid the timeout on Chunk#5.

Is there a knob where I can increase the 20 seconds timeout?

Is there a way to get a more detailed error message - not only a bool indicating false?

Any other good ideas?

Thanks

[yas...@google.com]

What timeout are you referring to here? Some logs might be helpful here.

If by chance, you are referring to keepalives then, `GRPC_ARG_KEEPALIVE_TIME_MS` and `GRPC_ARG_KEEPALIVE_TIMEOUT_MS` are the corresponding channel args.