High latency while concurrent unary rpc over same connection(grpc-go)

[jana...@dgraph.io]

Hi,

I wrote a test to check grpc latency for unary RPC on my machine. 

Link to the code: https://play.golang.org/p/YgS_xdzN_t

I noticed that the latency was very high, around 30-40ms when i make 2000 concurrent unary rpcs with payload of 1kb. 

But if i do only one rpc at a time then the latency was only 100 us. I noticed that the latency dropped down to around 1ms if

i do only 200 concurrent requests at a time or increase the payload size to 10kb.

I noticed that data is not flushed when there are concurrent writers in http_client.go, which might be adding to the latency.

We care more about the latency than the throughput for some use cases. Are there any options which we can specify on the client/server side to avoid these latency issues.

[Menghan Li]

Tracked in issue: https://github.com/grpc/grpc-go/issues/1512