long-distance hops - latency doubles when msg exceeds 16kb

[Alon Kaftan]

Hi
we have a go grpc client running in US and a go grpc server on APAC.
Both on AWS.
on the same connection we make Unary calls
when the message size is lower than 16KB, roundtrip latency is 250ms
when the message size is crossing the 16KKB, round trip latency jumps to 450ms
when the message size is crossing the 100KB, round trip latency jumps to 650ms

Also, if we increase the rate of the 16KB+ messages from 1/sec to 2/sec or more, the latency drops to 250ms

we ruled out load balancers as we connected the client & server pods directly and observed the behaviour remains the same

Any ideas where to start with this kind of behaviour ?

Thanks!

[Alon Kaftan]

Also, if we run few dummy (small payload) calls per seconds, in parallel to the big payload calls, on the same connection => the latency of the big payload calls is reduced to 250ms as well
Thoughts ?

[Alon Kaftan]

HI again
We tcpdump the traffic.
We see the first 16kb frame is sent from the client to the server, but the 2nd one is not sent until an ack is returned from the server.
In our case, our msg size is 30KB, and round trip latency is ~220ms.
So because of this, call takes ~440ms instead of ~200ms.

We set the InitialWindowSize and InitialConnWindowSize to 1MB, on both server and client - no change.
We checked both unary RPCs and Stream RPCs - the same.
We set the WriteBufferSize and ReadBufferSize to zero (write/read directly to/from the wire) - 95% latency remained the same, but Avg latency dropped by 100ms - not sure why it had this effect.

Again, in all of the conditions above, if we increase to rate of messages to more then 2 or 3 per second, the latency drops to 450ms
Looking at http2debug=2 logs, we see it seems that in the higher rate the when latency is low, grpc somehow uses a previous opened stream used by another RPC to sent the new RPC....

Anyone has encountered a similar behaviour. ? 
Anyone that understand GRPC implementation (Go lang) can explain why it behaves this why and if there is a way to make it work better ?
Any help would be very much appreciated 
Thanks !

[Gabriel Fabri]

"gRPC uses the HTTP/2 default max size for a data frame of 16kb. A message over 16kb may span multiple data frames, whereas a message below that size may share a data frame with some number of other messages." - grpc.io

"The size of a frame payload is limited by the maximum size that a receiver advertises in the SETTINGS_MAX_FRAME_SIZE setting. This setting can have any value between 2^14 (16,384) and 2^24-1 (16,777,215) octets, inclusive." - RFC 7540

I hope it helps.