Performance advice on bi-directional streaming

[Mengyao Li]

Hi,

I'm using golang gRPC library and have set up bi-directional streaming between client and servers (both golang). We saw a big chunk of our profile (30%) spent on Syscall on the sever (see profile image posted below), which is not super surprising because the stream is indeed busy (at peak 25k protos per second on Send and 50k protos per second on Recv). However we are wondering if that can be optimized in any way. The proto messages that we Send/Recv are very small (tiny), so I tried setting higher WriteBufferSize and ReadBufferSize on the server side trying to reduce the number of syscalls but that does seem to have any effect on the profile.

Any hints on how I could dig deeper into the profile / what's happening behind the scene and whether our usage of streaming has too much overhead? Please let me know if more details is needed.

Thanks,

Mengyao

[dfa...@google.com]

Sorry for the late reply - in grpc-go, we do batching of sends/receives by default.  Setting the Read and Write buffers larger can potentially help, but probably is only useful if your messages are large.  In your case, I don't believe there's much else you can do to optimize.

Is your primary concern the CPU utilization or are you seeing lower throughput than desired?

Doug