[Question] Using grpc compression weirdly leads to low throughput

Skip to first unread message

Hao Jiang

Jun 20, 2022, 4:20:02 PMJun 20
to grpc.io


I have a service using grpc to stream data to client-side.
For the original implementation, the average throughput could be ~110MBps with a 1Gbps NIC, so I think bandwidth itself is a bottleneck. The usual way is to enable compression/decompression.

I found grpc supports compression natively. The only line of code I added at server side is


with no change at client-side. But the throughput drops stably.

With dstat -n, the throughput looks like

33M 121k 2948B 1756B 306B 166B 3026B 688B

which looks like the average throughput is 7MBps.
And with response.ByteSize(), I calculated the throughput is ~11MBps, still way below the one before compression.

The observation doesn't change as the compression level, whether it's low, medium or high.

I want to know, if I'm using the compression in grpc wrongly?




Mark D. Roth

Jun 22, 2022, 2:13:54 PMJun 22
to Hao Jiang, grpc.io
It's hard to say what's going on here without knowing more about your environment.  A few things to consider:
  • Compression is negotiated between client and server based on what algorithms are supported by each one, as described in https://github.com/grpc/grpc/blob/master/doc/compression.md#compression-levels-and-algorithms.  You might want to check to see what compression is actually being negotiated on the wire.  In C-core, you can try setting the env vars GRPC_VERBOSITY=DEBUG and GRPC_TRACE=compression to enable additional trace logging for the compression code.
  • If your client and server are having CPU bottlenecks, then adding compression can slow things down, because it basically trades off network usage for CPU usage.
  • If your application is sending many small messages instead of fewer larger messages, then the CPU overhead of compression may be higher.
I hope this information is helpful.

You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/52e90489-0347-4614-824c-ceeac41c8086n%40googlegroups.com.

Mark D. Roth <ro...@google.com>
Software Engineer
Google, Inc.
Reply all
Reply to author
0 new messages