grpc-java server-side compression not working/implemented?

[Stephen Haberman]

Hi,

I'm using grpc-java 1.0.0.pre2, and it seems like server-side/response compression may not be working.

I'm making a synchronous call from a client, and sending ~40k objects, e.g. FooRequest.newBuilder().addAllPaths(...40k...).build().

The server then collects it's own 40k objects (again paths) and returns it as the response to the client's request, e.g.

      responseObserver.onNext(FooResponse.newBuilder().addAllPaths(...40k...).build());

      responseObserver.onCompleted();

These 40k objects are all basically file paths + a long modification time, and there will be a lot of duplication in the paths (e.g. /home/.../ in all of them), so compression should really help with the payload size, so I'm turning on gzip:

        Channel c = NettyChannelBuilder.forAddress(host, port).negotiationType(NegotiationType.PLAINTEXT).maxMessageSize(maxMessageSize).build();

        MirrorStub stub = MirrorGrpc.newStub(c).withCompression("gzip");

However, nethogs shows that the client request is ~400kb, but the server's response is ~5mb. Even though in theory, the request and response should be exactly the same size (the response is responding with essentially the same input the client sent).

Basically it looks like the server is not responding with a gzip'd response.

I'm not registering any decompression factory with the server, but AFAICT gzip is one of the defaults:

      ServerImpl rpc = NettyServerBuilder.forPort(port).maxMessageSize(maxMessageSize).addService(new MirrorServer()).build();

Before I go digging, is this expected? That the server's response would not be compressed?

Thanks,

Stephen

[Stephen Haberman]

Hi,

I've verified that ServerCallImpl has compressor=null. There is a setCompression method, but AFAICT no one calls it, and it's not immediately clear how I should go about calling it in my user code.

Naively, I'm wondering why somewhere around here:

https://github.com/grpc/grpc-java/blob/master/core/src/main/java/io/grpc/internal/ServerCallImpl.java#L85

When we recognize the client has sent us a compressed request, that we don't also assume they can handle/desire a compressed response, and just set the compressor to the matching encoding.

Then users could still call serverCall.setCompression(...) to change that (however you do that), but using the same encoding in the response as the request seems like a more reasonable/less surprising default?

Thanks!

- Stephen

[Stephen Haberman]

Okay, apologies for the self-replies, but I've got it working with an interceptor to turn it on explicitly:

  public static ServerServiceDefinition createWithCompressionEnabled() {

    return ServerInterceptors.intercept(new MyServer(), new ServerInterceptor() {

      @Override

      public <ReqT, RespT> Listener<ReqT> interceptCall(ServerCall<ReqT, RespT> call, Metadata headers, ServerCallHandler<ReqT, RespT> next) {

        call.setCompression("gzip");

        return next.startCall(call, headers);

      }

    });

  }

This cut my ~5mb response to ~500kb (kudos to `nethogs -v 3` to watch cumulative process network traffic), which exactly matches the client-sent compression with essentially the same payload.

I naively still think it'd be cool/desirable for the server to automatically match the client's compression, but IANAE. :-)

Thanks!

- Stephen

[ccl...@dieselpoint.com]

You should post an issue in the issue tracker.

Good catch, by the way.

[Stephen Haberman]

Cool, issue filed here:

https://github.com/grpc/grpc-java/issues/2196

Thanks!

- Stephen

[Eric Anderson]

I also filed an issue for making an example for server-side.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CANnRKuU6mvKvt4kUAJ9a97etubQZJwmjduDqHA0tJGtuPgd3sQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Newbie_GRPC]

where does this change should go? Even we are facing issue with heavy payload and would like to resolve it.

[Stephen Haberman]

Not sure if it's the canonical way, but here is an example of using an interceptor to enable server-side compression:

https://github.com/stephenh/mirror/blob/master/src/main/java/mirror/MirrorServer.java#L39

- Stephen