In your first attempt, you're correctly setting the grpc-accept-encoding header. I don't know the Java gRPC APIs, but it sounds like you're compressing the response correctly. Unfortunately, when the browser receives the response, it doesn't know how to decompress it and you're getting an error. The response can't decompress with standard gzip tools because it isn't valid gzip: following the gRPC protocol, each gzipped protobuf message is wrapped in five bytes of enveloping metadata. Because of the enveloping and the blob of gRPC-Web trailers at the end of the body, the response as a whole isn't valid gzip.
Your second attempt has the same problem. The correct response header is "Content-Encoding" (rather than "Content-Encryption"), but you end up with the same problem - gRPC's per-message compression doesn't produce the sort of compressed response that browsers expect.
Hope that helps,