String deduplication in grpc-go RPC message stream

[Giedrius Statkevičius]

Hello,

We have a stream of messages with map<string, string>. It is guaranteed that a lot of strings repeat themselves over multiple messages. I would like to avoid paying the price of allocating multiple copies of the same string in the same stream. Perhaps there is some easy way of implementing that on the gRPC level?

We could hook into `proto.Marshaler` but it's not possible to pass custom parameters there and if there were then there is no way to create a per-stream string/symbol table. Ideally, it wouldn't be a global symbol table because streams typically have very different strings.

Has anyone done this?

Thanks in advance,
Giedrius

[Michael Lumish]

Can you clarify what language you are using? Have you tried using compression?