Optional compression for redis protocol

[Andrew Armstrong]

Hello Salvatore and community,

It would be good if Redis supported a "COMPRESSION" command that could be sent by a client to allow bi-directional compressed communication between the client and server, improving response times and request latency.

This would reduce network communication overhead between application nodes talking to the Redis nodes and sending/receiving commands and data sets.

As this is a text protocol the compression should achieve good results.

Thoughts?

Regards,

Andrew

[Marc Gravell]

Some random thoughts on this (from just a random redis user):

a: the binary redis protocol is not really a text protocol (and indeed, keys do not need to be text)

b: this would primarily impact bandwidth, not latency

c: in most cases, I would expect the effect to be minimal - the only time this will make a significant difference is for keys with large values in them (blobs etc) - in which case a pragmatic solution might be to compress at origin (which is what I do in my usage)

d: you could always rename the commands, so "get" is "g" etc ;p

I think it is an interesting idea, but it isn't something that I would automatically assume is going to have as big an impact as you might think. Also, note that a lot of times "compression" makes things bigger. This is especially the case if you have small frames, which would be quite a lot of the time in redis messages (unless you are sending a batch).

Marc

[Salvatore Sanfilippo]

+1 to Marc to summarize so well what I also think about this issue.

A few more notes: latency will be worse actually not just unaffected.
Compressing large values client-side which accounts for a large part
of what is compressible will also result in less space used
server-side, and has the other advantage of being a distributed
computation (among the clients) automatically instead of requiring the
server to do the decompression work for all the clients.

I don't believe protocol compression would really improve Redis in
most use cases, but in the WAN replication specific case where you can
use anyway a proxy for encryption obtaining as a side effect
compression as well.

Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

To "attack a straw man" is to create the illusion of having refuted a
proposition by replacing it with a superficially similar yet
unequivalent proposition (the "straw man"), and to refute it
— Wikipedia (Straw man page)

[Josiah Carlson]

There are a few compression algorithms (mainly LZ4) that are fast enough that the latency of compression *might* be low enough to actually offer an improvement in overall latency, at least under higher network utilization scenarios (if the network is otherwise busy). But this is a fairly rare case in all but WAN scenarios. Though at least on the LZ4 side of things, the very high decompression speed would significantly reduce the chances of the server side suffering slowdowns if compression was only enabled for the client -> server direction.

 - Josiah

[김지욱]

I think this is a good idea. Lately, I sent multiple redis commands with the pipeline feature
in the client and I confirmed that there are three tcp packets to send all redis commands.
If redis support optional compress feature on the protocol, the number of packets can be decreased.
Therefore, this could impact not only bandwidth, but also latency.



2014년 1월 16일 목요일 오전 2시 18분 56초 UTC+9, Josiah Carlson 님의 말:

[Matt Palmer]

On Sun, May 01, 2016 at 07:20:57AM -0700, 김지욱 wrote:
> I think this is a good idea. Lately, I sent multiple redis commands with
> the pipeline feature
> in the client and I confirmed that there are three tcp packets to send all
> redis commands.
> If redis support optional compress feature on the protocol, the number of
> packets can be decreased.

The number of packets is unrelated to whether compression is available. How
many packets get sent is entirely a function of your client. If you want
one packet for three commands, make your client do that. Compression has
nothing to do with that.

- Matt

[Marc Gravell]

The number of packets is usually a separate issue, since it would be unusual for 3 simple Redis commands to not fit in one packet. Adding compression won't help at all if the specific client you are using isn't designed to minimise packets. However, you mention latency: actually, the reverse is true. Sending packets eagerly reduces latency. Holding packets to avoid packet fragmentation increases latency. Adding processing increases latency. If you want the minimum latency: you send lots of packets. If you want maximum bandwidth: you worry about packet fragmentation. They are necessarily competing requirements.

Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[김지욱]

Why not? Compression is related to the number of packets.

This is situation what I imagine. If a client sends commands like below.

MSET a_x 1, b_x 1, c_x 1, d_x 1, e_x 1, f_x 1

INC a_x

INC b_x

INC c_x

INC d_x

INC e_x

INC f_x

MGET a_x, b_x, c_x, d_x, e_x, f_x

PFADD hk_x 1 b b_1 c d e f j i kl

PFADD hk_y j j j j k k k k k k x x x x

PFCOUNT hk_x

PFCOUNT hk_y

If the client has to send a bunch of commands like above, it definitely over MTU size.

However, if Redis supports compression feature before sending commands, a size of data will be decreased.

Is there any other solution to solve this ?

- initproc

2016년 5월 2일 월요일 오전 7시 55분 52초 UTC+9, Matt Palmer 님의 말:

[김지욱]

My goal is not to send in one packet. Through the compression, let's reduce the length of data and send it.

Then, you can get shrink data and the number of packets could be reduced since data is smaller. You can minimize a possibility of TCP fragmentation.

2016년 5월 2일 월요일 오전 9시 1분 20초 UTC+9, Marc Gravell 님의 말: