Replication stream compression

[News Group]

I'm looking to run redis replication across data centres and I was looking at options to compress the stream between master and slave, but the last post I saw on it was about a year old.

Has there been any progress in this area or are people still using external programs to compress the stream?

[Josiah Carlson]

External programs are still the standard solution.

SSH + compression is still about the best solution available, as you even get encryption.

You can set the compression level for SSH on the command line, I'd stick with levels 1-5, which will compress fairly well, but will use significantly less processor than levels 6-9. At levels 6-9, you may not be able to compress fast enough for a high-throughput application.

 - Josiah

On Thu, May 16, 2013 at 12:45 PM, News Group <newsgro...@gmail.com> wrote:

I'm looking to run redis replication across data centres and I was looking at options to compress the stream between master and slave, but the last post I saw on it was about a year old.

Has there been any progress in this area or are people still using external programs to compress the stream?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Salvatore Sanfilippo]

+1 for what said Josiah, and I really have nothing to add if not the
perspective from the point of view of development and roadmap, that
is, why Redis is not likely at the moment (but I'm a famous
idea-changer...) to implement it.

1) It's a network layer problem, it's a shame to solve everything
inside Redis, there are good enough solutions after all, from VPN
software that is trivial to deploy and uses the tun/tap interface, to
user-level proxies (like ssh), and so forth.

2) Most over-the-wan problems of this kind also *require* encryption
otherwise your data is traversing the internet without protection.

3) Replication over the WAN is not a very common problem, so the cost
of a proxy / VPN setup is not high globally speaking. I mean, if you
need it, you can afford implementing that in your servers as you are
doing interesting stuff like cross-datacenter replication / disaster
recovery or alike.

Now since implementing encryption is remarkably harder if we don't
want a simple symmetric encryption, this means we need to link against
an ssl library, and then implement certificate handling, new config
options, and so forth. It's really not worth it IMHO.

Cheers,
Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Jay A. Kreibich]

On Fri, May 17, 2013 at 12:34:22AM +0200, Salvatore Sanfilippo scratched on the wall:

> +1 for what said Josiah, and I really have nothing to add if not the
> perspective from the point of view of development and roadmap, that
> is, why Redis is not likely at the moment (but I'm a famous
> idea-changer...) to implement it.

Just to clarify, I assume the initial synchronization process uses the
compressed dump format?

-j

--
Jay A. Kreibich < J A Y @ K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
but showing it to the wrong people has the tendency to make them
feel uncomfortable." -- Angela Johnson

[News Group]

On Thursday, 16 May 2013 18:34:22 UTC-4, Salvatore Sanfilippo wrote:

2) Most over-the-wan problems of this kind also *require* encryption
otherwise your data is traversing the internet without protection.

I'm actually sending the traffic through an IPSec tunnel - several of them in fact - but I'm looking to cut down the amount of data I'm sending with compression.  The main issue I have is that I'm running hundreds of redis instances in a twemproxy cluster, so having a separate application running for each instance is a lot of overhead and adds a lot of complexity, so having something in redis to do the compression of the replication stream would make things a lot simpler.

Thanks for the info, though!

[Josiah Carlson]

Hundreds of destination Redis servers is a little high on the SSH side of things. Usually that's my recommendation when replication is involved. It's a pity that ipsec (and the other vpn solutions) don't support compression, or when they do (like openvpn), they suck.

As a practical matter, it's entirely possible that your large number of connections may not be able to be reasonably sustained without dedicating a fairly large number of cores on the compression side of things. Would setting up local read-only replicas (which could have compression enabled over the link) be viable?

 - Josiah

[News Group]

On Friday, 17 May 2013 03:56:49 UTC-4, Josiah Carlson wrote:

As a practical matter, it's entirely possible that your large number of connections may not be able to be reasonably sustained without dedicating a fairly large number of cores on the compression side of things. Would setting up local read-only replicas (which could have compression enabled over the link) be viable?

The basic setup is 32-core boxes running 20 redis instances each, which leaves plenty of cores for compression.  To support the global distribution, there is 10 local read/write and 10 remote read-only replicated from the other DCs.

I can probably use a single wanproxy instance per box.

[Josiah Carlson]

Wanproxy looks like it will work, though their compression scheme is somewhat undocumented, and makes me think that it might not be as effective as zlib for Redis. For replication at least, 10x improvements in bandwidth usage have been observed for zlib over SSH: http://3scale.github.io/2012/07/25/fun-with-redis-replication/ .

 - Josiah

[Salvatore Sanfilippo]

On Fri, May 17, 2013 at 1:49 AM, Jay A. Kreibich <j...@kreibi.ch> wrote:
> Just to clarify, I assume the initial synchronization process uses the
> compressed dump format?

It does, and there is some idea to switch the AOF / replication
protocol to something binary that is much more space efficient, but
I'm still not sold on the idea.

Salvatore