Slow Redis Cluster performance when getting large (>100kB) objects

[Ian Trase]

I'm trying to use Redis Cluster as a backing cache for a primary datastore (Cassandra). I've noticed that, while Redis is significantly faster than Cassandra for fetching small objects, it falls behind for objects larger than 100kB.

My questions are:

Is Redis or Redis Cluster known to have performance issues when dealing with large objects?
Are there ways to optimize Redis Cluster for the storage of primarily large objects?
Is this indicative of a weakness in Redis or a known strength in Cassandra - does it line up with benchmarks comparing the two services?
Based on Yourkit profiling, the slowness seems to occur during either the network connection between Redis and the client or within Redis itself, not on the client side. Results from iperf show normal throughput for a 1Gb/s connection, and latency between the machines is sub-millisecond. SLOWLOG shows nothing out of the ordinary.

Some facts about my Redis Cluster: 18 instances spread out across 3 VMs with a replication factor of 1 (so 9 masters total and 6 instances on each VM). Each VM has 32GB of memory, and each instance uses 5GB. Each VM has 4 cores. Both forms of persistence (RDB and AOF) are disabled. Transparent Huge Pages are disabled.

Below I've included a table illustrating how throughput falls off with increasing object size. Tests on both Cassandra and Redis were run single threaded with a single connection for each.

Redis throughput: 0.003537595276169634 MB/s, Cassandra throughput: 0.002188710038885119 MB/s, object size in bytes: 10, throughput ratio: 1.6162923426675595

Redis throughput: 0.03797119469182343 MB/s, Cassandra throughput: 0.02551391762038427 MB/s, object size in bytes: 100, throughput ratio: 1.4882541856875189

Redis throughput: 0.3897608379431817 MB/s, Cassandra throughput: 0.2901892901215633 MB/s, object size in bytes: 1000, throughput ratio: 1.3431261980065041

Redis throughput: 2.7464192980834445 MB/s, Cassandra throughput: 2.5417614833282105 MB/s, object size in bytes: 10000, throughput ratio: 1.080518103723585

Redis throughput: 7.020086815887924 MB/s, Cassandra throughput: 12.797994158989074 MB/s, object size in bytes: 100000, throughput ratio: 0.5485302406515904

Redis throughput: 19.84828274884104 MB/s, Cassandra throughput: 44.10034458329361 MB/s, object size in bytes: 1000000, throughput ratio: 0.45007092203901056

The only noticeable difference in the network configuration is that the Cassandra cluster is composed of physical machines whereas the Redis cluster is composed of VMs. But it seems like that shouldn't account for a 50% slowdown, especially when Redis is reading from memory and Cassandra is (at least occasionally) reading from disk.

Any help would be appreciated!

[Didier Spezia]

First, do not underestimate the impact of a virtualized environment on

network throughput. It is significant, especially if your hypervisor

or the Linux kernel are not correctly tuned.

A 50% overhead is what I could routinely measure with redis-benchmark

on ESX VMs a few years ago. (i.e. half the throughput on a VM, compared

to an equivalent physical box).

That said, Redis is clearly not optimized to serve large objects.

On one side, the execution of all commands has to be atomic.

On the other side, transferring 1 MB of data on the wire can take time.

So Redis has to buffer the output of the command.

If the command is a GET on a 1 MB object, it means a 1 MB object will be copied

first in the output buffer, before being progressively copied in the socket buffer

while the client is ingesting the reply. This extra copy is needed because the

object can be updated just after the GET command by another session. It is

not enough to refer to the object in the output buffer, Redis has to copy it.

I don't know how Cassandra works, but perhaps it is able to avoid this copy by

referring to immutable objects.

Regards,

Didier.

[Josiah Carlson]

Lies, damn lies, and benchmarks. You are not testing what you think you are testing.

For the small object size performance, you're not testing Redis' network performance against Cassandra's, you're comparing command latency, which for Cassandra is likely limited by disk latency. Somewhere along the line, network bandwidth usage/buffer copies/VM slowdowns/overloading (thanks to 6 instances on 4 cores)/cosmic rays/etc. overwhelmed whatever latency advantage Redis had by being in-memory, and performance was no longer limited by initial disk seek time for Cassandra.

Run Redis on non-virtualized hardware, or run Cassandra in a virtualized environment. Then you've at least excluded *that* from being a potential factor. If you're not interested in benchmarking the two, because you already know you're going to run Redis in a VM and Cassandra on bare hardware, then it's likely you'll observe exactly what you are seeing in practice, and have to decide how you want to handle it.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.