Maximum throughput with lettuce RedisAsyncCommands for PubSub and Set together?

[kpr...@thoughtworks.com]

We are measuring performance of PubSub with Lettuce client for redis. 

We are getting latency below 5ms but we are stuck at throughput of 25k messages. 

Is this the hard limit for throughput with lettuce client?

Our publish api does two things => 

```

commands.publish(event.eventKey, event).toScala

commands.set(event.eventKey, event)

```

here commands is RedisAsyncCommands

we are are measuring throughput & latency at subscriber side that is when subscriber receives event sent by publisher.

Our Setup is like this:

4 machines  (2 Publishers + 2 Subscribers) => c4.xlarge ec2 instances (8GB RAM, 4vCPU)

1 machine for running single redis server => c4.xlarge ec2 instances (8GB RAM, 4vCPU)

We have measured this for multiple combinations 

for ex.

1. 30 publisher's publishing at 1000 rate and 30 subscribers

2. 50 publisher's publishing at 1000 rate and 50 subscribers

3. 100 publisher's publishing at 1000 rate and 100 subscribers

4. 200 publisher's publishing at 1000 rate and 200 subscribers

In all the cases, throughput is around 24 to 28k messages/sec.

CPU usage and memory usage are minimal (less than 50%)

[Mark Paluch]

Lettuce has no hard limits or such. It's interesting to see such a stability in throughput. Asking differently: 

1. Why is this an issue? Did you expect different numbers?

2. Have you tried to benchmark with other clients (redis-cli, Jedis, netty directly)?

3. Currently, your Redis server remained unchanged across your tests. Have you tested to up/downscale the Redis instance itself?

Cheers, 

Mark

[kpr...@thoughtworks.com]

Thank you @mark for quick response, please find my comments inline :

1. Why is this an issue? Did you expect different numbers?

[Ans] Yes, We have a requirement to process 40-50k messages/sec. Wondering if single instance of redis with Lettuce client gives throughput of 24-28k.

If that is the case, what are my options? 

• Redis cluster?
  
• Distribute load across multiple redis instances?
  
• Use pipelining for 'SET' (i.e. flush in batches)?
• Is their any performance tuning I can do at lettuce client side or redis side?

2. Have you tried to benchmark with other clients (redis-cli, Jedis, netty directly)?

[Ans] I tried redis-cli. For 'SET' operation, with 100 client and 197 bytes payload size, it gives throughput of 75-85k (Without pipelining)

I could not find a way to test PubSub with redis-cli. Is their a way to measure PubSub with redis-cli?

3. Currently, your Redis server remained unchanged across your tests. Have you tested to up/downscale the Redis instance itself?

[Ans] Yes, I am planning to test with multiple redis instances but have not done that yet.


In general, how much throughput you get with lettuce client for PubSub and SET operation with single redis instance, any guess or approximate number would really help?

[Mark Paluch]

Depending on your setup, usage, and multiplicity of instances (Threads, JVMs, synchronous/asynchronous API usage), you can affect performance. 

We have a test suite [0] that is executed with our CI [1] build where we publish our benchmark results for various usage scenarios. Running that suite on my local machine yields about 20k operations per second which sounds comparable to what you get. However using the asynchronous API or enabling batch mode (caution, batch mode can harm your application if sharing a single connection across threads as this create shared mutable state) I can get up to 80k ops/second.

What also can have an impact is the choice of the transport layer. Using Java NIO and Kqueue (Mac/BSD) have slight differences.

NIO:

Benchmark                                    Mode  Cnt      Score       Error  Units

RedisClientBenchmark.asyncSet               thrpt    5  19571,822 ±   802,931  ops/s

RedisClientBenchmark.asyncSetBatch          thrpt    5  84575,644 ±  9093,265  ops/s

RedisClientBenchmark.asyncSetBatchFlush     thrpt    5  86720,673 ±  2088,154  ops/s

Kqueue:

Benchmark                                    Mode  Cnt      Score       Error  Units

RedisClientBenchmark.asyncSet               thrpt    5  20351,674 ±  1959,457  ops/s

RedisClientBenchmark.asyncSetBatch          thrpt    5  86349,665 ± 23276,377  ops/s

RedisClientBenchmark.asyncSetBatchFlush     thrpt    5  92066,123 ±  3582,439  ops/s

There aren't many things you can tune in Lettuce. You can tune netty to some extent (heap/direct buffers, I/O ratios, transport type) but these options don't have a huge impact on such small messages.

Cheers, 

Mark

[0] https://github.com/lettuce-io/lettuce-core/blob/5.x/src/test/jmh/io/lettuce/core/RedisClientBenchmark.java#L81

[1] https://travis-ci.org/lettuce-io/lettuce-core/jobs/370070255

[kpr...@thoughtworks.com]

Thanks @Mark

I did run the lettuce jmh benchmarks on my machine and got the same results.

And I think issue is with the number of connections I am creating with redis. I have posted question to redis-db mailing list on this.