Use connection pool or not with redis cluster when having a mix of requests

[shob...@quickride.in]

Hi,

I want to try POC on Lettuce to connect to a redis cluster running in my cloud fronted by an internal load balancer.

I am not sure if I have to use connection pool or not. I have read some posts in this forum as well as on SO and github, but couldn't decide on the best approach since :

a) We have a mix of operation types - some use cases just set/get one single entity to/from the cache. Some other use cases set/get in bulk (multiple keys without using key hash). About 70%-80% of the times, each thread would set/get just one entity at a time. For the remaining 20%-30% of the times, there are threads that would start bulk write or bulk read. If one such thread takes a long time to read/write, then all other threads waiting to perform get/set would have to wait. Considering this, I feel connection pooling is important.

b) When we have a cluster with at least 3 nodes in it, wouldn't the performance be better if we use at least 3 concurrent connections (one to each redis server)?

However, posts like https://groups.google.com/forum/#!topic/lettuce-redis-client-users/NCmFSgYrQs8, https://github.com/lettuce-io/lettuce-core/wiki/Connection-Pooling-5.1 and https://github.com/lettuce-io/lettuce-core/issues/360 suggest pooling is not necessary if there are no transactions or blocking operations.

For our context, is connection pooling recommended or not?

Thank in advance,

Shobhana

[Mark Paluch]

Hi Shobhana, 

Unless you're using blocking commands (BLPOP, BRPOP, XREAD, BZPOP and such), you should be good with a single Redis cluster connection. Do not apply pooling. Lettuce is thread-safe so you can share a connection across multiple threads and Lettuce does not care how busy your threads are, because Lettuce "just" executes commands.

Redis itself is single-threaded so adding more connection increases just the load. In some cases (transactions, blocking commands, Pub/Sub), it might make sense to use more than one connection. With Redis cluster you get in any case more than a single connection because Lettuce needs to connect to each master node to write data, so you get natural distribution out of the box.

Does this make sense?

Cheers, 

Mark