connection time out while using Lettuce

[Tuco]

Hi Mark, 

We are using lettuce to interact with redis and have implemented JCache for Redis using it.

For this, the current code running on a single machine creates a no of connection using Lettuce.

Each table in the DB, corresponds to a connection, further, when the cache is updated, the event to notified to other listeners. Basically, for each cache/table, 4 connections are created.

However, when lettuce tries to connect to redis, many times(but not all), it gives exception that connection timed out, while connecting to either the sentinel or the master redis instance. Further, the behaviour is different for different machines. 

Using redis-cli client list | grep 'ip', we could check that some machines could create upto ~400 connections(each application creates ~200 connection, and there are 2 tomcat instances on the servers), but some developer machines could only create upto ~100-200 connections, and it starts to fail after that.

Also once the application gives this exception(connection timed out), we can still connect to redis from an application on a separate machine or even through redis-cli on the same machine.

The fact that other new machines(which have 0 connections) can connect with redis without any issues.

Also because we can connect to redis using redis-cli on the same machine, it seems to hint that its not even a network issue, may be a Lettuce one(not sure!!)

May be, we should reuse connection across tables, but that is the way code is written by the other team(sorry!!!).

In our Jar for JCache implementation, we are using lettuce version 4.1.Final, but finally in the main project, lettuce version 4.2.0.Final has been used. I have asked the team to upgrade to 4.2.2.Final in both projects, but not sure whether it will fix the issue.

We are using Redis Sentinels and normal master slave configuration for the same.

Would like your opinion on the above issue.

Thanks

[Mark Paluch]

Hi, 

creating many connections with lettuce (and I think with every other client) will cause congestion on your Redis servers that are reflected downstream in your application, causing errors.

Ideally, you deal only with one connection per application. In some cases (blocking commands, transactions, pub/sub) you're required to use multiple connections, but still a reasonable number. You should not create a connection per entity/cache but rather share a connection, if possible. If you're using pub/sub inside a cache, then you might want to keep one global listening connection and distribute notifications to your cache instances. If you're required to use blocking operations or transactions and you have to do that per cache instance, then maybe connection pooling is the right solution. The advantage of pooling is that connections can be allocated upon use and not upfront in a static manner.

Hope this helps. If you have further questions, don't hesitate to ask.

Cheers, 

Mark