long-lived sync connections appear to time out and hang

[mkpo...@porwit.net]

Hi,

I'm using the 4.3.3.Final in a jetty-based web-service implemented in kotlin. I have a singleton RedisCommand that I use for requests handling -- simple set/get/exists type of stuff. On the server side, we are not setting a timeout. If there is a lot of idle time (several minutes? 30 minutes?), the connection appears to go dead. Subsequent requests to redis fail, and the only fix appears to be a bounce of the server. Here's what the client-side connection code looks like:

    private fun makeRedis(host: String, port: Int): RedisCommands<String, String> {

        val redisClient: RedisClient = RedisClient.create(RedisURI.create(host, port))

        redisClient.options = ClientOptions.builder().socketOptions(SocketOptions.builder()

                .tcpNoDelay(true)

                .keepAlive(true)

                .connectTimeout(5L, TimeUnit.SECONDS)

                .build())

                .pingBeforeActivateConnection(true)

                .build()

        val redisConn: StatefulRedisConnection<String, String> = redisClient.connect()

        redisConn.setTimeout(5L, TimeUnit.SECONDS)

        val redisCommand = redisConn.sync()

        authHealthChecks.register("redis commands", RedisCommandsHealthCheck(redisCommand))

        return redisCommand

    }

Is there anything I should be configuring differently on the client side? Thanks for any insight.

Marcin

[Mark Paluch]

Do you see any disconnects in your logs? You might want to increase logging to see what's going on and investigate from both perspectives (client and Redis Server) whether the connection is in a functional state (i.e. if the connection is reported via netstat and whether your Redis server knows about your client via CLIENT LIST).

What exactly do you mean with connection goes dead and requests to Redis fail? Do you get any exceptions? Do you run into timeouts?

Cheers, 

Mark

[mkpo...@porwit.net]

Hi Mark,

Our HTTP service times out talking to redis once it wedges. I'll get back to investigating this next week, but for now what we've done is we're treating the RedisClient as the Singleton, and getting a new StatefulRedisConnection and a new RedisCommand from it on every redis request we make, and then closing the command. Not optimal, but it does allow our app to keep functioning.

Are all three constructs -- the RedisClient, StatefulRedisConnection, and RedisCommand -- supposed to be long-lived and have a similar life-cycle, or should we be managing one of them differently?

Marcin

[Mark Paluch]

All objects are intended to be long-lived. Lettuce auto-reconnects by default once the TCP connection gets disconnected. Upon successful reconnect it replays all queued commands. Make also sure that you have keep-alive enabled on your Redis machine when using keep-alive on the client side.