Observing "RedisCommandTimeoutException: Command timed out after 1 minute(s) during load test

[Jagadeesh Venkatesh]

Hi,

i have ran load test for my x application with redis as backend Database. while i'm executing for certain TPS, i'm facing an issue as RedisCommandTimeoutException: Command timed out after 1 minute(s). Can someone suggest me idea to fix this problem ?.

Lettuse version  : lettuce-core-5.1.3

Regards,

jagadeesh

[Greg Andrews]

There's too little information about your Redis server and clients to give you specific advice.  However, the hotel reservation site Trivago posted a blog entry a couple of years ago to the Reddit discussion forum about the lessons they learned on Redis configuration for their large-scale service.  It's very likely the trouble you're seeing has the same cause as one of the lessons they describe.

The Reddit forum thread is:  https://www.reddit.com/r/redis/comments/5q5ddr/learn_redis_the_hard_way_in_production/

It includes the link to their blog post and follow-up comments in the discussion (including one from me)

[Vennila Arumugam]

Hi Greg Andrews,

1. We use lettuce redis library version 5.1.3 and redis server 5.0.3. Redis cluster (2 shrds, 6 nodes) - 2 masters with 2 slaves each. Instance type: cache.r4.xlarge
2. We observed RedisCommandTimeoutException (60s) exception recently by hitting redis server with multiple HGET and HSET in parallel.

3. On reading about this exception, we understand that this occurs due to connection issue (channel getting disconnected) and the commands getting buffered and not getting out to the wire (ie., eventloop getting stuck-up).
4. We are unable to conclude the cause of this issue.
5. When does channel get disconnected or get stuck-up without sending out the command to network layer ? Do we have any design architecture to understand the command flow for lettuce architecture?
6. We observed HSET along with HSCAN in our flow and we are trying to reduce HSCAN which is time-consuming task.
7. We are also trying with lettuce core version 5.1.8 https://github.com/lettuce-io/lettuce-core/issues/1086
8. Please help us by sharing some connection design and possible cause of this exception

Thanks,
Vennila

[Jagadeesh Venkatesh]

Hi Greg Andrews,

Our application is doing alot of concurrent request to Redis Server using Lettuce. At any given time approximately 300~400 concurrent request will be made to redis server. while we do that concurrent operation , We observed "io.lettuce.core.RedisCommandTimeException: Command timedout after 1minute(s)". So, I am having trouble to tune Lettuce to improve performance and stop this timeout issue . Frankly , i unable to conclude the cause of this issue.

• We using lettuce redis library version 5.1.3 and redis server 5.0.3. Redis cluster (2 shrds, 6 nodes) - 2 masters with 2 slaves each. Instance type: cache.r4.xlarge
• RedisCommandTimeoutException (60s) exception occur by hitting redis server with multiple HGET and HSET in parallel.
  
• On reading about this (redis command timeout ) exception, we understand that this occurs due to connection issue (channel getting disconnected) or the commands getting buffered or not getting out to the wire (ie., eventloop getting stuck-up).
• I have to know - When does channel get disconnected or get stuck-up without sending out the command to network layer ? Do we have any design architecture to understand the command flow for lettuce architecture?

• We observed HSET along with HSCAN in our flow and we are trying to reduce HSCAN which is time-consuming task. 

• We are also tried with lettuce core version 5.1.8 https://github.com/lettuce-io/lettuce-core/issues/1086 but no luck.

• Please help us by sharing some connection design and possible cause of this exception

Regards,

Jagadeesh

[Greg Andrews]

I can't understand fully what you're asking me here.  You seem to be asking about the internals of the Lettuce client for Redis server.  I'm sorry, I'm not a Lettuce developer or user.  This is a discussion forum that focuses primarily on the Redis server and not so much on the various Redis clients.  The focused discussions for Lettuce appear to be the Google Group http://groups.google.com/group/lettuce-redis-client-users

That said, with the few details you gave in your original question (we start loading down our Redis with traffic and our clients report connection troubles) the Trivago blog post I pointed to matches quite well with your question.  Soon after they started applying higher loads to their Redis servers, they saw symptoms of trouble, and among the first symptoms was clients reporting connection trouble.  The journey they describe in the blog uncovered several causes for these symptoms, and the solutions they found for each one.  It still sounds to me like it matches your action (apply higher load) and resulting symptoms (clients report connection trouble), so it still sounds like it can be a useful source for you to understand what's causing these problems with your Redis server.  Therefore I still recommend the blog post to you.