How can we distinguish whether Redis Server is busy or hung?

[gkkum...@gmail.com]

Hi,

For the monitoring purpose, we will query the Redis server periodically. If the Redis server is unresponsive (busy/hung), the monitoring queries can fail. We need to distinguish whether the Redis Server is busy (i.e. due to slow commands execution in progress or any other operation) or hung.

Kindly let me if there is any way to achieve this.

Redis version: 5.0.3

Thanks,

Ganesh

[Greg Andrews]

The most powerful tool for you is the slow log file.  Allowing your monitoring querys to time out after a very long wait (e.g. 10 seconds) can also help you understand whether your Redis server process is merely slow or is actually hung.  Having your monitoring querys measure the response time from Redis server and pass the measurement to a graphing system is also useful to see when slowness happens and whether it's a steady increase in response time or sudden spikes.  Also whether it happens at regular times/intervals.

Another event that people often link to slowness or unresponsiveness in Redis server is if there is a sudden large spike in the rate of new connections being made to Redis.  If your clients only make short-lived connections that send one command/reply and disconnect again -- rather then keeping a connection open to your Redis server, then you can make your server vulnerable to this sort of thing.

If you're looking for extreme measures, attaching a client that issues the MONITOR command and logs the resulting output can show you what commands are being handled by your Redis server.  This can reveal unexpected commands that incur performance hits.

[gkkum...@gmail.com]

Thanks for the response.

Let me share some more details. Our setup is standalone redis with master and slave. The monitoring will periodically (i.e. every 5 secs) query the redis server to check the health. In case of continuous monitoring failures, the slave will take over as a master.

As I mentioned, If the redis is unresponsive (busy/hung), then monitoring queries can fail. Just wondering, if we could distinguish then the failover can be avoided in case of redis busy (slow/blocking command or other operations). Even if someone ran such commands in production mistakenly. 

We also monitor the slow logs, redis calls response time and other parameters (connection, eviction, etc). Actions will be taken accordingly. The slow log will have a commands history. To handle the above case, I am looking for the redis server state at that moment (if possible).

[Mehedi Hasan]

The failover you are thinking about, doesn’t Sentinel can ensure that?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/8bd16583-dbc8-479c-8c4f-c5479cdd295an%40googlegroups.com.

[Archana Sevak]

did you end up setting up MONITOR command or found any other way to check health of Redis instance?

[gkkum...@gmail.com]

In the case of planned maintenance, we need to have control over the failover. Hence the customized monitoring & failover handling at our end.

[gkkum...@gmail.com]

Monitor command will degrade the performance, so we stick with querying the Redis server periodically (i.e. 5 secs) to check the health.