Redis frequently restarts. No error logs or indication of OOM.

[Ralph G]

Hello,

I've been using Redis in a test environment in Kubernetes for a couple of weeks now, it's deployed alongside a flask app and a celery worker. After a couple days of the app working without issues, errors like "Connection reset by peer" but mostly "Connection refused" started to appear In the celery worker when trying to connect to Redis.

When checking Redis logs we saw that there's no errors, but we saw that Redis was apparently restarting every once in a while, here's a sample of the logs:

1:C 16 Feb 2021 19:21:19.896 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

1:M 16 Feb 2021 19:17:39.128 * Background saving terminated with success

49974:C 16 Feb 2021 19:17:39.033 * RDB: 0 MB of memory used by copy-on-write

49974:C 16 Feb 2021 19:17:39.032 * DB saved on disk

1:M 16 Feb 2021 19:17:39.028 * Background saving started by pid 49974

1:M 16 Feb 2021 19:17:39.027 * 1 changes in 3600 seconds. Saving...

1:M 16 Feb 2021 18:17:38.123 * Background saving terminated with success

47896:C 16 Feb 2021 18:17:38.027 * RDB: 0 MB of memory used by copy-on-write

47896:C 16 Feb 2021 18:17:38.027 * DB saved on disk

1:M 16 Feb 2021 18:17:38.023 * Background saving started by pid 47896

1:M 16 Feb 2021 18:17:38.022 * 1 changes in 3600 seconds. Saving...

1:M 16 Feb 2021 17:17:37.105 * Background saving terminated with success

Basically the "background saving" logs keeps repeating thought time with "Redis is starting" logs in between at what appears to be random intervals (coinciding in time with the connection error logs).

After checking the pod metrics, OOM was discarded since the pod memory use is very low (since is a test environment with little action),  here are the pod memory metric since Redis was added to the Kubernetes deployment:

The pod has a limit of 256MiB but the memory use is a constant 2-3 MiB. We also checked Kubernetes logs, but it shows 0 restarts and no state change since it was deployed.

The image we're using is a vanilla alpine Redis 6.0.10 without any config change.

Any one has an idea what the issue could be?

[Greg Andrews]

Speaking as another member of this Google Group (I'm not a Redis Labs employee), I can't give you a good answer to your question about why your Redis instance seems to be re-starting.  I'll make a pure guess at the end, but it's only a guess.

The reason I can't give a good answer is because there's something wrong with the logfile lines you posted in your question.  Wrong enough that I can't base any analysis on them.

The problem with the logfile lines is in the timestamps.  The the 2nd-6th lines follow the first line, but their timestamps are more than 3 minutes before the timestamp in the first line.  The 7th-10th lines timestamps are an hour before those lines.  the 11th and last line timestamp is an hour before that.

Did you post the logfile sample backwards?  If so, there's more than 3 minutes between the successful background save (19:17:39) and the restart message (19:21:19).  And the previous background save an hour before (18:17:39) did not cause a restart, nor did the one an hour before that (17:17:37).  So there's no reason to think the cause of the restart is background saves.

My best advice at this point is to find the process that has been automatically re-starting your Redis server and find out from its logfiles why it's re-starting the Redis server process.  In many Linux distributions this is done by the systemd daemon, but I don't know if this is true for your containers.  Perhaps it's dockerd, restarting the whole container rather than just the Redis daemon in the container.  But your best answer will come from the software that is restarting Redis.  Often this kind of software is configured to run a probe of the daemon it launched, and if the probe fails, kill and re-launch the daemon.  Confirmation (or denial) of this possibility will come from the logfile written by such software.