strange behavior with sharded LRU cache

[Pastion]

We are using a number of redis instances as a sharded LRU cache, and are noticing some strange behavior.

We have not yet been able to extract a simple, reproducible scenario, but in our environment we see this behavior repeatedly.

Our redis instances are not disk-backed, so the number of keys starts at 0.  In each instance, we have a number of sorted sets, which we update with ZINCRBY and trim with ZREMRANGEBYRANK.  The number of keys grows, and eventually stabilizes.  Sometime later, the number of keys starts dropping, until it hits 0, and stays there while INFO MEMORY reports that the affected instances are using maxmemory.  Redis displays no error messages, and neither does the client.  Both seem to believe that everything is fine.

Restarting the client does not seem to fix the problem.  The only fix we have currently is to restart redis, and start over.  We are baffled, and looking for suggestions on where to look.

Thanks, P.

[hva...@gmail.com]

What command(s) are you sending to Redis to read the number of keys?  Is there a master/slave relationship between any of your Redis instances?

A year ago there was a thread in the Reddit forum about a Redis install where the keys would mysteriously disappear but no solution was posted (https://www.reddit.com/r/redis/comments/4warra/sudden_almost_complete_data_loss/).

As I suggested in that thread, my first suspicion in situations where keys are mysteriously disappearing is either (a) keys are being expired because they had a TTL attached to them, or (b) because other unnoticed keys are consuming all the allowed memory.  I think capturing the output from the "INFO ALL" command during the time the number of keys in the database is stable, and then again after the number of keys has gone down to 0 (or near 0) will probably shed some light.

[Pastion]

thanks for the quick reply.

we are using DBSIZE to read the number of keys.

There is no master / slave relationship between any of the redis instances.

We will save the output of INFO ALL and look at the output from INFO ALL when the problem re-occurs.

Thanks again, P.

Sent with ProtonMail Secure Email.

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.

Visit this group at https://groups.google.com/group/redis-db.

For more options, visit https://groups.google.com/d/optout.

[hva...@gmail.com]

DBSIZE tells you the number of keys in the selected database.  If you haven't used the SELECT command in the same connection to select a database, DBSIZE will report on database number 0.  So there could be keys written to another database (1 through 15) and DBSIZE would not report them.

This makes me suggest another thing to do when a Redis server has reached the condition where there are no keys reported by DBSIZE but INFO MEMORY says the memory is being consumed:  Run the FLUSHALL command to empty all the databases.  (do this after you collect the INFO ALL ourput)

If that frees up the memory and lets the clients resume writing/reading to that Redis instance again, then you will know that the issue is some keys are not being deleted or expired and can track them down.