Redis GETs taking more than 12 msec

[Kashyap Mhaisekar]

Hi,

I have some 300K keys in the redis db totalling to around 8 GB. The keys are of the format <3chartext>:<number> and value is a 32 char string.

I have around 100 threads that connect to the redis DB and get values for the above keys. Occasionally, based on a criteria, i issue a PUBLISH and a DEL command on a single key (DEL on a single key should be o(1) if am right)

When I do the above, i see the GETs showing up on my slowlog logs and shows values till 12 msec. Latency doctor shows the following output -

Dave, I have observed latency spikes in this Redis instance. You don't mind talking about it, do you Dave?

1. fast-command: 18 latency spikes (average 14ms, mean deviation 2ms, period 6987.39 sec). Worst all time event 28ms.

I have a few advices for you:

- The system is slow to execute Redis code paths not containing system calls. This usually means the system does not provide Redis CPU time to run for long periods. You should try to:

  1) Lower the system load.

  2) Use a computer / VM just for Redis if you are running other softawre in the same system.

  3) Check if you have a "noisy neighbour" problem.

  4) Check with 'redis-cli --intrinsic-latency 100' what is the intrinsic latency in your system.

  5) Check if the problem is allocator-related by recompiling Redis with MALLOC=libc, if you are using Jemalloc. However this may create fragmentation problems.

Am not sure what I should be doing for latency doctor messages and the slow gets. Am not sure how to address the above. Any pointers?

Thanks

kashyap

[Josiah Carlson]

What does your on-disk Redis log say? Are you running in a VM like AWS, Rackspace, etc.? What does your Redis conf file have for its "save" line?

My hypothesis: Redis is forking as part of an automatic BGSAVE, which is causing a 10-15ms stutter. It happens because a del is the first write in X minutes, which induces an immediate snapshot.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Kashyap Mhaisekar]

Josiah,

This is a VCE private cloud. From the config -

appendfsync everysec

save 900 1

Dint see anything from logs. Will disabling save help?

Thanks

kashyap

[Josiah Carlson]

What do your Redis logs say right around the time you send your DEL operation and you see the delay?

 - Josiah