Redis falling short in debugging 100% CPU usage

[Birdy]

Hi,

We are using redis version 2.4.4. (Master-slave, No persistence, )

We are observing 100% CPU that lasts 8-10 seconds, 2-3 times in a day. And at this time it responds very poorly to client requests.

How do we nail this issues down? I feel that Redis falls short on debugging these CPU intensive issues. I think it lacks some tools that can easily debug such problems.

(Specially at production where you can not run gdb or poor man's profiler.)

I already checked following:

1. NO use of KEYS command. (Slow log prints only 10 entries, displaying other commands(debug object) taking 15 millisecond at max)

2. NO persistence. (AOF and RDB persistence is OFF from configuration file)

3. Number of client - 100 max. (According to redis logs)

4. Redis's page fault were not increasing at the time of high CPU.

5. No logging for those 20 seconds on master. (Though slave logged some lines, but shows that it is connected to master)

6. Memory consumption was 200 MB at max.

Kindly suggest how to look into such issues of redis.

Thanks,

Anurag Berdia

 

[Salvatore Sanfilippo]

Hello,

I suggest to following the steps here: http://redis.io/topics/latency

For instance I don't see you mentioning the Slow Log.

If the problem remains not addressed, you should upgrade to Redis 2.6
(suggested move anyway) and using the Software Watchdog in order to
obtain a stack trace when the server is slow.

However given that you are not using persistence it is likely that the
problem is that for some reason you run some very heavy command at
some point, like FLUSHALL, or you delete a very large key, or
something alike. The Slow Log will tell you if this is the case.

Please try what suggested in that page, and ping us again if you have
still problems as we can help you to investigate further.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Birdy]

Hi Salvatore,

Thanks for the reply.

I mentioned about slow log output in previous post. It is not showing any heavy commands.

I am attaching slow log and info output with the reply.

The suggested link talks about latency due to various factors, but the problem that I am trying to solve is high CPU usage(Not latency) by redis for 10-20 seconds.

I checked page faults(0 major page fault), client connection(100 max), persistence(OFF), cost of fork (No cost, since master-slave was connected during high CPU), expires (We do not use expire, we use delete), slow log(No heavy commands). Above all are not applicable in my use case.

Anyway since I don't have any more pointers to debug this issue so planning to move to redis 2.6. Moving to 2.6 would be a heavy exercise for us, as we have to do complete validation as per our use case.

If you can suggest anything else to monitor, I will try that in this setup(With current redis version).

T&R,

Anurag Berdia

[Salvatore Sanfilippo]

Hey, sorry but I noticed too late that you are using master-slave.

What happens is the following:

1) The slave contacts the master for synchronization.
2) The master generates an RDB file.
3) The file is generated by a child, ASAP, so 100% CPU. This is per
Redis design.
4) The synchronization happens transferring this file to the slave.

So what you are seeing is expected behavior. What bugs me is that you
mentioned the performances are not ok while this happens, that part is
unexpected, so it is probably something related to the latency page
issues, but now you know that you are using persistence, since you use
replication.

Starting with Redis 2.8, master and slave don't always need to full
resynchronize after disconnection, they'll try a partial resync. The
code works but is currently still in beta.

Regards,
Salvatore

>> On Thu, Apr 11, 2013 at 9:19 AM, Birdy <anurag...@gmail.com <javascript:>>

>> > email to redis-db+u...@googlegroups.com <javascript:>.

>> > To post to this group, send email to

>> > redi...@googlegroups.com<javascript:>.

[Jokea]

hi,

Check out the output of "SLOWLOG GET 1024" to see if there's any special command during the
time that redis became slow. Also check out master and slave's logs to see if the replication link
was broken and reconnection happened during that time.

I guess upgrade to 2.6 won't do any help in your case.


Regards,
Jokea
 

[Salvatore Sanfilippo]

On Fri, Apr 12, 2013 at 9:09 AM, Jokea <tfen...@gmail.com> wrote:

> I guess upgrade to 2.6 won't do any help in your case.

Hi Jokea, thanks for your reply,

about 2.6, probably not in this specific case but given the cause of
the problem is currently not certain, and that 2.6 addresses a number
of latency issues that 2.4 does not (mass-expiring of keys, RDB
transferring, for example), the ability to setup the watchdog
producing the stack trace, it is probably a good move anyway.

However from what the user writes I think it's just a saving child :-)
Maybe from that point of view it will help immensely that 2.8 changes
the name you see in ps/top to clearly state that it's a saving child.

Cheers,
Salvatore