Redis high CPU and fork rate on OpenVZ

[Damon]

Hi,

I'm currently having some issues with Redis spiking in CPU and have been unable to determine what is causing it.

Host OS: Centos 5.5 on an OpenVZ VM

Memory: 12GB soft limit, 16GB hard limit

Redis version: 2.2.4

Memory: 2.45GB, 10.71GB RSS

Keys in db0: ~4M with ~150k expired

Total "expired_keys" from redis-cli info: 49631528

Persistence is disabled

Any time the fork rate on the system spikes above about 100, redis hits 100% CPU and starts to become inaccessible. Sometimes I can't even connect to the redis port while this is happening. I've done a lot of debugging, but have yet to be able to figure out what is going on. I'm not sure what's causing the forking on the system as any redis writes to disk are disabled. This box is a backend for Resque and I've tried disabling all workers with only a limited amount (50 ops/sec) of rpush'es happening, and CPU still thrashes.

There are no keys commands, or large unions or anything like that. Any help in trying to figure out quite what is going on is appreciated.

Thanks,

Damon

[Didier Spezia]

Hi,

>> I'm not sure what's causing the forking on the system as any redis writes to disk are disabled.

Perhaps you meant "swapping" instead of "forking"?

Forking is the operation done by the system to create a new process.

Swapping is the operation done by the system to handle virtual memory.

I/Os or lack of physical memory can trigger implicit swapping. 

Forks are always explicit (i.e. the applications do them).

There is no way Redis could generate 100 forks a second.

It would be a major bug.

 

>> Redis version: 2.2.4

How about starting by migrating to Redis 2.4: you may have a pleasant surprise

regarding memory consumption.

Regards,

Didier.

[Damon]

No, I definitely mean forking. I'm using munin to monitor the system and whenever the fork rate spikes, redis slows down. This server is dedicated to redis functionality, so while I've tried to figure out what's responsible for the forking, I haven't been able to tell.

Since we're on an OpenVZ VM, there's no swap available to the VM. However, the only thing I can think of is that the host machine is actually swapping, causing redis to slow down.

And moving to 2.4 is definitely in the pipeline, I'm just trying to figure out what's happening on our current instance.

Thanks,

Damon

[Josiah Carlson]

Have you tried performing an strace on Redis to see if it is actually
the source of the problem? It should be pretty obvious if Redis is
forking repeatedly. Have you checked the logs? (syslog, messages, or
redis.log) Even if the VM is swapping out due to host pressure, that
shouldn't cause Redis to do anything with forking. Do you have any
slaves of that Redis installation?

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/LXWWVNu0uDkJ.
>
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.