redis crashing intermittently

[guilherme reis]

Hello,

It's been a time that my redis is not working properly, and then this weekend the thing got messy. It crashes without any reasons  and when i try to do a "DEBUG SEGFAULT" it crashes too. but give me the output: https://gist.github.com/guilherme/f1da5d34591058866875  and when i run the memory test it gets killed in the middle of the test: https://gist.github.com/guilherme/16257590d3e265de9947. 

I've tried to update to 2.8.12 https://gist.github.com/guilherme/2bbc32a62c5cd5ed1ab6. and the output of the memory test is the same.

Its running on a EC2 Instance, i think thats it problem, but i dont know which steps i could take. Somebody can help me ? The redis is only used to process some resque jobs. One of the queues is stop so i have 2.6 mi jobs, this could be a problem?

Thank you,

Guilherme

[Jan-Erik Rediger]

Both crash reports are from a DEBUG SEGFAULT which is expected. Do you
have a crash report when it crashes without you forcing the crash?

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[Josiah Carlson]

Your Redis server says that it is using 1.7 gigabytes of memory. If your EC2 instance does not have swap enabled (which is pretty common for default EC2 instances), then as your system memory utilization approaches the total system memory, the Linux kernel can and will kill processes that threaten to bring down the machine. There is a stack overflow question that offers a link to the OOM killer heuristics, and a way to discover if this was the cause of the crash:

http://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer

If you have other processes running on this instance, the memory pressure of those other processes combined with Redis is a likely culprit for being low on free memory.

 - Josiah

[guilherme reis]

Thank you @janerik, i've setup the gdb to get this, but i don't think that can be a serious problem.   I think that the problem is what Josiah Carlson said, because there are many occurrences of the OOM killing the redis:

Aug 18 13:35:22 x kernel: [11095395.952470] Killed process 18905 (redis-server) total-vm:1536452kB, anon-rss:1498656kB, file-rss:0kB

Aug 18 13:35:22 x kernel: [11095396.353784] Killed process 31210 (redis-server) total-vm:1536060kB, anon-rss:1498348kB, file-rss:0kB

Aug 18 14:30:29 x kernel: [11098703.208668] Killed process 21967 (redis-server) total-vm:1610180kB, anon-rss:1572848kB, file-rss:68kB

Aug 18 14:30:34 x kernel: [11098707.785461] Killed process 21969 (redis-server) total-vm:1610180kB, anon-rss:1572976kB, file-rss:0kB

Aug 18 14:30:34 x kernel: [11098708.307404] Killed process 19338 (redis-server) total-vm:1609788kB, anon-rss:1572448kB, file-rss:0kB

Aug 18 17:58:11 x kernel: [11111164.658156] Killed process 24120 (redis-server) total-vm:1827268kB, anon-rss:1793368kB, file-rss:84kB

Aug 18 17:58:19 x kernel: [11111173.192058] Killed process 24132 (redis-server) total-vm:1831364kB, anon-rss:1793544kB, file-rss:36kB

Aug 18 17:58:24 x kernel: [11111173.925859] Killed process 22268 (redis-server) total-vm:1830972kB, anon-rss:1793032kB, file-rss:0kB

Aug 18 18:44:47 x kernel: [11113961.560614] Killed process 24464 (redis-server) total-vm:1871932kB, anon-rss:1836408kB, file-rss:0kB

Aug 18 18:58:32 x kernel: [11114785.604505] Killed process 3217 (redis-server) total-vm:1888316kB, anon-rss:1853492kB, file-rss:0kB

Aug 18 18:58:35 x kernel: [11114789.328109] Killed process 3480 (redis-server) total-vm:4214840kB, anon-rss:2691904kB, file-rss:4kB

Aug 18 18:59:25 x kernel: [11114839.160501] Killed process 3741 (redis-server) total-vm:4214840kB, anon-rss:2692712kB, file-rss:0kB

Aug 18 19:29:24 x kernel: [11116638.068098] Killed process 5790 (redis-server) total-vm:1889972kB, anon-rss:1836696kB, file-rss:0kB

Aug 18 19:29:28 x kernel: [11116641.760204] Killed process 5801 (redis-server) total-vm:4216496kB, anon-rss:2689948kB, file-rss:0kB

I'll enable the swap and see what happens.

Thank you all.

[Josiah Carlson]

Rather than enabling swap, which will slow down Redis and everything else on the machine, I'd suggest either upgrading the machine, or trying to set hard limits on memory use *in Redis* to prevent the OOM killer from even being interested in the process. If you've also got other processes running on the machine, I'd suggest moving those processes to other machine(s), and/or moving Redis.

 - Josiah

[guilherme reis]

Enabling swap worked fine. thank you :) I don't have budget for now to move redis to other machine, but i appreciate the advice.