Re: redis background saving terminated with signal 11

[Josiah Carlson]

Signal 11 is a segfault; the forked child followed a pointer that lead
to unallocated memory. There should be a core file hanging out on
disk.

If you could, you should run a Redis memory test on that machine, as
that should be the simplest way to rule memory corruption out. If
possible, you also may want to run a recent version of memtest86
(assuming you are running on an x86 machine).

If you've got the core, Salvatore may be able to track down the exact
bug (if it's Redis).

Regards,
- Josiah

On Wed, Jul 18, 2012 at 5:08 PM, Michael RICORDEAU
<michael....@gmail.com> wrote:
> Hi there,
>
> Im using redis 2.4.15 :
>
> redis_version:2.4.15
> redis_git_sha1:00000000
> redis_git_dirty:0
> arch_bits:64
> multiplexing_api:epoll
> gcc_version:4.4.5
> process_id:10092
> uptime_in_seconds:496
> uptime_in_days:0
> lru_clock:47896
> used_cpu_sys:1.34
> used_cpu_user:0.68
> used_cpu_sys_children:7.84
> used_cpu_user_children:0.01
> connected_clients:259
> connected_slaves:0
> client_longest_output_list:0
> client_biggest_input_buf:0
> blocked_clients:21
> used_memory:3594384
> used_memory_human:3.43M
> used_memory_rss:3645440
> used_memory_peak:3594344
> used_memory_peak_human:3.43M
> mem_fragmentation_ratio:1.01
> mem_allocator:jemalloc-3.0.0
> loading:0
> aof_enabled:0
> changes_since_last_save:1854
> bgsave_in_progress:0
> last_save_time:1342655745
> bgrewriteaof_in_progress:0
> total_connections_received:428
> total_commands_processed:12820
> expired_keys:264
> evicted_keys:0
> keyspace_hits:272
> keyspace_misses:1360
> pubsub_channels:1
> pubsub_patterns:0
> latest_fork_usec:1028
> vm_enabled:0
> role:master
> db0:keys=10,expires=8
>
>
> In logs :
> Jul 19 00:05:25 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 18995
> Jul 19 00:05:25 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:25 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:25 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 18996
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 18997
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 18998
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 18999
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 19012
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:26 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 19013
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 19015
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 19016
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: Background saving terminated by
> signal 11
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: 10 changes in 300 seconds.
> Saving...
> Jul 19 00:05:27 ip-10-6-99-217 redis[10092]: Background saving started by
> pid 19017
>
> My vm (aws) :
> Debian 6.0.4 / squeeze
> Linux ip-10-6-99-217 2.6.32-5-xen-amd64 #1 SMP Mon Jan 16 20:48:30 UTC 2012
> x86_64 GNU/Linux
>
>
> No heavy writes, but when the first bgsave is fired, it is failing and
> trying to save in endless loop cause of signal 11 (I think) .
>
> Any idea ?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/redis-db/-/AG0Q8OCdvDwJ.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.

[M. Edward (Ed) Borasky]

On Fri, Jul 20, 2012 at 10:18 AM, Josiah Carlson
<josiah....@gmail.com> wrote:
> Signal 11 is a segfault; the forked child followed a pointer that lead
> to unallocated memory. There should be a core file hanging out on
> disk.
>
> If you could, you should run a Redis memory test on that machine, as
> that should be the simplest way to rule memory corruption out. If
> possible, you also may want to run a recent version of memtest86
> (assuming you are running on an x86 machine).
>
> If you've got the core, Salvatore may be able to track down the exact
> bug (if it's Redis).
>
> Regards,
> - Josiah

You might well have to tweak your environment variables and re-run to
get a core dump file. Most systems these days set it up so the core
dumps don't "clutter up your disk." ;-)

--
Twitter: http://twitter.com/znmeb Computational Journalism Studio
http://j.mp/CompJournStudio

Data is the new coal - abundant, dirty and difficult to mine.

[Michael RICORDEAU]

Hi,

finally I got it fixed, I was setting ulimit -s 240 .

Looks like redis is not happy with this stack size in aws instance 64bits .

Using ulimit -s 8192 is working like a charm .

>> redis-db+unsubscribe@googlegroups.com.

>> For more options, visit this group at
>> http://groups.google.com/group/redis-db?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.

> To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com.