Background saving terminated by signal 25?
80 views
Skip to first unread message
AAsfe Regss
Apr 7, 2016, 6:02:49 AM4/7/16
to Redis DB
Hi,
We have Redis running fine on our AWS t2.large instance, but all of a sudden we started encountering the following errors
23518:M 07 Apr 09:49:59.043 # Background saving terminated by signal 25
23518:M 07 Apr 09:50:03.071 * 1 changes in 900 seconds. Saving...
23518:M 07 Apr 09:50:03.130 * Background saving started by pid 3494
23518:M 07 Apr 09:50:05.033 # Background saving terminated by signal 25
23518:M 07 Apr 09:50:09.059 * 1 changes in 900 seconds. Saving...
23518:M 07 Apr 09:50:09.115 * Background saving started by pid 3497
23518:M 07 Apr 09:50:11.018 # Background saving terminated by signal 25
This error just keeps repeating and we are not able to run bgsave at all even after redirecting to another file location. The location where the file writes to have enough space and the issue would stop for a while if we restart, but come back eventually. Right now we had to run with
config set stop-writes-on-bgsave-error no
Thoughts?
Following are some configs of our system
root@ip-10-0-0-224:/home/ubuntu/logs# df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 12K 3.9G 1% /dev
tmpfs 799M 384K 799M 1% /run
/dev/xvda1 20G 16G 3.6G 81% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 3.9G 0 3.9G 0% /run/shm
none 100M 0 100M 0% /run/user
root@ip-10-0-0-224:/home/ubuntu/logs# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63794
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63794
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
127.0.0.1:6379> info
# Server
redis_version:3.0.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:de1194dbb0405e0a
redis_mode:standalone
os:Linux 3.13.0-74-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.4
process_id:23518
run_id:184f6ac2bc4d030e6da0571f1712037a1f2f1c4a
tcp_port:6379
uptime_in_seconds:29732
uptime_in_days:0
hz:10
lru_clock:403136
config_file:/etc/redis/6379.conf
# Clients
connected_clients:15
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:3012300200
used_memory_human:2.81G
used_memory_rss:3247767552
used_memory_peak:3012363424
used_memory_peak_human:2.81G
used_memory_lua:36864
mem_fragmentation_ratio:1.08
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:3438231
rdb_bgsave_in_progress:0
rdb_last_save_time:1459991196
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:2
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:113
total_commands_processed:4801540
instantaneous_ops_per_sec:102
total_net_input_bytes:5028222099
total_net_output_bytes:1892694510
instantaneous_input_kbps:113.43
instantaneous_output_kbps:0.77
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:236175
evicted_keys:0
keyspace_hits:222353
keyspace_misses:1140637
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:53451
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:433.62
used_cpu_user:52.91
used_cpu_sys_children:669.14
used_cpu_user_children:7791.56
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=3514388,expires=3514388,avg_ttl=61625398
127.0.0.1:6379> bgsave
Background saving started
127.0.0.1:6379> info
# Server
redis_version:3.0.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:de1194dbb0405e0a
redis_mode:standalone
os:Linux 3.13.0-74-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.4
process_id:23518
run_id:184f6ac2bc4d030e6da0571f1712037a1f2f1c4a
tcp_port:6379
uptime_in_seconds:31912
uptime_in_days:0
hz:10
lru_clock:405316
config_file:/etc/redis/6379.conf
# Clients
connected_clients:15
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:3173826096
used_memory_human:2.96G
used_memory_rss:3425054720
used_memory_peak:3173846928
used_memory_peak_human:2.96G
used_memory_lua:36864
mem_fragmentation_ratio:1.08
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:3678753
rdb_bgsave_in_progress:1
rdb_last_save_time:1459991196
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:117
total_commands_processed:5127080
instantaneous_ops_per_sec:127
total_net_input_bytes:5391554268
total_net_output_bytes:2018382464
instantaneous_input_kbps:138.69
instantaneous_output_kbps:10.59
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:255261
evicted_keys:0
keyspace_hits:237360
keyspace_misses:1210639
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:56598
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:472.39
used_cpu_user:55.95
used_cpu_sys_children:724.98
used_cpu_user_children:8373.77
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=3735784,expires=3735784,avg_ttl=67643048
mra...@digi2al.co.uk
Apr 8, 2016, 4:57:37 AM4/8/16
to Redis DB
Hi,
redis simply aborts the execution if it gets a signal from the kernel, no fancy logic here (check rdb.c).
Signal 25 is Core File size limit exceeded
redis simply aborts the execution if it gets a signal from the kernel, no fancy logic here (check rdb.c).
Signal 25 is Core File size limit exceeded
( see http://manpages.ubuntu.com/manpages/wily/man7/signal.7.html )
is your redis running as root (who has unlimited core) ?
another reason is why is core dumping :)
Cheers
AAsfe Regss
Apr 9, 2016, 9:27:35 AM4/9/16
to Redis DB
Hi I think I have figured this out, the problem seems to be that I don't have enough memory left for the RDB to run. I have set maxmemory to 2g now and the server seems to be running stable.
And yes redis was running as root.
And yes redis was running as root.
Reply all
Reply to author
Forward
0 new messages