Issue 395 in redis: 2.0.4 with VM crashes under load
re...@googlecode.com
Owner: ----
Labels: Type-Defect Priority-Medium
New issue 395 by patrick.huesler: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
What steps will reproduce the problem?
1. vm_enabled:1
2. vm_conf_max_memory less than actual size of data
3. fill redis with the data by replicating from a master
4. put read/write load on redis
What is the expected output? What do you see instead?
[16169] 01 Dec 08:28:42 # ======= Ooops! Redis 2.0.4 got signal: -11-
=======
[16169] 01 Dec 08:28:42 # redis_version:2.0.4
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:16169
uptime_in_seconds:50249
uptime_in_days:0
connected_clients:9
connected_slaves:1
blocked_clients:0
used_memory:18446726487676357472
used_memory_human:
changes_since_last_save:1408878
bgsave_in_progress:0
last_save_time:1291186181
bgrewriteaof_in_progress:0
total_connections_received:576
total_commands_processed:3207968
expired_keys:0
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:0
pubsub_patterns:0
vm_enabled:1
role:slave
master_host:199.34.126.34
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
vm_conf_max_memory:6442450944
vm_conf_page_size:8192
vm_conf_pages:50304000
vm_stats_used_pages:582139
vm_stats_swapped_objects:445519
vm_stats_swappin_count:9717
vm_stats_swappout_count:455259
vm_stats_io_newjobs_len:56
vm_stats_io_processing_len:4
vm_stats_io_processed_len:3758
vm_stats_io_active_threads:4
vm_stats_blocked_clients:2
db2:keys=591866,expires=0
db3:keys=246,expires=0
[16169] 01 Dec 08:28:42 # [0x34]
[16169] 01 Dec 08:28:42 # 2 redis-server 0x40d006 rdbSaveStringObject + 22
[16169] 01 Dec 08:28:42 # 3 redis-server 0x40d150 rdbSaveObject + 144
[16169] 01 Dec 08:28:42 # 4 redis-server 0x40d3b2 rdbSavedObjectLen + 50
[16169] 01 Dec 08:28:42 # 5 redis-server 0x40d3f9 rdbSavedObjectPages + 9
[16169] 01 Dec 08:28:42 # 6 redis-server 0x4195d6 IOThreadEntryPoint + 326
[16169] 01 Dec 08:28:42 # /lib/libpthread.so.0(+0x69ca) [0x7f771e2ab9ca]
[16169] 01 Dec 08:28:42 # /lib/libc.so.6(clone+0x6d) [0x7f771e00870d]
What version of the product are you using? On what operating system?
version 2.0.4, ubuntu 10.04 LTS (Lucid Lynx), amd64, Kernel 2.6.32-25-server
Please provide any additional information below.
this problem occurred two times. the first time, redis crashed after 5
minutes, while still being slave. on the second attempt, we disconnected
redis ("slave of no one") before putting on any load -- again redis
crashed, this time after 30 minutes.
in the end, we went back to version 2.0.1 that we had been using before,
and which seems to be stable.
re...@googlecode.com
Comment #1 on issue 395 by antirez: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
Thanks for the report Patrick. Does this happen in 2.0.4 only when VM is
enabled into a slave?
Btw it is very interesting there is no such problem in 2.0.1 as the changes
are not too big among the two versions. I'll investigate more and report.
Cheers,
Salvatore
re...@googlecode.com
Comment #2 on issue 395 by patrick.huesler: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
In both cases, a slave was connected to the instance that crashed. The
slave also had VM enabled, but did not crash. We did not test how it
behaves when there is no slave attached, because it happened in production.
re...@googlecode.com
Comment #3 on issue 395 by antirez: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
Thank you, just another question. Did you tried also 2.0.2, 2.0.3? Or after
the crash you just switched to the old version that was stable, so directly
from 2.0.4 to 2.0.1?
I'm asking in order to better trace where the problem could be.
Thanks,
Salvatore
re...@googlecode.com
Comment #4 on issue 395 by patrick.huesler: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
We switched right back to 2.0.1 because it was kind of urgent :-)
re...@googlecode.com
Comment #5 on issue 395 by antirez: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
Not sure actually I can parse your latest message: you said that "In both
cases, a slave was connected to the instance that crashed" but from the
INFO output it seems like the instance that crashed *was* a slave.
re...@googlecode.com
Comment #6 on issue 395 by patrick.huesler: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
I'll try to explain. While trying to switch datacenters without downtime,
we used redis replication to move over the data
A is the old redis master, B is the new redis master, C is the new failover
slave.
case 1)
A --> B
A --> B --> C
B crashed after both initial replications were finished and we started to
route traffic to B while still being slave of A
2)
A --> B
A --> B --> C
B --> C
B was no longer slave of A. This time it crashed after being the new
database master for about 30minutes under load. Unfortunately, we don't
have any log file entry for this crash.
Hope this helps. Regards patrick
re...@googlecode.com
Comment #7 on issue 395 by antirez: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
Patrick this definitely helps thanks. Please can you confirm you just tried
2.0.1 and 2.0.4 and not intermediate versions? Thanks.
Cheers,
Salvatore
re...@googlecode.com
Comment #8 on issue 395 by antirez: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
Note: also noticed something extremely odd in the INFO output in the crash:
used_memory:18446726487676357472
Another interesting fact is that the crash both times happened after that
an instance was both a slave and a master, and I fear this is a setup that
is rarely used, so the bug hunting my surely start there.
What is extremely strange is how narrow is the difference between 2.0.1 and
2.0.4.
I'll try to reproduce the same conditions, but if I'll fail I think I'll
have to ask for additional informations / help from you.
Thank you again for the information,
Salvatore
re...@googlecode.com
Comment #9 on issue 395 by patrick.huesler: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
Yes, I confirm that we just tried 2.0.1 and 2.0.4.
If you need any more information, I'll be glad to provide it.
By the way, can you give any news/clarifications on issue 394?
Thank you, I really appreciate you are looking into this.
cheers
Patrick
re...@googlecode.com
Comment #10 on issue 395 by tim.lossen.de: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
yesterday, after happily running for 20 days, redis 2.0.1 (with vm enabled
and one slave connected) suddenly crashed as well. there is nothing in the
logs, and the server had around 7 gigs of free memory at the time.
re...@googlecode.com
Comment #11 on issue 395 by antirez: 2.0.4 with VM crashes under load
http://code.google.com/p/redis/issues/detail?id=395
Apparently there is something wrong with 2.0 VM implementation and
replication. Tim: do you please have logs of the crash?
Since the VM was partially reimplemented in 2.2 I wonder if this problem is
still there.
Please if you happen to switch to 2.2 and you have this problem ping me.