Why dump.rdb file is still made when set save to emtpy string in a Redis cluster
1,063 views
Skip to first unread message
cwh...@gmail.com
Jul 31, 2014, 4:25:56 AM7/31/14
to redi...@googlegroups.com
When starting every instance in the cluster, all used these parameters "--save '' --appendonly no". But there is a file named 'dump.rdb' respectively in the folder of every instance. As my understanding, these "dump.rdb" file should not be made, why? Any ideas can be appreciated!
Jan-Erik Rediger
Jul 31, 2014, 4:38:56 AM7/31/14
to redi...@googlegroups.com
Are you attaching slaves to the master?
On initial sync the master will save an RDB file and send that over to
the slave. Thus with slaves Redis will always produce a RDB snapshot
when requested (but not in the usual intervals as with the save config)
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.
On initial sync the master will save an RDB file and send that over to
the slave. Thus with slaves Redis will always produce a RDB snapshot
when requested (but not in the usual intervals as with the save config)
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.
cwh...@gmail.com
Jul 31, 2014, 6:42:35 AM7/31/14
to redi...@googlegroups.com, jan...@fnordig.de
Hi janerik,
I am using master-slave model, so your explanation makes me clearly. Thanks.
But i still have some questions and hope your help.
1. Do the slave actively send the Sync command to the master?
2. Assuming that a master has only one slave, when the master crashed, the time window to possibly loss data is the interval from last Sync time to the time crashed?
Jan-Erik Rediger
Jul 31, 2014, 7:54:50 AM7/31/14
to redi...@googlegroups.com
1. When starting or after a "SLAVEOF host port" command the slave sends
the SYNC command.
2. No. Only the initial sync between a master and a slave are done via a
rdb snapshot. After that the Master asynchronously sends all write
commands to the slave. Thus, if the master crashes, the possible time
window for data loss is only the delay of the replication.
Also see the documentation at http://redis.io/topics/replication
the SYNC command.
2. No. Only the initial sync between a master and a slave are done via a
rdb snapshot. After that the Master asynchronously sends all write
commands to the slave. Thus, if the master crashes, the possible time
window for data loss is only the delay of the replication.
Also see the documentation at http://redis.io/topics/replication
cwh...@gmail.com
Aug 13, 2014, 10:19:18 PM8/13/14
to redi...@googlegroups.com, jan...@fnordig.de
Hi janerik,
You are right, i have read http://redis.io/topics/replication.
I want to only use Redis as a cache not a data store currently, so i don't want to have any consumption because of RDB or AOF. But i find that the RDB file was also updated occasionally, so i want to know when the RDB file will be updated aside from starting slave.
sbj...@onehealth.com
Aug 15, 2014, 1:27:37 PM8/15/14
to redi...@googlegroups.com, jan...@fnordig.de
Check your redis config file (one common location is /etc/redis/redis.conf)
in the "SNAPSHOTTING" section.
The default configuration is to save every 900 seconds (15 minutes) if at least 1 key has changed.
save 900 1
You may wish to change that section of your redis config file(s)
or whatever chef/puppet/other task generates your redis config files.
thanks,
- scottb
cwh...@gmail.com
Aug 19, 2014, 9:29:36 PM8/19/14
to redi...@googlegroups.com, jan...@fnordig.de, sbj...@onehealth.com
Hi scottb,
I have set "save ''" in the config file, but i find also the RDB file is updated occasionally.
Greg Andrews
Aug 19, 2014, 11:13:30 PM8/19/14
to redi...@googlegroups.com
When the redis-server process is restarted?
On Tue, Aug 19, 2014 at 6:29 PM, <cwh...@gmail.com> wrote:
Hi scottb,I have set "save ''" in the config file, but i find also the RDB file is updated occasionally.
--
cwh...@gmail.com
Aug 20, 2014, 9:57:18 PM8/20/14
to redi...@googlegroups.com
The time to start cluster is "12 Aug 09:55", but some RDB files time updated are "Aug 13 17:56", the others are "Aug 14 18:52".
Jan-Erik Rediger
Aug 20, 2014, 10:03:16 PM8/20/14
to redi...@googlegroups.com
Can you just show the log and also the output of "CONFIG GET *" please.
It should contain everything we need to sort out this issue.
On Wed, Aug 20, 2014 at 06:57:17PM -0700, cwh...@gmail.com wrote:
> The time to start cluster is "12 Aug 09:55", but some RDB files time
> updated are "Aug 13 17:56", the others are "Aug 14 18:52".
>
It should contain everything we need to sort out this issue.
On Wed, Aug 20, 2014 at 06:57:17PM -0700, cwh...@gmail.com wrote:
> The time to start cluster is "12 Aug 09:55", but some RDB files time
> updated are "Aug 13 17:56", the others are "Aug 14 18:52".
>
cwh...@gmail.com
Aug 20, 2014, 10:44:43 PM8/20/14
to redi...@googlegroups.com, jan...@fnordig.de
The result of the "config get *" command is copy into the file names "config.txt".
There are six nodes in the cluster, i choose a master and its slave, the files attached, named "master.txt" and "slave.txt", indicate the log respectively.
Please take a look, thanks!
Jan-Erik Rediger
Aug 21, 2014, 6:22:34 AM8/21/14
to redi...@googlegroups.com
From "master.txt"
39690:M 13 Aug 17:52:56.386 * Slave asks for synchronization
39690:M 13 Aug 17:52:56.386 * Full resync requested by slave.
39690:M 13 Aug 17:52:56.386 * Starting BGSAVE for SYNC
39690:M 13 Aug 17:52:56.484 * Background saving started by pid 32105
32105:C 13 Aug 17:53:13.585 * DB saved on disk
32105:C 13 Aug 17:53:13.659 * RDB: 16 MB of memory used by copy-on-write
39690:M 13 Aug 17:53:13.821 * Background saving terminated with success
From "slave.txt"
27782:M 13 Aug 17:56:24.168 * Slave asks for synchronization
27782:M 13 Aug 17:56:24.168 * Full resync requested by slave.
27782:M 13 Aug 17:56:24.168 * Starting BGSAVE for SYNC
27782:M 13 Aug 17:56:24.236 * Background saving started by pid 12909
12909:C 13 Aug 17:56:42.126 * DB saved on disk
12909:C 13 Aug 17:56:42.180 * RDB: 10 MB of memory used by copy-on-write
This clearly indicates that the then-master saves a RDB when a slave
asks for synchronization.
I repeat my mail from last month again:
> On initial sync the master will save an RDB file and send that over to
> the slave. Thus with slaves Redis will always produce a RDB snapshot
> when requested (but not in the usual intervals as with the save config)
There is no option to turn off this feature. You should make sure that
your system can handles these occasional disk writes.
--
Jan-Erik
> 127.0.0.1:6379> config get *
> 1) "dbfilename"
> 2) "dump.rdb"
> 3) "requirepass"
> 4) ""
> 5) "masterauth"
> 6) ""
> 7) "unixsocket"
> 8) ""
> 9) "logfile"
> 10) "/tmp/redis6379.log"
> 11) "pidfile"
> 12) "/tmp/redis6379.pid"
> 13) "maxmemory"
> 14) "3000000000"
> 15) "maxmemory-samples"
> 16) "5"
> 17) "timeout"
> 18) "0"
> 19) "tcp-keepalive"
> 20) "0"
> 21) "auto-aof-rewrite-percentage"
> 22) "100"
> 23) "auto-aof-rewrite-min-size"
> 24) "67108864"
> 25) "hash-max-ziplist-entries"
> 26) "512"
> 27) "hash-max-ziplist-value"
> 28) "64"
> 29) "list-max-ziplist-entries"
> 30) "512"
> 31) "list-max-ziplist-value"
> 32) "64"
> 33) "set-max-intset-entries"
> 34) "512"
> 35) "zset-max-ziplist-entries"
> 36) "128"
> 37) "zset-max-ziplist-value"
> 38) "64"
> 39) "hll-sparse-max-bytes"
> 40) "3000"
> 41) "lua-time-limit"
> 42) "5000"
> 43) "slowlog-log-slower-than"
> 44) "10000"
> 45) "slowlog-max-len"
> 46) "128"
> 47) "port"
> 48) "6379"
> 49) "tcp-backlog"
> 50) "511"
> 51) "databases"
> 52) "16"
> 53) "repl-ping-slave-period"
> 54) "10"
> 55) "repl-timeout"
> 56) "60"
> 57) "repl-backlog-size"
> 58) "1048576"
> 59) "repl-backlog-ttl"
> 60) "3600"
> 61) "maxclients"
> 62) "10000"
> 63) "watchdog-period"
> 64) "0"
> 65) "slave-priority"
> 66) "100"
> 67) "min-slaves-to-write"
> 68) "0"
> 69) "min-slaves-max-lag"
> 70) "10"
> 71) "hz"
> 72) "10"
> 73) "cluster-node-timeout"
> 74) "15000"
> 75) "cluster-migration-barrier"
> 76) "1"
> 77) "cluster-slave-validity-factor"
> 78) "10"
> 79) "no-appendfsync-on-rewrite"
> 80) "no"
> 81) "slave-serve-stale-data"
> 82) "yes"
> 83) "slave-read-only"
> 84) "yes"
> 85) "stop-writes-on-bgsave-error"
> 86) "yes"
> 87) "daemonize"
> 88) "yes"
> 89) "rdbcompression"
> 90) "yes"
> 91) "rdbchecksum"
> 92) "yes"
> 93) "activerehashing"
> 94) "yes"
> 95) "repl-disable-tcp-nodelay"
> 96) "no"
> 97) "aof-rewrite-incremental-fsync"
> 98) "yes"
> 99) "appendonly"
> 100) "no"
> 101) "dir"
> 102) "/bitauto/cwhtest"
> 103) "maxmemory-policy"
> 104) "noeviction"
> 105) "appendfsync"
> 106) "everysec"
> 107) "save"
> 108) ""
> 109) "loglevel"
> 110) "notice"
> 111) "client-output-buffer-limit"
> 112) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
> 113) "unixsocketperm"
> 114) "0"
> 115) "slaveof"
> 116) "192.168.1.151 6379"
> 117) "notify-keyspace-events"
> 118) ""
> 119) "bind"
> 120) ""
> 39690:M 12 Aug 09:54:56.156 # Server started, Redis version 2.9.56
> 39690:M 12 Aug 09:54:56.158 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
> 39690:M 12 Aug 09:54:56.159 * The server is now ready to accept connections on port 6379
> 39690:M 12 Aug 09:54:59.164 # Cluster state changed: ok
> 39690:M 12 Aug 09:55:27.900 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:M 12 Aug 09:55:34.521 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:M 12 Aug 09:56:07.015 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: slave is reachable again.
> 39690:M 12 Aug 09:56:08.010 * Slave asks for synchronization
> 39690:M 12 Aug 09:56:08.010 * Full resync requested by slave.
> 39690:M 12 Aug 09:56:08.010 * Starting BGSAVE for SYNC
> 39690:M 12 Aug 09:56:08.011 * Background saving started by pid 39829
> 39829:C 12 Aug 09:56:08.028 * DB saved on disk
> 39829:C 12 Aug 09:56:08.029 * RDB: 4 MB of memory used by copy-on-write
> 39690:M 12 Aug 09:56:08.115 * Background saving terminated with success
> 39690:M 12 Aug 09:56:08.115 * Synchronization with slave succeeded
> 39690:M 12 Aug 10:09:22.110 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 39690:M 12 Aug 10:09:22.111 # Cluster state changed: fail
> 39690:M 12 Aug 10:09:48.081 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 15
> 39690:M 12 Aug 10:09:48.082 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:M 12 Aug 10:09:48.082 # Failover auth denied to a9ce649f167b83dc477d195e62ca3fe9034db5e6: its master is up
> 39690:M 12 Aug 10:09:48.082 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 12 Aug 10:09:48.183 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 12 Aug 10:09:49.085 * Connecting to MASTER 192.168.1.151:6379
> 39690:S 12 Aug 10:09:49.085 * MASTER <-> SLAVE sync started
> 39690:S 12 Aug 10:09:49.085 * Non blocking connect for SYNC fired the event.
> 39690:S 12 Aug 10:09:49.085 * Master replied to PING, replication can continue...
> 39690:S 12 Aug 10:09:49.085 * Partial resynchronization not possible (no cached master)
> 39690:S 12 Aug 10:09:49.086 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:514204110
> 39690:S 12 Aug 10:09:54.933 * MASTER <-> SLAVE sync: receiving 314225183 bytes from master
> 39690:S 12 Aug 10:09:56.006 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 39690:S 12 Aug 10:09:56.006 # Cluster state changed: ok
> 39690:S 12 Aug 10:09:57.823 * MASTER <-> SLAVE sync: Flushing old data
> 39690:S 12 Aug 10:09:57.824 * MASTER <-> SLAVE sync: Loading DB in memory
> 39690:S 12 Aug 10:11:24.815 * MASTER <-> SLAVE sync: Finished with success
> 39690:S 12 Aug 13:54:20.904 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 12 Aug 13:54:22.020 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 12 Aug 14:04:45.468 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 12 Aug 14:04:45.994 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 13 Aug 14:26:17.151 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 14:26:17.157 # Cluster state changed: fail
> 39690:S 13 Aug 14:26:17.212 # Start of election delayed for 739 milliseconds (rank #0, offset 16950167508).
> 39690:S 13 Aug 14:26:34.373 # Starting a failover election for epoch 17.
> 39690:S 13 Aug 14:26:35.070 # Failover election won: I'm the new master.
> 39690:M 13 Aug 14:26:35.070 # Connection with master lost.
> 39690:M 13 Aug 14:26:35.070 * Caching the disconnected master state.
> 39690:M 13 Aug 14:26:35.070 * Discarding previously cached master state.
> 39690:M 13 Aug 14:26:35.070 # configEpoch set to 17 after successful failover
> 39690:M 13 Aug 14:26:35.070 # Cluster state changed: ok
> 39690:M 13 Aug 14:26:35.338 * Slave asks for synchronization
> 39690:M 13 Aug 14:26:35.338 * Full resync requested by slave.
> 39690:M 13 Aug 14:26:35.338 * Starting BGSAVE for SYNC
> 39690:M 13 Aug 14:26:35.343 * Background saving started by pid 40968
> 40968:C 13 Aug 14:26:35.405 * DB saved on disk
> 40968:C 13 Aug 14:26:35.411 * RDB: 4 MB of memory used by copy-on-write
> 39690:M 13 Aug 14:26:35.476 * Background saving terminated with success
> 39690:M 13 Aug 14:26:35.476 * Synchronization with slave succeeded
> 39690:M 13 Aug 14:26:41.088 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: slave is reachable again.
> 39690:M 13 Aug 14:27:07.139 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
> 39690:M 13 Aug 14:27:07.567 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:M 13 Aug 15:45:49.581 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 39690:M 13 Aug 15:45:50.927 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:M 13 Aug 15:45:56.593 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
> 39690:M 13 Aug 15:45:56.593 # Cluster state changed: fail
> 39690:M 13 Aug 15:45:57.375 # Failover auth granted to f7bbdafd8baee642981062877e51a7a47f85acdd for epoch 18
> 39690:M 13 Aug 15:45:57.378 # Cluster state changed: ok
> 39690:M 13 Aug 15:45:59.099 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
> 39690:M 13 Aug 15:48:43.423 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:M 13 Aug 15:48:43.423 # Cluster state changed: fail
> 39690:M 13 Aug 15:48:44.126 # Failover auth granted to 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 for epoch 19
> 39690:M 13 Aug 15:48:44.128 # Cluster state changed: ok
> 39690:M 13 Aug 15:48:49.636 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: master without slots is reachable again.
> 39690:M 13 Aug 16:03:59.241 # Manual failover requested by slave a9ce649f167b83dc477d195e62ca3fe9034db5e6.
> 39690:M 13 Aug 16:03:59.479 # Failover auth granted to a9ce649f167b83dc477d195e62ca3fe9034db5e6 for epoch 20
> 39690:M 13 Aug 16:03:59.480 # Connection with slave 192.168.1.151:6379 lost.
> 39690:M 13 Aug 16:03:59.482 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 16:04:00.041 * Connecting to MASTER 192.168.1.151:6379
> 39690:S 13 Aug 16:04:00.041 * MASTER <-> SLAVE sync started
> 39690:S 13 Aug 16:04:00.041 * Non blocking connect for SYNC fired the event.
> 39690:S 13 Aug 16:04:00.041 * Master replied to PING, replication can continue...
> 39690:S 13 Aug 16:04:00.042 * Partial resynchronization not possible (no cached master)
> 39690:S 13 Aug 16:04:00.042 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:21175677704
> 39690:S 13 Aug 16:04:00.079 * MASTER <-> SLAVE sync: receiving 18 bytes from master
> 39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Flushing old data
> 39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Loading DB in memory
> 39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Finished with success
> 39690:S 13 Aug 17:52:52.755 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 17:52:52.759 # Cluster state changed: fail
> 39690:S 13 Aug 17:52:52.826 # Start of election delayed for 810 milliseconds (rank #0, offset 24911255527).
> 39690:S 13 Aug 17:52:53.728 # Starting a failover election for epoch 21.
> 39690:S 13 Aug 17:52:53.729 # Failover election won: I'm the new master.
> 39690:M 13 Aug 17:52:53.729 # Connection with master lost.
> 39690:M 13 Aug 17:52:53.730 * Caching the disconnected master state.
> 39690:M 13 Aug 17:52:53.730 * Discarding previously cached master state.
> 39690:M 13 Aug 17:52:53.730 # configEpoch set to 21 after successful failover
> 39690:M 13 Aug 17:52:53.730 # Cluster state changed: ok
> 39690:M 13 Aug 17:52:55.686 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: master without slots is reachable again.
> 39690:M 13 Aug 17:52:56.386 * Slave asks for synchronization
> 39690:M 13 Aug 17:52:56.386 * Full resync requested by slave.
> 39690:M 13 Aug 17:52:56.386 * Starting BGSAVE for SYNC
> 39690:M 13 Aug 17:52:56.484 * Background saving started by pid 32105
> 32105:C 13 Aug 17:53:13.585 * DB saved on disk
> 32105:C 13 Aug 17:53:13.659 * RDB: 16 MB of memory used by copy-on-write
> 39690:M 13 Aug 17:53:13.821 * Background saving terminated with success
> 39690:M 13 Aug 17:53:15.621 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 39690:M 13 Aug 17:53:15.622 # Cluster state changed: fail
> 39690:M 13 Aug 17:53:16.699 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 22
> 39690:M 13 Aug 17:53:16.703 # Cluster state changed: ok
> 39690:M 13 Aug 17:53:19.509 * Synchronization with slave succeeded
> 39690:M 13 Aug 17:53:21.895 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 39690:M 13 Aug 17:53:40.666 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
> 39690:M 13 Aug 17:53:40.666 # Cluster state changed: fail
> 39690:M 13 Aug 17:53:41.320 # Failover auth granted to f7bbdafd8baee642981062877e51a7a47f85acdd for epoch 23
> 39690:M 13 Aug 17:53:41.331 # Cluster state changed: ok
> 39690:M 13 Aug 17:53:44.653 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
> 39690:M 13 Aug 17:56:08.165 # Failover auth granted to 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 for epoch 24
> 39690:M 13 Aug 17:56:19.448 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 25
> 39690:M 13 Aug 17:56:23.073 # Manual failover requested by slave a9ce649f167b83dc477d195e62ca3fe9034db5e6.
> 39690:M 13 Aug 17:56:23.237 # Failover auth granted to a9ce649f167b83dc477d195e62ca3fe9034db5e6 for epoch 26
> 39690:M 13 Aug 17:56:23.238 # Connection with slave 192.168.1.151:6379 lost.
> 39690:M 13 Aug 17:56:23.342 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 17:56:24.144 * Connecting to MASTER 192.168.1.151:6379
> 39690:S 13 Aug 17:56:24.145 * MASTER <-> SLAVE sync started
> 39690:S 13 Aug 17:56:24.145 * Non blocking connect for SYNC fired the event.
> 39690:S 13 Aug 17:56:24.145 * Master replied to PING, replication can continue...
> 39690:S 13 Aug 17:56:24.145 * Partial resynchronization not possible (no cached master)
> 39690:S 13 Aug 17:56:24.146 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:24911255809
> 39690:S 13 Aug 17:56:42.275 * MASTER <-> SLAVE sync: receiving 605174340 bytes from master
> 39690:S 13 Aug 17:56:51.350 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 13 Aug 17:56:58.593 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 13 Aug 17:56:59.868 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 13 Aug 17:57:06.538 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 17:57:06.539 # Cluster state changed: fail
> 39690:S 13 Aug 17:57:09.237 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 13 Aug 17:57:13.031 * MASTER <-> SLAVE sync: Flushing old data
> 39690:S 13 Aug 17:57:40.172 * MASTER <-> SLAVE sync: Loading DB in memory
> 39690:S 13 Aug 17:57:40.245 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 13 Aug 17:57:56.981 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
> 39690:S 13 Aug 17:58:59.149 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 13 Aug 17:59:16.577 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 13 Aug 17:59:20.745 * MASTER <-> SLAVE sync: Finished with success
> 39690:S 13 Aug 17:59:20.747 # Start of election delayed for 977 milliseconds (rank #0, offset 24911255809).
> 39690:S 13 Aug 17:59:44.583 # Starting a failover election for epoch 28.
> 39690:S 13 Aug 17:59:44.685 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 39690:S 13 Aug 17:59:44.685 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: is reachable again and nobody is serving its slots after some time.
> 39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: is reachable again and nobody is serving its slots after some time.
> 39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 13 Aug 17:59:44.686 # Cluster state changed: ok
> 39690:S 14 Aug 09:07:29.715 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:07:29.935 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 39690:S 14 Aug 09:08:04.660 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 14 Aug 09:08:07.882 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 14 Aug 09:09:58.659 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:10:03.627 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 39690:S 14 Aug 09:12:24.913 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:12:30.754 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 39690:S 14 Aug 09:19:55.594 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 14 Aug 09:20:02.620 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 14 Aug 09:26:47.543 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:26:47.543 # Cluster state changed: fail
> 39690:S 14 Aug 09:26:48.279 # Cluster state changed: ok
> 39690:S 14 Aug 09:26:59.014 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 39690:S 14 Aug 09:36:23.561 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 14 Aug 09:36:33.662 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 14 Aug 09:37:03.665 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:37:03.666 # Cluster state changed: fail
> 39690:S 14 Aug 09:37:36.770 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 39690:S 14 Aug 09:37:36.770 # Cluster state changed: ok
> 39690:S 14 Aug 09:38:35.890 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 14 Aug 09:38:40.976 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 14 Aug 18:52:38.620 # IP address for this node updated to 192.168.1.154
> 27782:M 12 Aug 09:56:06.930 # Server started, Redis version 2.9.56
> 27782:M 12 Aug 09:56:06.930 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
> 27782:M 12 Aug 09:56:06.930 * The server is now ready to accept connections on port 6379
> 27782:S 12 Aug 09:56:06.965 # Cluster state changed: ok
> 27782:S 12 Aug 09:56:07.968 * Connecting to MASTER 192.168.1.154:6379
> 27782:S 12 Aug 09:56:07.968 * MASTER <-> SLAVE sync started
> 27782:S 12 Aug 09:56:07.968 * Non blocking connect for SYNC fired the event.
> 27782:S 12 Aug 09:56:07.968 * Master replied to PING, replication can continue...
> 27782:S 12 Aug 09:56:07.968 * Partial resynchronization not possible (no cached master)
> 27782:S 12 Aug 09:56:07.969 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:1
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: receiving 18 bytes from master
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Flushing old data
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Loading DB in memory
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Finished with success
> 27782:S 12 Aug 10:09:22.068 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about 876f04626f806226755c98f15a8475c950a8446b
> 27782:S 12 Aug 10:09:22.068 # Cluster state changed: fail
> 27782:S 12 Aug 10:09:41.839 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:S 12 Aug 10:09:43.424 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:S 12 Aug 10:09:44.121 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:S 12 Aug 10:09:44.166 # Start of election delayed for 504 milliseconds (rank #0, offset 514204109).
> 27782:S 12 Aug 10:09:44.768 # Starting a failover election for epoch 16.
> 27782:S 12 Aug 10:09:44.770 # Failover election won: I'm the new master.
> 27782:M 12 Aug 10:09:44.770 # Connection with master lost.
> 27782:M 12 Aug 10:09:44.770 * Caching the disconnected master state.
> 27782:M 12 Aug 10:09:44.770 * Discarding previously cached master state.
> 27782:M 12 Aug 10:09:44.770 # configEpoch set to 16 after successful failover
> 27782:M 12 Aug 10:09:48.048 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: master without slots is reachable again.
> 27782:M 12 Aug 10:09:49.042 * Slave asks for synchronization
> 27782:M 12 Aug 10:09:49.042 * Full resync requested by slave.
> 27782:M 12 Aug 10:09:49.042 * Starting BGSAVE for SYNC
> 27782:M 12 Aug 10:09:49.045 * Background saving started by pid 31615
> 27782:M 12 Aug 10:09:52.684 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 27782:M 12 Aug 10:09:52.684 # Cluster state changed: ok
> 31615:C 12 Aug 10:09:54.850 * DB saved on disk
> 31615:C 12 Aug 10:09:54.851 * RDB: 12 MB of memory used by copy-on-write
> 27782:M 12 Aug 10:09:54.889 * Background saving terminated with success
> 27782:M 12 Aug 10:09:57.768 * Synchronization with slave succeeded
> 27782:M 12 Aug 10:30:56.488 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:M 12 Aug 10:31:00.255 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 12 Aug 13:54:20.887 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
> 27782:M 12 Aug 13:54:22.004 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 12 Aug 14:04:45.448 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:M 12 Aug 14:04:45.975 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 13 Aug 14:26:35.092 # Connection with slave 192.168.1.154:6379 lost.
> 27782:M 13 Aug 14:26:35.102 # Configuration change detected. Reconfiguring myself as a replica of 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:S 13 Aug 14:26:35.358 * Connecting to MASTER 192.168.1.154:6379
> 27782:S 13 Aug 14:26:35.358 * MASTER <-> SLAVE sync started
> 27782:S 13 Aug 14:26:35.358 * Non blocking connect for SYNC fired the event.
> 27782:S 13 Aug 14:26:35.359 * Master replied to PING, replication can continue...
> 27782:S 13 Aug 14:26:35.359 * Partial resynchronization not possible (no cached master)
> 27782:S 13 Aug 14:26:35.359 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:16950167555
> 27782:S 13 Aug 14:26:35.498 * MASTER <-> SLAVE sync: receiving 18 bytes from master
> 27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Flushing old data
> 27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Loading DB in memory
> 27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Finished with success
> 27782:S 13 Aug 14:26:51.288 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 27782:S 13 Aug 14:26:51.858 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:S 13 Aug 14:27:07.162 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:S 13 Aug 14:27:07.588 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:S 13 Aug 15:45:49.607 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:S 13 Aug 15:45:50.952 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:S 13 Aug 15:45:54.104 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
> 27782:S 13 Aug 15:45:54.104 # Cluster state changed: fail
> 27782:S 13 Aug 15:45:57.403 # Cluster state changed: ok
> 27782:S 13 Aug 15:45:59.124 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
> 27782:S 13 Aug 15:48:43.447 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:S 13 Aug 15:48:43.447 # Cluster state changed: fail
> 27782:S 13 Aug 15:48:44.153 # Cluster state changed: ok
> 27782:S 13 Aug 15:48:49.660 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: master without slots is reachable again.
> 27782:S 13 Aug 16:03:59.263 # Manual failover user request accepted.
> 27782:S 13 Aug 16:03:59.361 # Received replication offset for paused master manual failover: 21175677703
> 27782:S 13 Aug 16:03:59.400 # All master replication stream processed, manual failover can start.
> 27782:S 13 Aug 16:03:59.400 # Start of election delayed for 0 milliseconds (rank #0, offset 21175677703).
> 27782:S 13 Aug 16:03:59.500 # Starting a failover election for epoch 20.
> 27782:S 13 Aug 16:03:59.501 # Failover election won: I'm the new master.
> 27782:M 13 Aug 16:03:59.501 # Connection with master lost.
> 27782:M 13 Aug 16:03:59.502 * Caching the disconnected master state.
> 27782:M 13 Aug 16:03:59.502 * Discarding previously cached master state.
> 27782:M 13 Aug 16:03:59.502 # configEpoch set to 20 after successful failover
> 27782:M 13 Aug 16:04:00.064 * Slave asks for synchronization
> 27782:M 13 Aug 16:04:00.064 * Full resync requested by slave.
> 27782:M 13 Aug 16:04:00.064 * Starting BGSAVE for SYNC
> 27782:M 13 Aug 16:04:00.069 * Background saving started by pid 37373
> 37373:C 13 Aug 16:04:00.077 * DB saved on disk
> 37373:C 13 Aug 16:04:00.082 * RDB: 4 MB of memory used by copy-on-write
> 27782:M 13 Aug 16:04:00.100 * Background saving terminated with success
> 27782:M 13 Aug 16:04:00.100 * Synchronization with slave succeeded
> 27782:M 13 Aug 17:52:55.707 # Failover auth denied to 3e6c92b2137397573846ca7ce3385de19b033c13: its master is up
> 27782:M 13 Aug 17:52:55.708 # Configuration change detected. Reconfiguring myself as a replica of 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:S 13 Aug 17:52:56.408 * Connecting to MASTER 192.168.1.154:6379
> 27782:S 13 Aug 17:52:56.409 * MASTER <-> SLAVE sync started
> 27782:S 13 Aug 17:52:56.409 * Non blocking connect for SYNC fired the event.
> 27782:S 13 Aug 17:52:56.409 * Master replied to PING, replication can continue...
> 27782:S 13 Aug 17:52:56.409 * Partial resynchronization not possible (no cached master)
> 27782:S 13 Aug 17:52:56.410 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:24911255528
> 27782:S 13 Aug 17:53:13.845 * MASTER <-> SLAVE sync: receiving 605174340 bytes from master
> 27782:S 13 Aug 17:53:15.346 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:S 13 Aug 17:53:15.346 # Cluster state changed: fail
> 27782:S 13 Aug 17:53:16.727 # Cluster state changed: ok
> 27782:S 13 Aug 17:53:19.560 * MASTER <-> SLAVE sync: Flushing old data
> 27782:S 13 Aug 17:53:19.561 * MASTER <-> SLAVE sync: Loading DB in memory
> 27782:S 13 Aug 17:53:21.974 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 27782:S 13 Aug 17:53:40.960 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
> 27782:S 13 Aug 17:55:03.945 * MASTER <-> SLAVE sync: Finished with success
> 27782:S 13 Aug 17:55:04.085 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: slave is reachable again.
> 27782:S 13 Aug 17:56:23.096 # Manual failover user request accepted.
> 27782:S 13 Aug 17:56:23.114 # Received replication offset for paused master manual failover: 24911255808
> 27782:S 13 Aug 17:56:23.159 # All master replication stream processed, manual failover can start.
> 27782:S 13 Aug 17:56:23.159 # Start of election delayed for 0 milliseconds (rank #0, offset 24911255808).
> 27782:S 13 Aug 17:56:23.259 # Starting a failover election for epoch 26.
> 27782:S 13 Aug 17:56:23.261 # Failover election won: I'm the new master.
> 27782:M 13 Aug 17:56:23.261 # Connection with master lost.
> 27782:M 13 Aug 17:56:23.261 * Caching the disconnected master state.
> 27782:M 13 Aug 17:56:23.261 * Discarding previously cached master state.
> 27782:M 13 Aug 17:56:23.261 # configEpoch set to 26 after successful failover
> 27782:M 13 Aug 17:56:24.168 * Slave asks for synchronization
> 27782:M 13 Aug 17:56:24.168 * Full resync requested by slave.
> 27782:M 13 Aug 17:56:24.168 * Starting BGSAVE for SYNC
> 27782:M 13 Aug 17:56:24.236 * Background saving started by pid 12909
> 12909:C 13 Aug 17:56:42.126 * DB saved on disk
> 12909:C 13 Aug 17:56:42.180 * RDB: 10 MB of memory used by copy-on-write
> 27782:M 13 Aug 17:56:42.298 * Background saving terminated with success
> 27782:M 13 Aug 17:57:09.967 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:M 13 Aug 17:57:09.967 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:M 13 Aug 17:57:10.234 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 13 Aug 17:57:10.258 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 13 Aug 17:57:12.788 * Synchronization with slave succeeded
> 27782:M 13 Aug 17:57:30.906 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 13 Aug 17:57:30.906 # Cluster state changed: fail
> 27782:M 13 Aug 17:57:33.912 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:M 13 Aug 17:57:40.267 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 13 Aug 17:57:56.963 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
> 27782:M 13 Aug 17:58:00.970 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 27782:M 13 Aug 17:58:31.030 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: is reachable again and nobody is serving its slots after some time.
> 27782:M 13 Aug 17:58:31.030 # Cluster state changed: ok
> 27782:M 13 Aug 17:58:59.087 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:M 13 Aug 17:59:04.190 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 13 Aug 17:59:16.311 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:M 13 Aug 17:59:27.808 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 13 Aug 17:59:38.165 * Marking node 3e6c92b2137397573846ca7ce3385de19b033c13 as failing (quorum reached).
> 27782:M 13 Aug 17:59:44.607 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 14 Aug 09:07:28.597 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 27782:M 14 Aug 09:07:28.597 # Cluster state changed: fail
> 27782:M 14 Aug 09:07:28.973 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:M 14 Aug 09:07:29.328 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 29
> 27782:M 14 Aug 09:07:29.331 # Cluster state changed: ok
> 27782:M 14 Aug 09:07:29.724 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 14 Aug 09:07:29.960 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 27782:M 14 Aug 09:08:04.664 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
> 27782:M 14 Aug 09:08:07.886 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 14 Aug 09:09:58.662 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 14 Aug 09:10:03.628 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 27782:M 14 Aug 09:12:24.912 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 14 Aug 09:12:30.754 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 27782:M 14 Aug 09:19:13.143 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 30
> 27782:M 14 Aug 09:19:55.592 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:M 14 Aug 09:20:02.619 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 14 Aug 09:26:47.547 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 14 Aug 09:26:47.547 # Cluster state changed: fail
> 27782:M 14 Aug 09:26:48.282 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 31
> 27782:M 14 Aug 09:26:48.284 # Cluster state changed: ok
> 27782:M 14 Aug 09:26:59.019 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 27782:M 14 Aug 09:35:45.300 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 32
> 27782:M 14 Aug 09:36:23.575 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 27782:M 14 Aug 09:36:33.678 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 14 Aug 09:37:03.681 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 27782:M 14 Aug 09:37:03.681 # Cluster state changed: fail
> 27782:M 14 Aug 09:37:33.774 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 27782:M 14 Aug 09:37:33.775 # Cluster state changed: ok
> 27782:M 14 Aug 09:38:35.906 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 27782:M 14 Aug 09:38:40.992 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:M 13 Aug 17:52:56.386 * Slave asks for synchronization
39690:M 13 Aug 17:52:56.386 * Full resync requested by slave.
39690:M 13 Aug 17:52:56.386 * Starting BGSAVE for SYNC
39690:M 13 Aug 17:52:56.484 * Background saving started by pid 32105
32105:C 13 Aug 17:53:13.585 * DB saved on disk
32105:C 13 Aug 17:53:13.659 * RDB: 16 MB of memory used by copy-on-write
39690:M 13 Aug 17:53:13.821 * Background saving terminated with success
From "slave.txt"
27782:M 13 Aug 17:56:24.168 * Slave asks for synchronization
27782:M 13 Aug 17:56:24.168 * Full resync requested by slave.
27782:M 13 Aug 17:56:24.168 * Starting BGSAVE for SYNC
27782:M 13 Aug 17:56:24.236 * Background saving started by pid 12909
12909:C 13 Aug 17:56:42.126 * DB saved on disk
12909:C 13 Aug 17:56:42.180 * RDB: 10 MB of memory used by copy-on-write
This clearly indicates that the then-master saves a RDB when a slave
asks for synchronization.
I repeat my mail from last month again:
> On initial sync the master will save an RDB file and send that over to
> the slave. Thus with slaves Redis will always produce a RDB snapshot
> when requested (but not in the usual intervals as with the save config)
your system can handles these occasional disk writes.
--
Jan-Erik
> 1) "dbfilename"
> 2) "dump.rdb"
> 3) "requirepass"
> 4) ""
> 5) "masterauth"
> 6) ""
> 7) "unixsocket"
> 8) ""
> 9) "logfile"
> 10) "/tmp/redis6379.log"
> 11) "pidfile"
> 12) "/tmp/redis6379.pid"
> 13) "maxmemory"
> 14) "3000000000"
> 15) "maxmemory-samples"
> 16) "5"
> 17) "timeout"
> 18) "0"
> 19) "tcp-keepalive"
> 20) "0"
> 21) "auto-aof-rewrite-percentage"
> 22) "100"
> 23) "auto-aof-rewrite-min-size"
> 24) "67108864"
> 25) "hash-max-ziplist-entries"
> 26) "512"
> 27) "hash-max-ziplist-value"
> 28) "64"
> 29) "list-max-ziplist-entries"
> 30) "512"
> 31) "list-max-ziplist-value"
> 32) "64"
> 33) "set-max-intset-entries"
> 34) "512"
> 35) "zset-max-ziplist-entries"
> 36) "128"
> 37) "zset-max-ziplist-value"
> 38) "64"
> 39) "hll-sparse-max-bytes"
> 40) "3000"
> 41) "lua-time-limit"
> 42) "5000"
> 43) "slowlog-log-slower-than"
> 44) "10000"
> 45) "slowlog-max-len"
> 46) "128"
> 47) "port"
> 48) "6379"
> 49) "tcp-backlog"
> 50) "511"
> 51) "databases"
> 52) "16"
> 53) "repl-ping-slave-period"
> 54) "10"
> 55) "repl-timeout"
> 56) "60"
> 57) "repl-backlog-size"
> 58) "1048576"
> 59) "repl-backlog-ttl"
> 60) "3600"
> 61) "maxclients"
> 62) "10000"
> 63) "watchdog-period"
> 64) "0"
> 65) "slave-priority"
> 66) "100"
> 67) "min-slaves-to-write"
> 68) "0"
> 69) "min-slaves-max-lag"
> 70) "10"
> 71) "hz"
> 72) "10"
> 73) "cluster-node-timeout"
> 74) "15000"
> 75) "cluster-migration-barrier"
> 76) "1"
> 77) "cluster-slave-validity-factor"
> 78) "10"
> 79) "no-appendfsync-on-rewrite"
> 80) "no"
> 81) "slave-serve-stale-data"
> 82) "yes"
> 83) "slave-read-only"
> 84) "yes"
> 85) "stop-writes-on-bgsave-error"
> 86) "yes"
> 87) "daemonize"
> 88) "yes"
> 89) "rdbcompression"
> 90) "yes"
> 91) "rdbchecksum"
> 92) "yes"
> 93) "activerehashing"
> 94) "yes"
> 95) "repl-disable-tcp-nodelay"
> 96) "no"
> 97) "aof-rewrite-incremental-fsync"
> 98) "yes"
> 99) "appendonly"
> 100) "no"
> 101) "dir"
> 102) "/bitauto/cwhtest"
> 103) "maxmemory-policy"
> 104) "noeviction"
> 105) "appendfsync"
> 106) "everysec"
> 107) "save"
> 108) ""
> 109) "loglevel"
> 110) "notice"
> 111) "client-output-buffer-limit"
> 112) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"
> 113) "unixsocketperm"
> 114) "0"
> 115) "slaveof"
> 116) "192.168.1.151 6379"
> 117) "notify-keyspace-events"
> 118) ""
> 119) "bind"
> 120) ""
> 39690:M 12 Aug 09:54:56.156 # Server started, Redis version 2.9.56
> 39690:M 12 Aug 09:54:56.158 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
> 39690:M 12 Aug 09:54:56.159 * The server is now ready to accept connections on port 6379
> 39690:M 12 Aug 09:54:59.164 # Cluster state changed: ok
> 39690:M 12 Aug 09:55:27.900 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:M 12 Aug 09:55:34.521 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:M 12 Aug 09:56:07.015 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: slave is reachable again.
> 39690:M 12 Aug 09:56:08.010 * Slave asks for synchronization
> 39690:M 12 Aug 09:56:08.010 * Full resync requested by slave.
> 39690:M 12 Aug 09:56:08.010 * Starting BGSAVE for SYNC
> 39690:M 12 Aug 09:56:08.011 * Background saving started by pid 39829
> 39829:C 12 Aug 09:56:08.028 * DB saved on disk
> 39829:C 12 Aug 09:56:08.029 * RDB: 4 MB of memory used by copy-on-write
> 39690:M 12 Aug 09:56:08.115 * Background saving terminated with success
> 39690:M 12 Aug 09:56:08.115 * Synchronization with slave succeeded
> 39690:M 12 Aug 10:09:22.110 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 39690:M 12 Aug 10:09:22.111 # Cluster state changed: fail
> 39690:M 12 Aug 10:09:48.081 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 15
> 39690:M 12 Aug 10:09:48.082 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:M 12 Aug 10:09:48.082 # Failover auth denied to a9ce649f167b83dc477d195e62ca3fe9034db5e6: its master is up
> 39690:M 12 Aug 10:09:48.082 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 12 Aug 10:09:48.183 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 12 Aug 10:09:49.085 * Connecting to MASTER 192.168.1.151:6379
> 39690:S 12 Aug 10:09:49.085 * MASTER <-> SLAVE sync started
> 39690:S 12 Aug 10:09:49.085 * Non blocking connect for SYNC fired the event.
> 39690:S 12 Aug 10:09:49.085 * Master replied to PING, replication can continue...
> 39690:S 12 Aug 10:09:49.085 * Partial resynchronization not possible (no cached master)
> 39690:S 12 Aug 10:09:49.086 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:514204110
> 39690:S 12 Aug 10:09:54.933 * MASTER <-> SLAVE sync: receiving 314225183 bytes from master
> 39690:S 12 Aug 10:09:56.006 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 39690:S 12 Aug 10:09:56.006 # Cluster state changed: ok
> 39690:S 12 Aug 10:09:57.823 * MASTER <-> SLAVE sync: Flushing old data
> 39690:S 12 Aug 10:09:57.824 * MASTER <-> SLAVE sync: Loading DB in memory
> 39690:S 12 Aug 10:11:24.815 * MASTER <-> SLAVE sync: Finished with success
> 39690:S 12 Aug 13:54:20.904 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 12 Aug 13:54:22.020 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 12 Aug 14:04:45.468 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 12 Aug 14:04:45.994 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 13 Aug 14:26:17.151 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 14:26:17.157 # Cluster state changed: fail
> 39690:S 13 Aug 14:26:17.212 # Start of election delayed for 739 milliseconds (rank #0, offset 16950167508).
> 39690:S 13 Aug 14:26:34.373 # Starting a failover election for epoch 17.
> 39690:S 13 Aug 14:26:35.070 # Failover election won: I'm the new master.
> 39690:M 13 Aug 14:26:35.070 # Connection with master lost.
> 39690:M 13 Aug 14:26:35.070 * Caching the disconnected master state.
> 39690:M 13 Aug 14:26:35.070 * Discarding previously cached master state.
> 39690:M 13 Aug 14:26:35.070 # configEpoch set to 17 after successful failover
> 39690:M 13 Aug 14:26:35.070 # Cluster state changed: ok
> 39690:M 13 Aug 14:26:35.338 * Slave asks for synchronization
> 39690:M 13 Aug 14:26:35.338 * Full resync requested by slave.
> 39690:M 13 Aug 14:26:35.338 * Starting BGSAVE for SYNC
> 39690:M 13 Aug 14:26:35.343 * Background saving started by pid 40968
> 40968:C 13 Aug 14:26:35.405 * DB saved on disk
> 40968:C 13 Aug 14:26:35.411 * RDB: 4 MB of memory used by copy-on-write
> 39690:M 13 Aug 14:26:35.476 * Background saving terminated with success
> 39690:M 13 Aug 14:26:35.476 * Synchronization with slave succeeded
> 39690:M 13 Aug 14:26:41.088 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: slave is reachable again.
> 39690:M 13 Aug 14:27:07.139 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
> 39690:M 13 Aug 14:27:07.567 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:M 13 Aug 15:45:49.581 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 39690:M 13 Aug 15:45:50.927 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:M 13 Aug 15:45:56.593 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
> 39690:M 13 Aug 15:45:56.593 # Cluster state changed: fail
> 39690:M 13 Aug 15:45:57.375 # Failover auth granted to f7bbdafd8baee642981062877e51a7a47f85acdd for epoch 18
> 39690:M 13 Aug 15:45:57.378 # Cluster state changed: ok
> 39690:M 13 Aug 15:45:59.099 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
> 39690:M 13 Aug 15:48:43.423 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:M 13 Aug 15:48:43.423 # Cluster state changed: fail
> 39690:M 13 Aug 15:48:44.126 # Failover auth granted to 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 for epoch 19
> 39690:M 13 Aug 15:48:44.128 # Cluster state changed: ok
> 39690:M 13 Aug 15:48:49.636 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: master without slots is reachable again.
> 39690:M 13 Aug 16:03:59.241 # Manual failover requested by slave a9ce649f167b83dc477d195e62ca3fe9034db5e6.
> 39690:M 13 Aug 16:03:59.479 # Failover auth granted to a9ce649f167b83dc477d195e62ca3fe9034db5e6 for epoch 20
> 39690:M 13 Aug 16:03:59.480 # Connection with slave 192.168.1.151:6379 lost.
> 39690:M 13 Aug 16:03:59.482 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 16:04:00.041 * Connecting to MASTER 192.168.1.151:6379
> 39690:S 13 Aug 16:04:00.041 * MASTER <-> SLAVE sync started
> 39690:S 13 Aug 16:04:00.041 * Non blocking connect for SYNC fired the event.
> 39690:S 13 Aug 16:04:00.041 * Master replied to PING, replication can continue...
> 39690:S 13 Aug 16:04:00.042 * Partial resynchronization not possible (no cached master)
> 39690:S 13 Aug 16:04:00.042 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:21175677704
> 39690:S 13 Aug 16:04:00.079 * MASTER <-> SLAVE sync: receiving 18 bytes from master
> 39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Flushing old data
> 39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Loading DB in memory
> 39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Finished with success
> 39690:S 13 Aug 17:52:52.755 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 17:52:52.759 # Cluster state changed: fail
> 39690:S 13 Aug 17:52:52.826 # Start of election delayed for 810 milliseconds (rank #0, offset 24911255527).
> 39690:S 13 Aug 17:52:53.728 # Starting a failover election for epoch 21.
> 39690:S 13 Aug 17:52:53.729 # Failover election won: I'm the new master.
> 39690:M 13 Aug 17:52:53.729 # Connection with master lost.
> 39690:M 13 Aug 17:52:53.730 * Caching the disconnected master state.
> 39690:M 13 Aug 17:52:53.730 * Discarding previously cached master state.
> 39690:M 13 Aug 17:52:53.730 # configEpoch set to 21 after successful failover
> 39690:M 13 Aug 17:52:53.730 # Cluster state changed: ok
> 39690:M 13 Aug 17:52:55.686 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: master without slots is reachable again.
> 39690:M 13 Aug 17:52:56.386 * Slave asks for synchronization
> 39690:M 13 Aug 17:52:56.386 * Full resync requested by slave.
> 39690:M 13 Aug 17:52:56.386 * Starting BGSAVE for SYNC
> 39690:M 13 Aug 17:52:56.484 * Background saving started by pid 32105
> 32105:C 13 Aug 17:53:13.585 * DB saved on disk
> 32105:C 13 Aug 17:53:13.659 * RDB: 16 MB of memory used by copy-on-write
> 39690:M 13 Aug 17:53:13.821 * Background saving terminated with success
> 39690:M 13 Aug 17:53:15.621 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 39690:M 13 Aug 17:53:15.622 # Cluster state changed: fail
> 39690:M 13 Aug 17:53:16.699 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 22
> 39690:M 13 Aug 17:53:16.703 # Cluster state changed: ok
> 39690:M 13 Aug 17:53:19.509 * Synchronization with slave succeeded
> 39690:M 13 Aug 17:53:21.895 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 39690:M 13 Aug 17:53:40.666 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
> 39690:M 13 Aug 17:53:40.666 # Cluster state changed: fail
> 39690:M 13 Aug 17:53:41.320 # Failover auth granted to f7bbdafd8baee642981062877e51a7a47f85acdd for epoch 23
> 39690:M 13 Aug 17:53:41.331 # Cluster state changed: ok
> 39690:M 13 Aug 17:53:44.653 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
> 39690:M 13 Aug 17:56:08.165 # Failover auth granted to 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 for epoch 24
> 39690:M 13 Aug 17:56:19.448 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 25
> 39690:M 13 Aug 17:56:23.073 # Manual failover requested by slave a9ce649f167b83dc477d195e62ca3fe9034db5e6.
> 39690:M 13 Aug 17:56:23.237 # Failover auth granted to a9ce649f167b83dc477d195e62ca3fe9034db5e6 for epoch 26
> 39690:M 13 Aug 17:56:23.238 # Connection with slave 192.168.1.151:6379 lost.
> 39690:M 13 Aug 17:56:23.342 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 17:56:24.144 * Connecting to MASTER 192.168.1.151:6379
> 39690:S 13 Aug 17:56:24.145 * MASTER <-> SLAVE sync started
> 39690:S 13 Aug 17:56:24.145 * Non blocking connect for SYNC fired the event.
> 39690:S 13 Aug 17:56:24.145 * Master replied to PING, replication can continue...
> 39690:S 13 Aug 17:56:24.145 * Partial resynchronization not possible (no cached master)
> 39690:S 13 Aug 17:56:24.146 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:24911255809
> 39690:S 13 Aug 17:56:42.275 * MASTER <-> SLAVE sync: receiving 605174340 bytes from master
> 39690:S 13 Aug 17:56:51.350 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 13 Aug 17:56:58.593 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 13 Aug 17:56:59.868 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 13 Aug 17:57:06.538 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about a9ce649f167b83dc477d195e62ca3fe9034db5e6
> 39690:S 13 Aug 17:57:06.539 # Cluster state changed: fail
> 39690:S 13 Aug 17:57:09.237 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 13 Aug 17:57:13.031 * MASTER <-> SLAVE sync: Flushing old data
> 39690:S 13 Aug 17:57:40.172 * MASTER <-> SLAVE sync: Loading DB in memory
> 39690:S 13 Aug 17:57:40.245 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 13 Aug 17:57:56.981 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
> 39690:S 13 Aug 17:58:59.149 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 13 Aug 17:59:16.577 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 13 Aug 17:59:20.745 * MASTER <-> SLAVE sync: Finished with success
> 39690:S 13 Aug 17:59:20.747 # Start of election delayed for 977 milliseconds (rank #0, offset 24911255809).
> 39690:S 13 Aug 17:59:44.583 # Starting a failover election for epoch 28.
> 39690:S 13 Aug 17:59:44.685 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 39690:S 13 Aug 17:59:44.685 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: is reachable again and nobody is serving its slots after some time.
> 39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: is reachable again and nobody is serving its slots after some time.
> 39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 13 Aug 17:59:44.686 # Cluster state changed: ok
> 39690:S 14 Aug 09:07:29.715 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:07:29.935 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 39690:S 14 Aug 09:08:04.660 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 39690:S 14 Aug 09:08:07.882 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 39690:S 14 Aug 09:09:58.659 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:10:03.627 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 39690:S 14 Aug 09:12:24.913 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:12:30.754 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 39690:S 14 Aug 09:19:55.594 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 14 Aug 09:20:02.620 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 14 Aug 09:26:47.543 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:26:47.543 # Cluster state changed: fail
> 39690:S 14 Aug 09:26:48.279 # Cluster state changed: ok
> 39690:S 14 Aug 09:26:59.014 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 39690:S 14 Aug 09:36:23.561 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 14 Aug 09:36:33.662 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 14 Aug 09:37:03.665 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 39690:S 14 Aug 09:37:03.666 # Cluster state changed: fail
> 39690:S 14 Aug 09:37:36.770 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 39690:S 14 Aug 09:37:36.770 # Cluster state changed: ok
> 39690:S 14 Aug 09:38:35.890 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 39690:S 14 Aug 09:38:40.976 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 39690:S 14 Aug 18:52:38.620 # IP address for this node updated to 192.168.1.154
> 27782:M 12 Aug 09:56:06.930 # Server started, Redis version 2.9.56
> 27782:M 12 Aug 09:56:06.930 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
> 27782:M 12 Aug 09:56:06.930 * The server is now ready to accept connections on port 6379
> 27782:S 12 Aug 09:56:06.965 # Cluster state changed: ok
> 27782:S 12 Aug 09:56:07.968 * Connecting to MASTER 192.168.1.154:6379
> 27782:S 12 Aug 09:56:07.968 * MASTER <-> SLAVE sync started
> 27782:S 12 Aug 09:56:07.968 * Non blocking connect for SYNC fired the event.
> 27782:S 12 Aug 09:56:07.968 * Master replied to PING, replication can continue...
> 27782:S 12 Aug 09:56:07.968 * Partial resynchronization not possible (no cached master)
> 27782:S 12 Aug 09:56:07.969 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:1
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: receiving 18 bytes from master
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Flushing old data
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Loading DB in memory
> 27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Finished with success
> 27782:S 12 Aug 10:09:22.068 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about 876f04626f806226755c98f15a8475c950a8446b
> 27782:S 12 Aug 10:09:22.068 # Cluster state changed: fail
> 27782:S 12 Aug 10:09:41.839 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:S 12 Aug 10:09:43.424 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:S 12 Aug 10:09:44.121 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:S 12 Aug 10:09:44.166 # Start of election delayed for 504 milliseconds (rank #0, offset 514204109).
> 27782:S 12 Aug 10:09:44.768 # Starting a failover election for epoch 16.
> 27782:S 12 Aug 10:09:44.770 # Failover election won: I'm the new master.
> 27782:M 12 Aug 10:09:44.770 # Connection with master lost.
> 27782:M 12 Aug 10:09:44.770 * Caching the disconnected master state.
> 27782:M 12 Aug 10:09:44.770 * Discarding previously cached master state.
> 27782:M 12 Aug 10:09:44.770 # configEpoch set to 16 after successful failover
> 27782:M 12 Aug 10:09:48.048 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: master without slots is reachable again.
> 27782:M 12 Aug 10:09:49.042 * Slave asks for synchronization
> 27782:M 12 Aug 10:09:49.042 * Full resync requested by slave.
> 27782:M 12 Aug 10:09:49.042 * Starting BGSAVE for SYNC
> 27782:M 12 Aug 10:09:49.045 * Background saving started by pid 31615
> 27782:M 12 Aug 10:09:52.684 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 27782:M 12 Aug 10:09:52.684 # Cluster state changed: ok
> 31615:C 12 Aug 10:09:54.850 * DB saved on disk
> 31615:C 12 Aug 10:09:54.851 * RDB: 12 MB of memory used by copy-on-write
> 27782:M 12 Aug 10:09:54.889 * Background saving terminated with success
> 27782:M 12 Aug 10:09:57.768 * Synchronization with slave succeeded
> 27782:M 12 Aug 10:30:56.488 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:M 12 Aug 10:31:00.255 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 12 Aug 13:54:20.887 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
> 27782:M 12 Aug 13:54:22.004 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 12 Aug 14:04:45.448 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:M 12 Aug 14:04:45.975 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 13 Aug 14:26:35.092 # Connection with slave 192.168.1.154:6379 lost.
> 27782:M 13 Aug 14:26:35.102 # Configuration change detected. Reconfiguring myself as a replica of 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:S 13 Aug 14:26:35.358 * Connecting to MASTER 192.168.1.154:6379
> 27782:S 13 Aug 14:26:35.358 * MASTER <-> SLAVE sync started
> 27782:S 13 Aug 14:26:35.358 * Non blocking connect for SYNC fired the event.
> 27782:S 13 Aug 14:26:35.359 * Master replied to PING, replication can continue...
> 27782:S 13 Aug 14:26:35.359 * Partial resynchronization not possible (no cached master)
> 27782:S 13 Aug 14:26:35.359 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:16950167555
> 27782:S 13 Aug 14:26:35.498 * MASTER <-> SLAVE sync: receiving 18 bytes from master
> 27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Flushing old data
> 27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Loading DB in memory
> 27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Finished with success
> 27782:S 13 Aug 14:26:51.288 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 27782:S 13 Aug 14:26:51.858 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:S 13 Aug 14:27:07.162 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:S 13 Aug 14:27:07.588 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:S 13 Aug 15:45:49.607 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:S 13 Aug 15:45:50.952 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:S 13 Aug 15:45:54.104 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
> 27782:S 13 Aug 15:45:54.104 # Cluster state changed: fail
> 27782:S 13 Aug 15:45:57.403 # Cluster state changed: ok
> 27782:S 13 Aug 15:45:59.124 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
> 27782:S 13 Aug 15:48:43.447 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:S 13 Aug 15:48:43.447 # Cluster state changed: fail
> 27782:S 13 Aug 15:48:44.153 # Cluster state changed: ok
> 27782:S 13 Aug 15:48:49.660 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: master without slots is reachable again.
> 27782:S 13 Aug 16:03:59.263 # Manual failover user request accepted.
> 27782:S 13 Aug 16:03:59.361 # Received replication offset for paused master manual failover: 21175677703
> 27782:S 13 Aug 16:03:59.400 # All master replication stream processed, manual failover can start.
> 27782:S 13 Aug 16:03:59.400 # Start of election delayed for 0 milliseconds (rank #0, offset 21175677703).
> 27782:S 13 Aug 16:03:59.500 # Starting a failover election for epoch 20.
> 27782:S 13 Aug 16:03:59.501 # Failover election won: I'm the new master.
> 27782:M 13 Aug 16:03:59.501 # Connection with master lost.
> 27782:M 13 Aug 16:03:59.502 * Caching the disconnected master state.
> 27782:M 13 Aug 16:03:59.502 * Discarding previously cached master state.
> 27782:M 13 Aug 16:03:59.502 # configEpoch set to 20 after successful failover
> 27782:M 13 Aug 16:04:00.064 * Slave asks for synchronization
> 27782:M 13 Aug 16:04:00.064 * Full resync requested by slave.
> 27782:M 13 Aug 16:04:00.064 * Starting BGSAVE for SYNC
> 27782:M 13 Aug 16:04:00.069 * Background saving started by pid 37373
> 37373:C 13 Aug 16:04:00.077 * DB saved on disk
> 37373:C 13 Aug 16:04:00.082 * RDB: 4 MB of memory used by copy-on-write
> 27782:M 13 Aug 16:04:00.100 * Background saving terminated with success
> 27782:M 13 Aug 16:04:00.100 * Synchronization with slave succeeded
> 27782:M 13 Aug 17:52:55.707 # Failover auth denied to 3e6c92b2137397573846ca7ce3385de19b033c13: its master is up
> 27782:M 13 Aug 17:52:55.708 # Configuration change detected. Reconfiguring myself as a replica of 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:S 13 Aug 17:52:56.408 * Connecting to MASTER 192.168.1.154:6379
> 27782:S 13 Aug 17:52:56.409 * MASTER <-> SLAVE sync started
> 27782:S 13 Aug 17:52:56.409 * Non blocking connect for SYNC fired the event.
> 27782:S 13 Aug 17:52:56.409 * Master replied to PING, replication can continue...
> 27782:S 13 Aug 17:52:56.409 * Partial resynchronization not possible (no cached master)
> 27782:S 13 Aug 17:52:56.410 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:24911255528
> 27782:S 13 Aug 17:53:13.845 * MASTER <-> SLAVE sync: receiving 605174340 bytes from master
> 27782:S 13 Aug 17:53:15.346 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:S 13 Aug 17:53:15.346 # Cluster state changed: fail
> 27782:S 13 Aug 17:53:16.727 # Cluster state changed: ok
> 27782:S 13 Aug 17:53:19.560 * MASTER <-> SLAVE sync: Flushing old data
> 27782:S 13 Aug 17:53:19.561 * MASTER <-> SLAVE sync: Loading DB in memory
> 27782:S 13 Aug 17:53:21.974 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 27782:S 13 Aug 17:53:40.960 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
> 27782:S 13 Aug 17:55:03.945 * MASTER <-> SLAVE sync: Finished with success
> 27782:S 13 Aug 17:55:04.085 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: slave is reachable again.
> 27782:S 13 Aug 17:56:23.096 # Manual failover user request accepted.
> 27782:S 13 Aug 17:56:23.114 # Received replication offset for paused master manual failover: 24911255808
> 27782:S 13 Aug 17:56:23.159 # All master replication stream processed, manual failover can start.
> 27782:S 13 Aug 17:56:23.159 # Start of election delayed for 0 milliseconds (rank #0, offset 24911255808).
> 27782:S 13 Aug 17:56:23.259 # Starting a failover election for epoch 26.
> 27782:S 13 Aug 17:56:23.261 # Failover election won: I'm the new master.
> 27782:M 13 Aug 17:56:23.261 # Connection with master lost.
> 27782:M 13 Aug 17:56:23.261 * Caching the disconnected master state.
> 27782:M 13 Aug 17:56:23.261 * Discarding previously cached master state.
> 27782:M 13 Aug 17:56:23.261 # configEpoch set to 26 after successful failover
> 27782:M 13 Aug 17:56:24.168 * Slave asks for synchronization
> 27782:M 13 Aug 17:56:24.168 * Full resync requested by slave.
> 27782:M 13 Aug 17:56:24.168 * Starting BGSAVE for SYNC
> 27782:M 13 Aug 17:56:24.236 * Background saving started by pid 12909
> 12909:C 13 Aug 17:56:42.126 * DB saved on disk
> 12909:C 13 Aug 17:56:42.180 * RDB: 10 MB of memory used by copy-on-write
> 27782:M 13 Aug 17:56:42.298 * Background saving terminated with success
> 27782:M 13 Aug 17:57:09.967 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:M 13 Aug 17:57:09.967 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:M 13 Aug 17:57:10.234 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 13 Aug 17:57:10.258 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 13 Aug 17:57:12.788 * Synchronization with slave succeeded
> 27782:M 13 Aug 17:57:30.906 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 13 Aug 17:57:30.906 # Cluster state changed: fail
> 27782:M 13 Aug 17:57:33.912 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:M 13 Aug 17:57:40.267 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 13 Aug 17:57:56.963 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
> 27782:M 13 Aug 17:58:00.970 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 27782:M 13 Aug 17:58:31.030 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: is reachable again and nobody is serving its slots after some time.
> 27782:M 13 Aug 17:58:31.030 # Cluster state changed: ok
> 27782:M 13 Aug 17:58:59.087 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
> 27782:M 13 Aug 17:59:04.190 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 13 Aug 17:59:16.311 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:M 13 Aug 17:59:27.808 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 13 Aug 17:59:38.165 * Marking node 3e6c92b2137397573846ca7ce3385de19b033c13 as failing (quorum reached).
> 27782:M 13 Aug 17:59:44.607 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 14 Aug 09:07:28.597 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 27782:M 14 Aug 09:07:28.597 # Cluster state changed: fail
> 27782:M 14 Aug 09:07:28.973 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
> 27782:M 14 Aug 09:07:29.328 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 29
> 27782:M 14 Aug 09:07:29.331 # Cluster state changed: ok
> 27782:M 14 Aug 09:07:29.724 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
> 27782:M 14 Aug 09:07:29.960 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 27782:M 14 Aug 09:08:04.664 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
> 27782:M 14 Aug 09:08:07.886 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
> 27782:M 14 Aug 09:09:58.662 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 14 Aug 09:10:03.628 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 27782:M 14 Aug 09:12:24.912 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 14 Aug 09:12:30.754 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
> 27782:M 14 Aug 09:19:13.143 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 30
> 27782:M 14 Aug 09:19:55.592 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
> 27782:M 14 Aug 09:20:02.619 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 14 Aug 09:26:47.547 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
> 27782:M 14 Aug 09:26:47.547 # Cluster state changed: fail
> 27782:M 14 Aug 09:26:48.282 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 31
> 27782:M 14 Aug 09:26:48.284 # Cluster state changed: ok
> 27782:M 14 Aug 09:26:59.019 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
> 27782:M 14 Aug 09:35:45.300 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 32
> 27782:M 14 Aug 09:36:23.575 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 27782:M 14 Aug 09:36:33.678 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
> 27782:M 14 Aug 09:37:03.681 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
> 27782:M 14 Aug 09:37:03.681 # Cluster state changed: fail
> 27782:M 14 Aug 09:37:33.774 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
> 27782:M 14 Aug 09:37:33.775 # Cluster state changed: ok
> 27782:M 14 Aug 09:38:35.906 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
> 27782:M 14 Aug 09:38:40.992 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
cwh...@gmail.com
Aug 24, 2014, 9:31:58 PM8/24/14
to redi...@googlegroups.com, jan...@fnordig.de
Hi janerik,
You said "There is no option to turn off this feature", but i have set save property to an empty string, is not it to turn off this feature to forbid background save.
Greg Andrews
Aug 25, 2014, 11:27:28 AM8/25/14
to redi...@googlegroups.com
Jan-Erik is talking about when a slave connects and sends the SYNC command to the master.
The process of syncing a slave involves the master dumping the whole data set to a snapshot dump.rdb file to the master's disk, transferring the snapshot file to the slave, which saves it to the slave's disk, the slave reads the snapshot file from its local disk. Meanwhile the master has been buffering updates to the data set as the save/transfer/read process has been happening. When the slave is finished reading its snapshot file, the slave starts reading the buffered updates.
On Sun, Aug 24, 2014 at 6:31 PM, <cwh...@gmail.com> wrote:
Hi janerik,You said "There is no option to turn off this feature", but i have set save property to an empty string, is not it to turn off this feature to forbid background save.
--
cwh...@gmail.com
Aug 25, 2014, 9:18:35 PM8/25/14
to redi...@googlegroups.com
Hi GregA and janerik,
Thanks for your concrete description.
If the save property is set to an empty string in the cluster, will the slaves also send SYNC command to its master during the cluster runs aside from startup?
Jan-Erik Rediger
Aug 25, 2014, 9:21:43 PM8/25/14
to redi...@googlegroups.com
SYNC is only needed for the initial sync. If a master or a slave
restarts, a new synchronization is needed and thus SYNC is send from the
slave to the master.
restarts, a new synchronization is needed and thus SYNC is send from the
slave to the master.
cwh...@gmail.com
Aug 25, 2014, 11:35:37 PM8/25/14
to redi...@googlegroups.com, jan...@fnordig.de
Ok, i see, thanks again.
Reply all
Reply to author
Forward
0 new messages