27782:M 12 Aug 09:56:06.930 # Server started, Redis version 2.9.56
27782:M 12 Aug 09:56:06.930 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
27782:M 12 Aug 09:56:06.930 * The server is now ready to accept connections on port 6379
27782:S 12 Aug 09:56:06.965 # Cluster state changed: ok
27782:S 12 Aug 09:56:07.968 * Connecting to MASTER 192.168.1.154:6379
27782:S 12 Aug 09:56:07.968 * MASTER <-> SLAVE sync started
27782:S 12 Aug 09:56:07.968 * Non blocking connect for SYNC fired the event.
27782:S 12 Aug 09:56:07.968 * Master replied to PING, replication can continue...
27782:S 12 Aug 09:56:07.968 * Partial resynchronization not possible (no cached master)
27782:S 12 Aug 09:56:07.969 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:1
27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: receiving 18 bytes from master
27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Flushing old data
27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Loading DB in memory
27782:S 12 Aug 09:56:08.074 * MASTER <-> SLAVE sync: Finished with success
27782:S 12 Aug 10:09:22.068 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about 876f04626f806226755c98f15a8475c950a8446b
27782:S 12 Aug 10:09:22.068 # Cluster state changed: fail
27782:S 12 Aug 10:09:41.839 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
27782:S 12 Aug 10:09:43.424 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
27782:S 12 Aug 10:09:44.121 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
27782:S 12 Aug 10:09:44.166 # Start of election delayed for 504 milliseconds (rank #0, offset 514204109).
27782:S 12 Aug 10:09:44.768 # Starting a failover election for epoch 16.
27782:S 12 Aug 10:09:44.770 # Failover election won: I'm the new master.
27782:M 12 Aug 10:09:44.770 # Connection with master lost.
27782:M 12 Aug 10:09:44.770 * Caching the disconnected master state.
27782:M 12 Aug 10:09:44.770 * Discarding previously cached master state.
27782:M 12 Aug 10:09:44.770 # configEpoch set to 16 after successful failover
27782:M 12 Aug 10:09:48.048 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: master without slots is reachable again.
27782:M 12 Aug 10:09:49.042 * Slave asks for synchronization
27782:M 12 Aug 10:09:49.042 * Full resync requested by slave.
27782:M 12 Aug 10:09:49.042 * Starting BGSAVE for SYNC
27782:M 12 Aug 10:09:49.045 * Background saving started by pid 31615
27782:M 12 Aug 10:09:52.684 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
27782:M 12 Aug 10:09:52.684 # Cluster state changed: ok
31615:C 12 Aug 10:09:54.850 * DB saved on disk
31615:C 12 Aug 10:09:54.851 * RDB: 12 MB of memory used by copy-on-write
27782:M 12 Aug 10:09:54.889 * Background saving terminated with success
27782:M 12 Aug 10:09:57.768 * Synchronization with slave succeeded
27782:M 12 Aug 10:30:56.488 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about 3e6c92b2137397573846ca7ce3385de19b033c13
27782:M 12 Aug 10:31:00.255 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
27782:M 12 Aug 13:54:20.887 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
27782:M 12 Aug 13:54:22.004 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
27782:M 12 Aug 14:04:45.448 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
27782:M 12 Aug 14:04:45.975 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
27782:M 13 Aug 14:26:35.092 # Connection with slave 192.168.1.154:6379 lost.
27782:M 13 Aug 14:26:35.102 # Configuration change detected. Reconfiguring myself as a replica of 3e6c92b2137397573846ca7ce3385de19b033c13
27782:S 13 Aug 14:26:35.358 * Connecting to MASTER 192.168.1.154:6379
27782:S 13 Aug 14:26:35.358 * MASTER <-> SLAVE sync started
27782:S 13 Aug 14:26:35.358 * Non blocking connect for SYNC fired the event.
27782:S 13 Aug 14:26:35.359 * Master replied to PING, replication can continue...
27782:S 13 Aug 14:26:35.359 * Partial resynchronization not possible (no cached master)
27782:S 13 Aug 14:26:35.359 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:16950167555
27782:S 13 Aug 14:26:35.498 * MASTER <-> SLAVE sync: receiving 18 bytes from master
27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Flushing old data
27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Loading DB in memory
27782:S 13 Aug 14:26:35.522 * MASTER <-> SLAVE sync: Finished with success
27782:S 13 Aug 14:26:51.288 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
27782:S 13 Aug 14:26:51.858 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
27782:S 13 Aug 14:27:07.162 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about f7bbdafd8baee642981062877e51a7a47f85acdd
27782:S 13 Aug 14:27:07.588 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
27782:S 13 Aug 15:45:49.607 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
27782:S 13 Aug 15:45:50.952 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
27782:S 13 Aug 15:45:54.104 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
27782:S 13 Aug 15:45:54.104 # Cluster state changed: fail
27782:S 13 Aug 15:45:57.403 # Cluster state changed: ok
27782:S 13 Aug 15:45:59.124 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
27782:S 13 Aug 15:48:43.447 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
27782:S 13 Aug 15:48:43.447 # Cluster state changed: fail
27782:S 13 Aug 15:48:44.153 # Cluster state changed: ok
27782:S 13 Aug 15:48:49.660 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: master without slots is reachable again.
27782:S 13 Aug 16:03:59.263 # Manual failover user request accepted.
27782:S 13 Aug 16:03:59.361 # Received replication offset for paused master manual failover: 21175677703
27782:S 13 Aug 16:03:59.400 # All master replication stream processed, manual failover can start.
27782:S 13 Aug 16:03:59.400 # Start of election delayed for 0 milliseconds (rank #0, offset 21175677703).
27782:S 13 Aug 16:03:59.500 # Starting a failover election for epoch 20.
27782:S 13 Aug 16:03:59.501 # Failover election won: I'm the new master.
27782:M 13 Aug 16:03:59.501 # Connection with master lost.
27782:M 13 Aug 16:03:59.502 * Caching the disconnected master state.
27782:M 13 Aug 16:03:59.502 * Discarding previously cached master state.
27782:M 13 Aug 16:03:59.502 # configEpoch set to 20 after successful failover
27782:M 13 Aug 16:04:00.064 * Slave asks for synchronization
27782:M 13 Aug 16:04:00.064 * Full resync requested by slave.
27782:M 13 Aug 16:04:00.064 * Starting BGSAVE for SYNC
27782:M 13 Aug 16:04:00.069 * Background saving started by pid 37373
37373:C 13 Aug 16:04:00.077 * DB saved on disk
37373:C 13 Aug 16:04:00.082 * RDB: 4 MB of memory used by copy-on-write
27782:M 13 Aug 16:04:00.100 * Background saving terminated with success
27782:M 13 Aug 16:04:00.100 * Synchronization with slave succeeded
27782:M 13 Aug 17:52:55.707 # Failover auth denied to 3e6c92b2137397573846ca7ce3385de19b033c13: its master is up
27782:M 13 Aug 17:52:55.708 # Configuration change detected. Reconfiguring myself as a replica of 3e6c92b2137397573846ca7ce3385de19b033c13
27782:S 13 Aug 17:52:56.408 * Connecting to MASTER 192.168.1.154:6379
27782:S 13 Aug 17:52:56.409 * MASTER <-> SLAVE sync started
27782:S 13 Aug 17:52:56.409 * Non blocking connect for SYNC fired the event.
27782:S 13 Aug 17:52:56.409 * Master replied to PING, replication can continue...
27782:S 13 Aug 17:52:56.409 * Partial resynchronization not possible (no cached master)
27782:S 13 Aug 17:52:56.410 * Full resync from master: 5176724a6dce0de4655eebd7fb847306eb1e6bb4:24911255528
27782:S 13 Aug 17:53:13.845 * MASTER <-> SLAVE sync: receiving 605174340 bytes from master
27782:S 13 Aug 17:53:15.346 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
27782:S 13 Aug 17:53:15.346 # Cluster state changed: fail
27782:S 13 Aug 17:53:16.727 # Cluster state changed: ok
27782:S 13 Aug 17:53:19.560 * MASTER <-> SLAVE sync: Flushing old data
27782:S 13 Aug 17:53:19.561 * MASTER <-> SLAVE sync: Loading DB in memory
27782:S 13 Aug 17:53:21.974 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
27782:S 13 Aug 17:53:40.960 * FAIL message received from 3e6c92b2137397573846ca7ce3385de19b033c13 about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
27782:S 13 Aug 17:55:03.945 * MASTER <-> SLAVE sync: Finished with success
27782:S 13 Aug 17:55:04.085 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: slave is reachable again.
27782:S 13 Aug 17:56:23.096 # Manual failover user request accepted.
27782:S 13 Aug 17:56:23.114 # Received replication offset for paused master manual failover: 24911255808
27782:S 13 Aug 17:56:23.159 # All master replication stream processed, manual failover can start.
27782:S 13 Aug 17:56:23.159 # Start of election delayed for 0 milliseconds (rank #0, offset 24911255808).
27782:S 13 Aug 17:56:23.259 # Starting a failover election for epoch 26.
27782:S 13 Aug 17:56:23.261 # Failover election won: I'm the new master.
27782:M 13 Aug 17:56:23.261 # Connection with master lost.
27782:M 13 Aug 17:56:23.261 * Caching the disconnected master state.
27782:M 13 Aug 17:56:23.261 * Discarding previously cached master state.
27782:M 13 Aug 17:56:23.261 # configEpoch set to 26 after successful failover
27782:M 13 Aug 17:56:24.168 * Slave asks for synchronization
27782:M 13 Aug 17:56:24.168 * Full resync requested by slave.
27782:M 13 Aug 17:56:24.168 * Starting BGSAVE for SYNC
27782:M 13 Aug 17:56:24.236 * Background saving started by pid 12909
12909:C 13 Aug 17:56:42.126 * DB saved on disk
12909:C 13 Aug 17:56:42.180 * RDB: 10 MB of memory used by copy-on-write
27782:M 13 Aug 17:56:42.298 * Background saving terminated with success
27782:M 13 Aug 17:57:09.967 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
27782:M 13 Aug 17:57:09.967 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
27782:M 13 Aug 17:57:10.234 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
27782:M 13 Aug 17:57:10.258 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
27782:M 13 Aug 17:57:12.788 * Synchronization with slave succeeded
27782:M 13 Aug 17:57:30.906 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
27782:M 13 Aug 17:57:30.906 # Cluster state changed: fail
27782:M 13 Aug 17:57:33.912 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
27782:M 13 Aug 17:57:40.267 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
27782:M 13 Aug 17:57:56.963 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
27782:M 13 Aug 17:58:00.970 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
27782:M 13 Aug 17:58:31.030 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: is reachable again and nobody is serving its slots after some time.
27782:M 13 Aug 17:58:31.030 # Cluster state changed: ok
27782:M 13 Aug 17:58:59.087 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
27782:M 13 Aug 17:59:04.190 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
27782:M 13 Aug 17:59:16.311 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
27782:M 13 Aug 17:59:27.808 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
27782:M 13 Aug 17:59:38.165 * Marking node 3e6c92b2137397573846ca7ce3385de19b033c13 as failing (quorum reached).
27782:M 13 Aug 17:59:44.607 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
27782:M 14 Aug 09:07:28.597 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
27782:M 14 Aug 09:07:28.597 # Cluster state changed: fail
27782:M 14 Aug 09:07:28.973 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 3e6c92b2137397573846ca7ce3385de19b033c13
27782:M 14 Aug 09:07:29.328 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 29
27782:M 14 Aug 09:07:29.331 # Cluster state changed: ok
27782:M 14 Aug 09:07:29.724 * Clear FAIL state for node 3e6c92b2137397573846ca7ce3385de19b033c13: slave is reachable again.
27782:M 14 Aug 09:07:29.960 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
27782:M 14 Aug 09:08:04.664 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
27782:M 14 Aug 09:08:07.886 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
27782:M 14 Aug 09:09:58.662 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
27782:M 14 Aug 09:10:03.628 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
27782:M 14 Aug 09:12:24.912 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
27782:M 14 Aug 09:12:30.754 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
27782:M 14 Aug 09:19:13.143 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 30
27782:M 14 Aug 09:19:55.592 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
27782:M 14 Aug 09:20:02.619 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
27782:M 14 Aug 09:26:47.547 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
27782:M 14 Aug 09:26:47.547 # Cluster state changed: fail
27782:M 14 Aug 09:26:48.282 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 31
27782:M 14 Aug 09:26:48.284 # Cluster state changed: ok
27782:M 14 Aug 09:26:59.019 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
27782:M 14 Aug 09:35:45.300 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 32
27782:M 14 Aug 09:36:23.575 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
27782:M 14 Aug 09:36:33.678 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
27782:M 14 Aug 09:37:03.681 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
27782:M 14 Aug 09:37:03.681 # Cluster state changed: fail
27782:M 14 Aug 09:37:33.774 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
27782:M 14 Aug 09:37:33.775 # Cluster state changed: ok
27782:M 14 Aug 09:38:35.906 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
27782:M 14 Aug 09:38:40.992 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.