39690:M 12 Aug 09:54:56.156 # Server started, Redis version 2.9.56
39690:M 12 Aug 09:54:56.158 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
39690:M 12 Aug 09:54:56.159 * The server is now ready to accept connections on port 6379
39690:M 12 Aug 09:54:59.164 # Cluster state changed: ok
39690:M 12 Aug 09:55:27.900 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:M 12 Aug 09:55:34.521 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
39690:M 12 Aug 09:56:07.015 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: slave is reachable again.
39690:M 12 Aug 09:56:08.010 * Slave asks for synchronization
39690:M 12 Aug 09:56:08.010 * Full resync requested by slave.
39690:M 12 Aug 09:56:08.010 * Starting BGSAVE for SYNC
39690:M 12 Aug 09:56:08.011 * Background saving started by pid 39829
39829:C 12 Aug 09:56:08.028 * DB saved on disk
39829:C 12 Aug 09:56:08.029 * RDB: 4 MB of memory used by copy-on-write
39690:M 12 Aug 09:56:08.115 * Background saving terminated with success
39690:M 12 Aug 09:56:08.115 * Synchronization with slave succeeded
39690:M 12 Aug 10:09:22.110 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
39690:M 12 Aug 10:09:22.111 # Cluster state changed: fail
39690:M 12 Aug 10:09:48.081 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 15
39690:M 12 Aug 10:09:48.082 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
39690:M 12 Aug 10:09:48.082 # Failover auth denied to a9ce649f167b83dc477d195e62ca3fe9034db5e6: its master is up
39690:M 12 Aug 10:09:48.082 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
39690:S 12 Aug 10:09:48.183 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:S 12 Aug 10:09:49.085 * Connecting to MASTER 192.168.1.151:6379
39690:S 12 Aug 10:09:49.085 * MASTER <-> SLAVE sync started
39690:S 12 Aug 10:09:49.085 * Non blocking connect for SYNC fired the event.
39690:S 12 Aug 10:09:49.085 * Master replied to PING, replication can continue...
39690:S 12 Aug 10:09:49.085 * Partial resynchronization not possible (no cached master)
39690:S 12 Aug 10:09:49.086 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:514204110
39690:S 12 Aug 10:09:54.933 * MASTER <-> SLAVE sync: receiving 314225183 bytes from master
39690:S 12 Aug 10:09:56.006 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
39690:S 12 Aug 10:09:56.006 # Cluster state changed: ok
39690:S 12 Aug 10:09:57.823 * MASTER <-> SLAVE sync: Flushing old data
39690:S 12 Aug 10:09:57.824 * MASTER <-> SLAVE sync: Loading DB in memory
39690:S 12 Aug 10:11:24.815 * MASTER <-> SLAVE sync: Finished with success
39690:S 12 Aug 13:54:20.904 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about f7bbdafd8baee642981062877e51a7a47f85acdd
39690:S 12 Aug 13:54:22.020 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
39690:S 12 Aug 14:04:45.468 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
39690:S 12 Aug 14:04:45.994 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
39690:S 13 Aug 14:26:17.151 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about a9ce649f167b83dc477d195e62ca3fe9034db5e6
39690:S 13 Aug 14:26:17.157 # Cluster state changed: fail
39690:S 13 Aug 14:26:17.212 # Start of election delayed for 739 milliseconds (rank #0, offset 16950167508).
39690:S 13 Aug 14:26:34.373 # Starting a failover election for epoch 17.
39690:S 13 Aug 14:26:35.070 # Failover election won: I'm the new master.
39690:M 13 Aug 14:26:35.070 # Connection with master lost.
39690:M 13 Aug 14:26:35.070 * Caching the disconnected master state.
39690:M 13 Aug 14:26:35.070 * Discarding previously cached master state.
39690:M 13 Aug 14:26:35.070 # configEpoch set to 17 after successful failover
39690:M 13 Aug 14:26:35.070 # Cluster state changed: ok
39690:M 13 Aug 14:26:35.338 * Slave asks for synchronization
39690:M 13 Aug 14:26:35.338 * Full resync requested by slave.
39690:M 13 Aug 14:26:35.338 * Starting BGSAVE for SYNC
39690:M 13 Aug 14:26:35.343 * Background saving started by pid 40968
40968:C 13 Aug 14:26:35.405 * DB saved on disk
40968:C 13 Aug 14:26:35.411 * RDB: 4 MB of memory used by copy-on-write
39690:M 13 Aug 14:26:35.476 * Background saving terminated with success
39690:M 13 Aug 14:26:35.476 * Synchronization with slave succeeded
39690:M 13 Aug 14:26:41.088 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: slave is reachable again.
39690:M 13 Aug 14:27:07.139 * Marking node f7bbdafd8baee642981062877e51a7a47f85acdd as failing (quorum reached).
39690:M 13 Aug 14:27:07.567 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
39690:M 13 Aug 15:45:49.581 * Marking node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff as failing (quorum reached).
39690:M 13 Aug 15:45:50.927 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:M 13 Aug 15:45:56.593 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
39690:M 13 Aug 15:45:56.593 # Cluster state changed: fail
39690:M 13 Aug 15:45:57.375 # Failover auth granted to f7bbdafd8baee642981062877e51a7a47f85acdd for epoch 18
39690:M 13 Aug 15:45:57.378 # Cluster state changed: ok
39690:M 13 Aug 15:45:59.099 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
39690:M 13 Aug 15:48:43.423 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about f7bbdafd8baee642981062877e51a7a47f85acdd
39690:M 13 Aug 15:48:43.423 # Cluster state changed: fail
39690:M 13 Aug 15:48:44.126 # Failover auth granted to 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 for epoch 19
39690:M 13 Aug 15:48:44.128 # Cluster state changed: ok
39690:M 13 Aug 15:48:49.636 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: master without slots is reachable again.
39690:M 13 Aug 16:03:59.241 # Manual failover requested by slave a9ce649f167b83dc477d195e62ca3fe9034db5e6.
39690:M 13 Aug 16:03:59.479 # Failover auth granted to a9ce649f167b83dc477d195e62ca3fe9034db5e6 for epoch 20
39690:M 13 Aug 16:03:59.480 # Connection with slave 192.168.1.151:6379 lost.
39690:M 13 Aug 16:03:59.482 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
39690:S 13 Aug 16:04:00.041 * Connecting to MASTER 192.168.1.151:6379
39690:S 13 Aug 16:04:00.041 * MASTER <-> SLAVE sync started
39690:S 13 Aug 16:04:00.041 * Non blocking connect for SYNC fired the event.
39690:S 13 Aug 16:04:00.041 * Master replied to PING, replication can continue...
39690:S 13 Aug 16:04:00.042 * Partial resynchronization not possible (no cached master)
39690:S 13 Aug 16:04:00.042 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:21175677704
39690:S 13 Aug 16:04:00.079 * MASTER <-> SLAVE sync: receiving 18 bytes from master
39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Flushing old data
39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Loading DB in memory
39690:S 13 Aug 16:04:00.087 * MASTER <-> SLAVE sync: Finished with success
39690:S 13 Aug 17:52:52.755 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about a9ce649f167b83dc477d195e62ca3fe9034db5e6
39690:S 13 Aug 17:52:52.759 # Cluster state changed: fail
39690:S 13 Aug 17:52:52.826 # Start of election delayed for 810 milliseconds (rank #0, offset 24911255527).
39690:S 13 Aug 17:52:53.728 # Starting a failover election for epoch 21.
39690:S 13 Aug 17:52:53.729 # Failover election won: I'm the new master.
39690:M 13 Aug 17:52:53.729 # Connection with master lost.
39690:M 13 Aug 17:52:53.730 * Caching the disconnected master state.
39690:M 13 Aug 17:52:53.730 * Discarding previously cached master state.
39690:M 13 Aug 17:52:53.730 # configEpoch set to 21 after successful failover
39690:M 13 Aug 17:52:53.730 # Cluster state changed: ok
39690:M 13 Aug 17:52:55.686 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: master without slots is reachable again.
39690:M 13 Aug 17:52:56.386 * Slave asks for synchronization
39690:M 13 Aug 17:52:56.386 * Full resync requested by slave.
39690:M 13 Aug 17:52:56.386 * Starting BGSAVE for SYNC
39690:M 13 Aug 17:52:56.484 * Background saving started by pid 32105
32105:C 13 Aug 17:53:13.585 * DB saved on disk
32105:C 13 Aug 17:53:13.659 * RDB: 16 MB of memory used by copy-on-write
39690:M 13 Aug 17:53:13.821 * Background saving terminated with success
39690:M 13 Aug 17:53:15.621 * Marking node 876f04626f806226755c98f15a8475c950a8446b as failing (quorum reached).
39690:M 13 Aug 17:53:15.622 # Cluster state changed: fail
39690:M 13 Aug 17:53:16.699 # Failover auth granted to fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff for epoch 22
39690:M 13 Aug 17:53:16.703 # Cluster state changed: ok
39690:M 13 Aug 17:53:19.509 * Synchronization with slave succeeded
39690:M 13 Aug 17:53:21.895 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
39690:M 13 Aug 17:53:40.666 * Marking node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 as failing (quorum reached).
39690:M 13 Aug 17:53:40.666 # Cluster state changed: fail
39690:M 13 Aug 17:53:41.320 # Failover auth granted to f7bbdafd8baee642981062877e51a7a47f85acdd for epoch 23
39690:M 13 Aug 17:53:41.331 # Cluster state changed: ok
39690:M 13 Aug 17:53:44.653 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: master without slots is reachable again.
39690:M 13 Aug 17:56:08.165 # Failover auth granted to 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 for epoch 24
39690:M 13 Aug 17:56:19.448 # Failover auth granted to 876f04626f806226755c98f15a8475c950a8446b for epoch 25
39690:M 13 Aug 17:56:23.073 # Manual failover requested by slave a9ce649f167b83dc477d195e62ca3fe9034db5e6.
39690:M 13 Aug 17:56:23.237 # Failover auth granted to a9ce649f167b83dc477d195e62ca3fe9034db5e6 for epoch 26
39690:M 13 Aug 17:56:23.238 # Connection with slave 192.168.1.151:6379 lost.
39690:M 13 Aug 17:56:23.342 # Configuration change detected. Reconfiguring myself as a replica of a9ce649f167b83dc477d195e62ca3fe9034db5e6
39690:S 13 Aug 17:56:24.144 * Connecting to MASTER 192.168.1.151:6379
39690:S 13 Aug 17:56:24.145 * MASTER <-> SLAVE sync started
39690:S 13 Aug 17:56:24.145 * Non blocking connect for SYNC fired the event.
39690:S 13 Aug 17:56:24.145 * Master replied to PING, replication can continue...
39690:S 13 Aug 17:56:24.145 * Partial resynchronization not possible (no cached master)
39690:S 13 Aug 17:56:24.146 * Full resync from master: 57beb0e3305365d9576deb5b4afd8000362655d0:24911255809
39690:S 13 Aug 17:56:42.275 * MASTER <-> SLAVE sync: receiving 605174340 bytes from master
39690:S 13 Aug 17:56:51.350 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
39690:S 13 Aug 17:56:58.593 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
39690:S 13 Aug 17:56:59.868 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
39690:S 13 Aug 17:57:06.538 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about a9ce649f167b83dc477d195e62ca3fe9034db5e6
39690:S 13 Aug 17:57:06.539 # Cluster state changed: fail
39690:S 13 Aug 17:57:09.237 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:S 13 Aug 17:57:13.031 * MASTER <-> SLAVE sync: Flushing old data
39690:S 13 Aug 17:57:40.172 * MASTER <-> SLAVE sync: Loading DB in memory
39690:S 13 Aug 17:57:40.245 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
39690:S 13 Aug 17:57:56.981 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 6f1a37d66aacbcb0128c6306b5b989ad81c160c1
39690:S 13 Aug 17:58:59.149 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
39690:S 13 Aug 17:59:16.577 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
39690:S 13 Aug 17:59:20.745 * MASTER <-> SLAVE sync: Finished with success
39690:S 13 Aug 17:59:20.747 # Start of election delayed for 977 milliseconds (rank #0, offset 24911255809).
39690:S 13 Aug 17:59:44.583 # Starting a failover election for epoch 28.
39690:S 13 Aug 17:59:44.685 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
39690:S 13 Aug 17:59:44.685 * Clear FAIL state for node a9ce649f167b83dc477d195e62ca3fe9034db5e6: is reachable again and nobody is serving its slots after some time.
39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node 6f1a37d66aacbcb0128c6306b5b989ad81c160c1: is reachable again and nobody is serving its slots after some time.
39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
39690:S 13 Aug 17:59:44.686 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:S 13 Aug 17:59:44.686 # Cluster state changed: ok
39690:S 14 Aug 09:07:29.715 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
39690:S 14 Aug 09:07:29.935 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
39690:S 14 Aug 09:08:04.660 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about f7bbdafd8baee642981062877e51a7a47f85acdd
39690:S 14 Aug 09:08:07.882 * Clear FAIL state for node f7bbdafd8baee642981062877e51a7a47f85acdd: slave is reachable again.
39690:S 14 Aug 09:09:58.659 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 876f04626f806226755c98f15a8475c950a8446b
39690:S 14 Aug 09:10:03.627 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
39690:S 14 Aug 09:12:24.913 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about 876f04626f806226755c98f15a8475c950a8446b
39690:S 14 Aug 09:12:30.754 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: slave is reachable again.
39690:S 14 Aug 09:19:55.594 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
39690:S 14 Aug 09:20:02.620 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:S 14 Aug 09:26:47.543 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
39690:S 14 Aug 09:26:47.543 # Cluster state changed: fail
39690:S 14 Aug 09:26:48.279 # Cluster state changed: ok
39690:S 14 Aug 09:26:59.014 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: master without slots is reachable again.
39690:S 14 Aug 09:36:23.561 * FAIL message received from a9ce649f167b83dc477d195e62ca3fe9034db5e6 about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
39690:S 14 Aug 09:36:33.662 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:S 14 Aug 09:37:03.665 * FAIL message received from 6f1a37d66aacbcb0128c6306b5b989ad81c160c1 about 876f04626f806226755c98f15a8475c950a8446b
39690:S 14 Aug 09:37:03.666 # Cluster state changed: fail
39690:S 14 Aug 09:37:36.770 * Clear FAIL state for node 876f04626f806226755c98f15a8475c950a8446b: is reachable again and nobody is serving its slots after some time.
39690:S 14 Aug 09:37:36.770 # Cluster state changed: ok
39690:S 14 Aug 09:38:35.890 * FAIL message received from 876f04626f806226755c98f15a8475c950a8446b about fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff
39690:S 14 Aug 09:38:40.976 * Clear FAIL state for node fbb5b1d8d5433be8f5a98d8854fc87b3b266cdff: slave is reachable again.
39690:S 14 Aug 18:52:38.620 # IP address for this node updated to 192.168.1.154