19:10:37.612 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:10:37.802 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:10:42.613 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:10:42.803 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:10:50.734 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:10:55.742 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:11:05.036 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:11:10.645 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:11:17.481 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:11:19.263 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:11:22.533 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:11:26.488 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:11:27.280 # Cluster state changed: fail redis_10.200.5.117.log:16009:M 19:11:32.280 # Cluster state changed: ok redis_10.200.5.117.log:16009:M 19:11:34.838 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:11:34.975 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:11:36.088 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:11:40.009 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:11:41.078 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:11:41.823 * Marking node 5440860bf0c56106f22384be3bc7f74eccc318ca as failing (quorum reached). redis_10.200.2.171.log:1200:S 19:11:41.824 # Cluster state changed: fail redis_10.200.2.171.log:1200:S 19:11:41.877 # Start of election delayed for 602 milliseconds (rank #0, offset 2753804932435). redis_10.200.2.171.log:1200:S 19:11:42.454 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:11:42.479 # Starting a failover election for epoch 89. redis_10.200.2.171.log:1200:S 19:11:42.523 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.100.85.log:21636:M 19:11:43.187 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.2.235.log:1200:M 19:11:43.254 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.5.117.log:16009:M 19:11:43.881 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.100.58.log:22037:M 19:11:44.231 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.5.228.log:1205:M 19:11:44.387 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.101.90.log:35522:M 19:11:44.421 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.100.194.log:10116:M 19:11:44.988 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.100.99.log:34158:M 19:11:46.954 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.100.72.log:22084:M 19:11:47.899 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:11:48.899 # Cluster state changed: fail redis_10.200.100.187.log:22016:S 19:11:48.951 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:11:49.067 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:11:49.518 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:11:50.074 # Cluster state changed: ok redis_10.200.100.187.log:22016:S 19:11:50.083 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:11:51.455 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:11:51.643 * Marking node a8f9373511a75b9c97c1a6ffe017c31ca8725a48 as failing (quorum reached). redis_10.200.100.55.log:21922:S 19:11:51.644 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:11:51.893 # Currently unable to failover: Waiting for votes, but majority still not reached. redis_10.200.2.171.log:1200:S 19:11:52.006 # Cluster state changed: fail redis_10.200.101.90.log:35522:M 19:11:52.495 # Currently unable to failover: Failover attempt expired. redis_10.200.2.171.log:1200:S 19:11:52.903 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:11:53.038 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:11:53.038 * Marking node 5440860bf0c56106f22384be3bc7f74eccc318ca as failing (quorum reached). redis_10.200.100.58.log:22037:M 19:11:53.040 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.101.90.log:35522:M 19:11:53.140 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.85.log:21636:M 19:11:53.141 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.101.8.log:8114:S 19:11:53.142 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:11:53.146 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.88.log:21852:S 19:11:53.147 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:11:53.152 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:11:53.152 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.99.log:34158:M 19:11:53.152 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.3.31.log:15815:S 19:11:53.155 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:11:53.155 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.72.log:22084:M 19:11:53.155 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.101.179.log:2261:S 19:11:53.157 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.55.log:21922:S 19:11:53.158 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:11:53.158 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.5.91.log:15823:S 19:11:53.165 # Cluster state changed: fail redis_10.200.100.187.log:22016:S 19:11:53.165 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.187.log:22016:S 19:11:53.167 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.49.log:646447:S 19:11:53.168 # Cluster state changed: fail redis_10.200.100.49.log:646447:S 19:11:53.589 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.2.235.log:1200:M 19:11:53.590 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:11:54.844 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.2.171.log:1200:S 19:11:54.844 # Cluster state changed: ok redis_10.200.2.171.log:1200:S 19:11:55.471 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.5.228.log:1205:M 19:11:55.514 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:12:02.424 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: is reachable again and nobody is serving its slots after some time. redis_10.200.100.55.log:21922:S 19:12:03.212 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.101.179.log:2261:S 19:12:03.212 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:12:03.218 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.3.31.log:15815:S 19:12:03.219 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.5.91.log:15823:S 19:12:03.219 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:12:03.220 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.100.187.log:22016:S 19:12:03.220 # Cluster state changed: ok redis_10.200.100.187.log:22016:S 19:12:03.220 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:12:03.223 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.100.55.log:21922:S 19:12:03.223 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:12:04.637 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.100.99.log:34158:M 19:12:04.638 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:12:04.639 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.100.72.log:22084:M 19:12:04.639 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:12:04.641 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.2.235.log:1200:M 19:12:04.648 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:12:04.752 # Cluster state changed: fail redis_10.200.2.171.log:1200:S 19:12:04.752 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.2.171.log:1200:S 19:12:04.754 * Marking node 5440860bf0c56106f22384be3bc7f74eccc318ca as failing (quorum reached). redis_10.200.100.194.log:10116:M 19:12:04.762 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:12:04.769 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.5.91.log:15823:S 19:12:04.770 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:12:04.817 # Start of election delayed for 754 milliseconds (rank #0, offset 2753806750215). redis_10.200.2.171.log:1200:S 19:12:04.870 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:12:04.870 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.101.179.log:2261:S 19:12:04.875 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:12:04.875 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.55.log:21922:S 19:12:04.880 # Cluster state changed: fail redis_10.200.100.187.log:22016:S 19:12:04.880 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.187.log:22016:S 19:12:04.881 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:12:04.881 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.3.31.log:15815:S 19:12:05.348 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.101.90.log:35522:M 19:12:05.349 # Cluster state changed: ok redis_10.200.101.90.log:35522:M 19:12:05.623 # Starting a failover election for epoch 90. redis_10.200.2.171.log:1200:S 19:12:05.631 # Failover auth granted to 127fc5b043bd75068aebd810c40673cb514859f7 for epoch 90 redis_10.200.5.228.log:1205:M 19:12:05.633 # Failover auth granted to 127fc5b043bd75068aebd810c40673cb514859f7 for epoch 90 redis_10.200.100.85.log:21636:M 19:12:05.636 # Failover auth granted to 127fc5b043bd75068aebd810c40673cb514859f7 for epoch 90 redis_10.200.100.194.log:10116:M 19:12:05.636 # Failover auth granted to 127fc5b043bd75068aebd810c40673cb514859f7 for epoch 90 redis_10.200.100.58.log:22037:M 19:12:05.951 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.5.228.log:1205:M 19:12:05.951 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:12:05.957 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: is reachable again and nobody is serving its slots after some time. redis_10.200.100.58.log:22037:M 19:12:05.957 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:12:06.337 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.2.235.log:1200:M 19:12:06.351 # Failover auth granted to 127fc5b043bd75068aebd810c40673cb514859f7 for epoch 90 redis_10.200.2.235.log:1200:M 19:12:06.357 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:12:06.471 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.99.log:34158:M 19:12:06.477 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:12:06.479 * Caching the disconnected master state. redis_10.200.2.171.log:1200:M 19:12:06.479 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:12:06.479 # configEpoch set to 90 after successful failover redis_10.200.2.171.log:1200:S 19:12:06.479 # Connection with master lost. redis_10.200.2.171.log:1200:M 19:12:06.479 * Discarding previously cached master state. redis_10.200.2.171.log:1200:M 19:12:06.479 # Failover election won: I'm the new master. redis_10.200.2.171.log:1200:S 19:12:06.489 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:12:06.491 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:12:06.492 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:12:06.492 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:12:06.498 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:12:06.499 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:12:06.499 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:12:06.500 # Cluster state changed: ok redis_10.200.100.187.log:22016:S 19:12:06.504 # Cluster state changed: ok redis_10.200.100.49.log:646447:S 19:12:06.505 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:12:06.528 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:12:06.624 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.100.88.log:21852:S 19:12:06.625 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.101.8.log:8114:S 19:12:06.631 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.100.49.log:646447:S 19:12:06.632 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.5.117.log:16009:M 19:12:06.645 # Failover auth granted to 127fc5b043bd75068aebd810c40673cb514859f7 for epoch 90 redis_10.200.100.99.log:34158:M 19:12:06.656 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:12:06.657 # Configuration change detected. Reconfiguring myself as a replica of 127fc5b043bd75068aebd810c40673cb514859f7 redis_10.200.5.117.log:16009:M 19:12:06.657 # Connection with slave 10.200.2.171:6003 lost. redis_10.200.5.117.log:16009:S 19:12:06.841 * Connecting to MASTER 10.200.2.171:6003 redis_10.200.5.117.log:16009:S 19:12:06.842 * MASTER <-> SLAVE sync started redis_10.200.5.117.log:16009:S 19:12:06.843 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.5.91.log:15823:S 19:12:06.848 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.100.187.log:22016:S 19:12:06.849 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.3.31.log:15815:S 19:12:06.853 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.100.55.log:21922:S 19:12:06.854 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.101.179.log:2261:S 19:12:06.916 * Non blocking connect for SYNC fired the event. redis_10.200.5.117.log:16009:S 19:12:06.991 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.2.171.log:1200:M 19:12:07.597 # Failover auth denied to 127fc5b043bd75068aebd810c40673cb514859f7: its master is up redis_10.200.101.90.log:35522:M 19:12:08.048 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.72.log:22084:M 19:12:08.069 # Failover auth granted to 127fc5b043bd75068aebd810c40673cb514859f7 for epoch 90 redis_10.200.100.72.log:22084:M 19:12:10.275 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: master without slots is reachable again. redis_10.200.2.235.log:1200:M 19:12:10.549 * Master replied to PING, replication can continue... redis_10.200.5.117.log:16009:S 19:12:11.702 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.99.log:34158:M 19:12:11.712 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:12:11.743 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:12:11.802 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:12:11.817 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.72.log:22084:M 19:12:11.825 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:12:12.336 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:12:12.390 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:12:12.576 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:12:12.596 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:12:12.714 # Cluster state changed: fail redis_10.200.101.90.log:35522:M 19:12:12.916 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:12:12.926 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:12:12.984 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:12:13.100 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:12:13.358 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:12:13.419 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:12:13.572 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:12:13.578 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.85.log:21636:M 19:12:13.634 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.194.log:10116:M 19:12:13.689 * Partial resynchronization not possible (no cached master) redis_10.200.5.117.log:16009:S 19:12:13.759 * Full resync requested by slave 10.200.5.117:6000 redis_10.200.2.171.log:1200:M 19:12:13.759 * Slave 10.200.5.117:6000 asks for synchronization redis_10.200.2.171.log:1200:M 19:12:13.759 * Starting BGSAVE for SYNC with target: disk redis_10.200.2.171.log:1200:M 19:12:13.902 * Background saving started by pid 1528 redis_10.200.2.171.log:1200:M 19:12:13.907 * Full resync from master: 2cb47bb8d4737966c10c963f73b376baace782b5:2753806881534 redis_10.200.5.117.log:16009:S 19:12:15.143 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:12:17.044 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:12:17.490 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:12:17.733 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:12:18.126 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:12:18.175 # Cluster state changed: ok redis_10.200.101.90.log:35522:M 19:12:18.568 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:12:18.652 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:12:18.673 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:12:20.241 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:12:24.402 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:12:24.880 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:12:25.060 # Cluster state changed: fail redis_10.200.101.90.log:35522:M 19:12:25.212 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:12:25.379 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:12:25.569 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:12:25.648 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:12:26.010 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:12:26.372 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:12:30.027 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:12:30.154 # Cluster state changed: ok redis_10.200.101.90.log:35522:M 19:12:30.220 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:12:30.376 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:12:30.401 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:12:30.674 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:12:30.695 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:12:31.104 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:12:31.839 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:12:36.210 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:12:36.268 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:12:36.566 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:12:37.014 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:12:37.118 # Cluster state changed: fail redis_10.200.101.90.log:35522:M 19:12:37.122 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:12:37.124 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:12:37.308 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:12:37.562 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:12:41.212 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:12:41.280 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:12:41.613 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:12:42.081 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:12:42.124 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:12:42.143 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:12:42.166 # Cluster state changed: ok redis_10.200.101.90.log:35522:M 19:12:42.509 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:12:42.621 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:12:47.422 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:12:47.480 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:12:48.291 # Cluster state changed: fail redis_10.200.101.90.log:35522:M 19:12:48.683 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:12:53.135 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:12:53.287 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:12:53.708 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:12:53.768 # Cluster state changed: ok redis_10.200.101.90.log:35522:M 19:12:56.116 * DB saved on disk redis_10.200.2.171.log:1528:C 19:12:56.202 * RDB: 733 MB of memory used by copy-on-write redis_10.200.2.171.log:1528:C 19:12:56.451 * Background saving terminated with success redis_10.200.2.171.log:1200:M 19:12:56.468 * MASTER <-> SLAVE sync: receiving 1407591020 bytes from master redis_10.200.5.117.log:16009:S 19:13:00.265 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:13:00.577 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:13:00.880 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:13:01.413 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:13:01.871 # Cluster state changed: fail redis_10.200.101.90.log:35522:M 19:13:01.905 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:13:01.918 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:13:02.320 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:13:05.950 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:13:06.543 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:13:06.612 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:13:06.982 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:13:06.996 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:13:07.066 # Cluster state changed: ok redis_10.200.101.90.log:35522:M 19:13:07.083 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:13:07.382 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:13:12.665 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:13:13.040 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:13:13.054 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:13:13.148 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:13:13.157 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:13:13.200 # Cluster state changed: fail redis_10.200.100.85.log:21636:M 19:13:13.235 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:13:13.247 # Cluster state changed: fail redis_10.200.100.187.log:22016:S 19:13:13.342 # Cluster state changed: ok redis_10.200.100.187.log:22016:S 19:13:13.342 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:13:13.345 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:13:13.350 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:13:13.351 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:13:13.455 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:13:13.498 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:13:13.505 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:13:13.518 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:13:13.720 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:13:14.105 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:13:14.383 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.2.171.log:1200:M 19:13:14.384 * Marking node a8f9373511a75b9c97c1a6ffe017c31ca8725a48 as failing (quorum reached). redis_10.200.100.85.log:21636:M 19:13:14.386 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.99.log:34158:M 19:13:14.388 # Cluster state changed: fail redis_10.200.5.117.log:16009:S 19:13:14.388 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.5.117.log:16009:S 19:13:14.389 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:13:14.389 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.101.179.log:2261:S 19:13:14.390 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.49.log:646447:S 19:13:14.390 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.72.log:22084:M 19:13:14.391 # Cluster state changed: fail redis_10.200.100.49.log:646447:S 19:13:14.391 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.58.log:22037:M 19:13:14.392 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.5.228.log:1205:M 19:13:14.397 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:13:14.397 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:13:14.397 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.88.log:21852:S 19:13:14.397 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.101.8.log:8114:S 19:13:14.400 # Cluster state changed: fail redis_10.200.100.187.log:22016:S 19:13:14.400 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:13:14.400 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:13:14.400 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.187.log:22016:S 19:13:14.400 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.3.31.log:15815:S 19:13:14.400 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.5.91.log:15823:S 19:13:14.404 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.55.log:21922:S 19:13:14.406 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:13:14.420 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.2.235.log:1200:M 19:13:14.441 # Start of election delayed for 956 milliseconds (rank #0, offset 2736440983035). redis_10.200.100.49.log:646447:S 19:13:14.531 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:13:14.547 # Cluster state changed: fail redis_10.200.101.90.log:35522:M 19:13:14.674 * FAIL message received from aee685a8f8747185f5463ce0a61821e24b2de18b about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.194.log:10116:M 19:13:15.452 # Starting a failover election for epoch 91. redis_10.200.100.49.log:646447:S 19:13:15.459 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.5.228.log:1205:M 19:13:15.460 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.2.171.log:1200:M 19:13:15.463 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.2.235.log:1200:M 19:13:15.475 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.100.99.log:34158:M 19:13:15.512 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.100.85.log:21636:M 19:13:15.524 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.100.72.log:22084:M 19:13:15.538 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.100.58.log:22037:M 19:13:15.584 * Caching the disconnected master state. redis_10.200.100.49.log:646447:M 19:13:15.584 # configEpoch set to 91 after successful failover redis_10.200.100.49.log:646447:S 19:13:15.584 # Connection with master lost. redis_10.200.100.49.log:646447:M 19:13:15.584 # Failover election won: I'm the new master. redis_10.200.100.49.log:646447:S 19:13:15.585 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:13:15.585 * Discarding previously cached master state. redis_10.200.100.49.log:646447:M 19:13:15.589 # Cluster state changed: ok redis_10.200.5.117.log:16009:S 19:13:15.591 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:13:15.592 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:13:15.596 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:13:15.597 # Cluster state changed: ok redis_10.200.100.187.log:22016:S 19:13:15.597 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:13:15.602 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:13:15.604 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:13:15.633 # Failover auth denied to 34efe005d6275cb0045156baa277d90c07ff0284: its master is up redis_10.200.101.90.log:35522:M 19:13:15.645 # Configuration change detected. Reconfiguring myself as a replica of 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.101.90.log:35522:M 19:13:15.645 # Connection with slave client id #61399902 lost. redis_10.200.101.90.log:35522:S 19:13:15.665 # Failover auth granted to 34efe005d6275cb0045156baa277d90c07ff0284 for epoch 91 redis_10.200.100.194.log:10116:M 19:13:15.736 # Cluster state changed: ok redis_10.200.101.90.log:35522:S 19:13:16.269 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: master without slots is reachable again. redis_10.200.5.91.log:15823:S 19:13:16.670 * Connecting to MASTER 10.200.100.49:6379 redis_10.200.101.90.log:35522:S 19:13:16.670 * MASTER <-> SLAVE sync started redis_10.200.101.90.log:35522:S 19:13:16.671 * Non blocking connect for SYNC fired the event. redis_10.200.101.90.log:35522:S 19:13:17.168 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: master without slots is reachable again. redis_10.200.5.117.log:16009:S 19:13:17.199 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.101.179.log:2261:S 19:13:17.219 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: master without slots is reachable again. redis_10.200.100.99.log:34158:M 19:13:17.258 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: master without slots is reachable again. redis_10.200.3.31.log:15815:S 19:13:17.263 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: master without slots is reachable again. redis_10.200.100.187.log:22016:S 19:13:17.269 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: master without slots is reachable again. redis_10.200.100.55.log:21922:S 19:13:17.278 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.88.log:21852:S 19:13:17.296 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.72.log:22084:M 19:13:17.353 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.2.235.log:1200:M 19:13:17.450 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: master without slots is reachable again. redis_10.200.2.171.log:1200:M 19:13:17.474 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.58.log:22037:M 19:13:17.476 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.85.log:21636:M 19:13:17.478 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.5.228.log:1205:M 19:13:17.554 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.101.8.log:8114:S 19:13:17.673 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.194.log:10116:M 19:13:18.464 * Master replied to PING, replication can continue... redis_10.200.101.90.log:35522:S 19:13:18.523 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:13:18.577 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:13:18.683 # Cluster state changed: ok redis_10.200.100.85.log:21636:M 19:13:18.745 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:13:18.799 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:13:18.879 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:13:18.928 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.49.log:646447:M 19:13:19.151 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:13:19.366 * Partial resynchronization not possible (no cached master) redis_10.200.101.90.log:35522:S 19:13:19.535 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:13:19.648 * Full resync requested by slave 10.200.101.90:6379 redis_10.200.100.49.log:646447:M 19:13:19.648 * Slave 10.200.101.90:6379 asks for synchronization redis_10.200.100.49.log:646447:M 19:13:19.648 * Starting BGSAVE for SYNC with target: disk redis_10.200.100.49.log:646447:M 19:13:19.784 * Background saving started by pid 278786 redis_10.200.100.49.log:646447:M 19:13:19.787 * Full resync from master: 093fc644b07b4793d4d8241fb5c03435213296a0:2736441669994 redis_10.200.101.90.log:35522:S 19:13:24.527 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:13:24.764 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:13:24.796 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:13:25.150 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:13:25.252 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:13:25.271 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:13:25.605 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:13:25.725 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.99.log:34158:M 19:13:25.725 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.2.235.log:1200:M 19:13:25.725 * Marking node aee685a8f8747185f5463ce0a61821e24b2de18b as failing (quorum reached). redis_10.200.100.194.log:10116:M 19:13:25.727 # Cluster state changed: fail redis_10.200.100.187.log:22016:S 19:13:25.727 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.187.log:22016:S 19:13:25.727 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.2.171.log:1200:M 19:13:25.728 # Cluster state changed: fail redis_10.200.5.117.log:16009:S 19:13:25.728 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.5.117.log:16009:S 19:13:25.729 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:13:25.729 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.55.log:21922:S 19:13:25.729 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.72.log:22084:M 19:13:25.729 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.5.228.log:1205:M 19:13:25.730 # Cluster state changed: fail redis_10.200.101.90.log:35522:S 19:13:25.730 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.101.90.log:35522:S 19:13:25.734 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:13:25.734 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.101.179.log:2261:S 19:13:25.738 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:13:25.738 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.101.8.log:8114:S 19:13:25.741 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:13:25.741 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.88.log:21852:S 19:13:25.742 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:13:25.742 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.3.31.log:15815:S 19:13:25.743 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:13:25.743 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.5.91.log:15823:S 19:13:25.792 # Start of election delayed for 563 milliseconds (rank #0, offset 2130333340877). redis_10.200.100.187.log:22016:S 19:13:25.806 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.58.log:22037:M 19:13:26.032 * FAIL message received from 93a628522165203a7ac85a35550879a66fe92142 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.49.log:646447:M 19:13:26.033 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:13:26.392 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.2.171.log:1200:M 19:13:26.393 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.5.228.log:1205:M 19:13:26.394 # Starting a failover election for epoch 92. redis_10.200.100.187.log:22016:S 19:13:26.395 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.100.99.log:34158:M 19:13:26.395 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.2.235.log:1200:M 19:13:26.398 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.100.72.log:22084:M 19:13:26.399 * Caching the disconnected master state. redis_10.200.100.187.log:22016:M 19:13:26.399 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:13:26.399 # configEpoch set to 92 after successful failover redis_10.200.100.187.log:22016:S 19:13:26.399 # Connection with master lost. redis_10.200.100.187.log:22016:M 19:13:26.399 * Discarding previously cached master state. redis_10.200.100.187.log:22016:M 19:13:26.399 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.100.194.log:10116:M 19:13:26.399 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.100.49.log:646447:M 19:13:26.399 # Failover auth granted to bb957a8b15470f47cb5b0008bbf7fc0474630213 for epoch 92 redis_10.200.100.58.log:22037:M 19:13:26.399 # Failover election won: I'm the new master. redis_10.200.100.187.log:22016:S 19:13:26.404 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:13:26.406 # Cluster state changed: ok redis_10.200.101.90.log:35522:S 19:13:26.423 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:13:26.439 # Cluster state changed: ok redis_10.200.5.117.log:16009:S 19:13:26.446 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:13:26.447 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:13:26.450 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:13:26.451 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:13:26.455 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:13:26.951 # Connection with slave 10.200.100.187:6379 lost. redis_10.200.100.85.log:21636:M 19:13:27.061 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.100.49.log:646447:M 19:13:27.341 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.2.171.log:1200:M 19:13:27.343 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.2.235.log:1200:M 19:13:27.350 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.100.72.log:22084:M 19:13:27.352 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.100.58.log:22037:M 19:13:27.353 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.5.228.log:1205:M 19:13:27.358 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.100.99.log:34158:M 19:13:27.358 # Configuration change detected. Reconfiguring myself as a replica of bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.85.log:21636:M 19:13:27.362 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.100.194.log:10116:M 19:13:27.969 * Connecting to MASTER 10.200.100.187:6379 redis_10.200.100.85.log:21636:S 19:13:27.969 * MASTER <-> SLAVE sync started redis_10.200.100.85.log:21636:S 19:13:27.969 * Non blocking connect for SYNC fired the event. redis_10.200.100.85.log:21636:S 19:13:28.532 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.5.91.log:15823:S 19:13:28.534 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.101.8.log:8114:S 19:13:28.534 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.101.179.log:2261:S 19:13:28.546 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.101.90.log:35522:S 19:13:28.559 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.55.log:21922:S 19:13:28.567 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.88.log:21852:S 19:13:28.578 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.5.117.log:16009:S 19:13:28.616 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: master without slots is reachable again. redis_10.200.3.31.log:15815:S 19:13:29.798 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:13:29.808 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:13:29.851 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:13:30.325 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:13:30.338 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:13:30.369 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:13:30.391 * Master replied to PING, replication can continue... redis_10.200.100.85.log:21636:S 19:13:30.805 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:13:30.907 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.187.log:22016:M 19:13:31.371 * Partial resynchronization not possible (no cached master) redis_10.200.100.85.log:21636:S 19:13:31.681 * Full resync requested by slave 10.200.100.85:6379 redis_10.200.100.187.log:22016:M 19:13:31.681 * Slave 10.200.100.85:6379 asks for synchronization redis_10.200.100.187.log:22016:M 19:13:31.681 * Starting BGSAVE for SYNC with target: disk redis_10.200.100.187.log:22016:M 19:13:31.814 * Full resync from master: bb160fa66d9a7cb4fce0c297fbafc2ec575f4e4d:2130333406460 redis_10.200.100.85.log:21636:S 19:13:31.816 * Background saving started by pid 663877 redis_10.200.100.187.log:22016:M 19:13:35.939 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:13:36.287 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:13:36.685 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:13:36.709 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:13:36.754 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:13:36.925 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:13:38.790 * Synchronization with slave 10.200.5.117:6000 succeeded redis_10.200.2.171.log:1200:M 19:13:39.060 * MASTER <-> SLAVE sync: Flushing old data redis_10.200.5.117.log:16009:S 19:13:41.446 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:13:41.693 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:13:41.725 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:13:41.771 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:13:41.812 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:13:41.999 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:13:47.512 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:13:47.743 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:13:48.533 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:13:48.971 * Marking node 5440860bf0c56106f22384be3bc7f74eccc318ca as failing (quorum reached). redis_10.200.101.90.log:35522:S 19:13:49.314 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.5.228.log:1205:M 19:13:49.320 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.85.log:21636:S 19:13:49.320 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.99.log:34158:M 19:13:49.321 * Marking node 5440860bf0c56106f22384be3bc7f74eccc318ca as failing (quorum reached). redis_10.200.100.72.log:22084:M 19:13:49.323 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.187.log:22016:M 19:13:49.323 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.49.log:646447:M 19:13:49.323 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.101.179.log:2261:S 19:13:49.325 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.55.log:21922:S 19:13:49.328 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.3.31.log:15815:S 19:13:49.334 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.88.log:21852:S 19:13:49.334 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.101.8.log:8114:S 19:13:49.339 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.5.91.log:15823:S 19:13:49.571 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.2.171.log:1200:M 19:13:50.153 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.100.58.log:22037:M 19:13:50.179 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5440860bf0c56106f22384be3bc7f74eccc318ca redis_10.200.2.235.log:1200:M 19:13:50.812 * Marking node 5440860bf0c56106f22384be3bc7f74eccc318ca as failing (quorum reached). redis_10.200.100.194.log:10116:M 19:13:52.712 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:13:52.773 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:13:53.544 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:13:59.086 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:13:59.898 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:14:02.153 * DB saved on disk redis_10.200.100.49.log:278786:C 19:14:02.266 * RDB: 570 MB of memory used by copy-on-write redis_10.200.100.49.log:278786:C 19:14:02.602 * Background saving terminated with success redis_10.200.100.49.log:646447:M 19:14:02.627 * MASTER <-> SLAVE sync: receiving 1453362974 bytes from master redis_10.200.101.90.log:35522:S 19:14:04.937 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:14:05.087 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:14:10.472 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:14:10.531 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:14:10.669 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:14:11.888 * MASTER <-> SLAVE sync: Loading DB in memory redis_10.200.5.117.log:16009:S 19:14:12.001 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.2.171.log:1200:M 19:14:12.001 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.5.228.log:1205:M 19:14:12.008 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.85.log:21636:S 19:14:12.010 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.187.log:22016:M 19:14:12.010 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.72.log:22084:M 19:14:12.010 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.101.179.log:2261:S 19:14:12.011 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.101.8.log:8114:S 19:14:12.012 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.55.log:21922:S 19:14:12.015 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.101.90.log:35522:S 19:14:12.018 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.3.31.log:15815:S 19:14:12.019 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.88.log:21852:S 19:14:12.023 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.5.91.log:15823:S 19:14:12.436 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:14:12.648 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.49.log:646447:M 19:14:13.057 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.58.log:22037:M 19:14:13.202 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.99.log:34158:M 19:14:13.492 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:14:13.492 * Marking node 34efe005d6275cb0045156baa277d90c07ff0284 as failing (quorum reached). redis_10.200.2.235.log:1200:M 19:14:13.498 # Cluster state changed: fail redis_10.200.100.85.log:21636:S 19:14:13.498 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.85.log:21636:S 19:14:13.502 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:14:13.502 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.55.log:21922:S 19:14:13.504 # Cluster state changed: fail redis_10.200.101.90.log:35522:S 19:14:13.504 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.101.90.log:35522:S 19:14:13.512 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:14:13.512 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.88.log:21852:S 19:14:13.580 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.2.171.log:1200:M 19:14:13.589 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:14:13.589 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.72.log:22084:M 19:14:13.589 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.3.31.log:15815:S 19:14:13.590 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.5.228.log:1205:M 19:14:13.591 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:14:13.591 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:14:13.591 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.101.179.log:2261:S 19:14:13.591 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.101.8.log:8114:S 19:14:13.596 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:14:13.596 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.5.91.log:15823:S 19:14:13.669 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.2.235.log:1200:M 19:14:13.951 * Clear FAIL state for node 5440860bf0c56106f22384be3bc7f74eccc318ca: slave is reachable again. redis_10.200.100.194.log:10116:M 19:14:13.952 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:14:13.952 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.194.log:10116:M 19:14:14.793 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.58.log:22037:M 19:14:14.932 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:14:15.046 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.99.log:34158:M 19:14:15.047 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:14:16.783 * DB saved on disk redis_10.200.100.187.log:663877:C 19:14:16.972 * RDB: 815 MB of memory used by copy-on-write redis_10.200.100.187.log:663877:C 19:14:17.540 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about 34efe005d6275cb0045156baa277d90c07ff0284 redis_10.200.100.187.log:22016:M 19:14:17.542 * Background saving terminated with success redis_10.200.100.187.log:22016:M 19:14:17.543 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:14:17.635 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:14:17.659 * MASTER <-> SLAVE sync: receiving 1408062523 bytes from master redis_10.200.100.85.log:21636:S 19:14:23.514 # Currently unable to failover: Disconnected from master for longer than allowed. Please check the 'cluster-slave-validity-factor' configuration option. redis_10.200.101.90.log:35522:S 19:14:24.294 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.2.171.log:1200:M 19:14:24.294 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.5.228.log:1205:M 19:14:24.294 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:14:24.294 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:14:24.295 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.2.235.log:1200:M 19:14:24.295 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:14:24.303 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.72.log:22084:M 19:14:24.304 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.194.log:10116:M 19:14:24.304 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:14:24.305 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:14:24.313 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.88.log:21852:S 19:14:24.314 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.55.log:21922:S 19:14:24.314 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:14:24.314 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:14:26.176 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.85.log:21636:S 19:14:26.176 # Cluster state changed: ok redis_10.200.100.85.log:21636:S 19:14:26.179 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.101.179.log:2261:S 19:14:26.179 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:14:26.181 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.101.90.log:35522:S 19:14:26.181 # Cluster state changed: ok redis_10.200.101.90.log:35522:S 19:14:26.182 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.101.8.log:8114:S 19:14:26.182 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:14:26.188 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.5.91.log:15823:S 19:14:26.188 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:14:26.192 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.3.31.log:15815:S 19:14:26.192 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:14:29.213 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.99.log:34158:M 19:14:29.213 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:14:29.218 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.58.log:22037:M 19:14:29.218 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:14:30.355 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:14:30.485 * Clear FAIL state for node 34efe005d6275cb0045156baa277d90c07ff0284: is reachable again and nobody is serving its slots after some time. redis_10.200.100.187.log:22016:M 19:14:31.038 # Cluster state changed: fail redis_10.200.101.90.log:35522:S 19:14:31.329 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:14:31.361 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:14:31.376 # Cluster state changed: ok redis_10.200.101.90.log:35522:S 19:14:31.378 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:14:31.382 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:14:31.427 # Cluster state changed: fail redis_10.200.100.85.log:21636:S 19:14:31.428 # Cluster state changed: ok redis_10.200.100.85.log:21636:S 19:14:32.277 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:14:32.764 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:14:32.913 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:14:33.333 * MASTER <-> SLAVE sync: Flushing old data redis_10.200.100.85.log:21636:S 19:14:33.334 * Synchronization with slave 10.200.100.85:6379 succeeded redis_10.200.100.187.log:22016:M 19:14:36.302 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:14:37.295 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:14:37.335 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:14:37.805 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:14:37.934 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:14:42.791 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:14:43.187 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:14:43.263 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:14:44.306 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:14:44.327 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:14:44.348 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:14:44.507 * Marking node aee685a8f8747185f5463ce0a61821e24b2de18b as failing (quorum reached). redis_10.200.101.90.log:35522:S 19:14:44.515 * Marking node f24444696fb319fb6c85dd9e8e01d9b182852070 as failing (quorum reached). redis_10.200.101.179.log:2261:S 19:14:44.517 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:14:45.243 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:14:45.244 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:14:45.781 * Marking node aee685a8f8747185f5463ce0a61821e24b2de18b as failing (quorum reached). redis_10.200.101.8.log:8114:S 19:14:45.792 * Marking node aee685a8f8747185f5463ce0a61821e24b2de18b as failing (quorum reached). redis_10.200.100.55.log:21922:S 19:14:45.953 * Marking node aee685a8f8747185f5463ce0a61821e24b2de18b as failing (quorum reached). redis_10.200.2.235.log:1200:M 19:14:45.955 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.2.171.log:1200:M 19:14:45.955 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.5.228.log:1205:M 19:14:45.961 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.99.log:34158:M 19:14:45.964 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.72.log:22084:M 19:14:45.965 * Marking node aee685a8f8747185f5463ce0a61821e24b2de18b as failing (quorum reached). redis_10.200.100.194.log:10116:M 19:14:45.971 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.88.log:21852:S 19:14:45.973 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.101.179.log:2261:S 19:14:45.978 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.3.31.log:15815:S 19:14:45.979 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.5.91.log:15823:S 19:14:45.994 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.49.log:646447:M 19:14:46.009 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.5.117.log:16009:S 19:14:46.030 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.187.log:22016:M 19:14:46.346 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about aee685a8f8747185f5463ce0a61821e24b2de18b redis_10.200.100.58.log:22037:M 19:14:48.195 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:14:49.344 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:14:49.434 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:14:49.481 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:14:49.578 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:14:49.691 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:14:50.288 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:14:50.373 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:14:55.472 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:14:55.698 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:14:55.838 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:14:55.848 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:14:55.970 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:14:56.518 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:14:56.568 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:14:56.605 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: is reachable again and nobody is serving its slots after some time. redis_10.200.101.179.log:2261:S 19:14:56.605 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:14:56.701 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:14:57.197 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:14:57.403 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:14:57.743 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:15:01.238 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:15:01.286 * Synchronization with slave 10.200.101.90:6379 succeeded redis_10.200.100.49.log:646447:M 19:15:01.573 * MASTER <-> SLAVE sync: Flushing old data redis_10.200.101.90.log:35522:S 19:15:01.609 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:15:01.655 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:15:01.741 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:15:01.786 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:15:01.800 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:15:02.411 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:15:02.677 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:15:02.770 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:15:07.690 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:15:07.781 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:15:08.184 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:15:08.652 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:15:08.775 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:15:08.964 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:15:09.183 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:15:09.817 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:15:10.002 * Marking node a8f9373511a75b9c97c1a6ffe017c31ca8725a48 as failing (quorum reached). redis_10.200.2.171.log:1200:M 19:15:10.006 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.5.228.log:1205:M 19:15:10.011 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.194.log:10116:M 19:15:10.011 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.72.log:22084:M 19:15:10.012 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.49.log:646447:M 19:15:10.012 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.58.log:22037:M 19:15:10.013 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.187.log:22016:M 19:15:10.017 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.88.log:21852:S 19:15:10.020 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.101.8.log:8114:S 19:15:10.021 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.55.log:21922:S 19:15:10.024 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.5.91.log:15823:S 19:15:10.027 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.101.179.log:2261:S 19:15:10.027 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.3.31.log:15815:S 19:15:10.063 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.2.235.log:1200:M 19:15:10.082 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.5.117.log:16009:S 19:15:10.863 * MASTER <-> SLAVE sync: Loading DB in memory redis_10.200.100.85.log:21636:S 19:15:11.003 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.85.log:21636:S 19:15:11.014 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.2.171.log:1200:M 19:15:11.016 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.5.228.log:1205:M 19:15:11.019 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.72.log:22084:M 19:15:11.020 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.194.log:10116:M 19:15:11.020 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.101.8.log:8114:S 19:15:11.021 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.58.log:22037:M 19:15:11.021 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.2.235.log:1200:M 19:15:11.022 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.49.log:646447:M 19:15:11.028 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.5.91.log:15823:S 19:15:11.029 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.55.log:21922:S 19:15:11.032 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.88.log:21852:S 19:15:11.033 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.101.179.log:2261:S 19:15:11.034 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.187.log:22016:M 19:15:11.037 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.3.31.log:15815:S 19:15:11.902 * FAIL message received from 127fc5b043bd75068aebd810c40673cb514859f7 about a8f9373511a75b9c97c1a6ffe017c31ca8725a48 redis_10.200.100.99.log:34158:M 19:15:12.628 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.100.99.log:34158:M 19:15:13.657 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:15:13.884 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:15:13.908 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:15:13.911 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:15:13.914 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:15:14.240 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:15:14.331 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:15:14.833 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:15:19.866 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:15:20.456 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:15:20.606 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:15:20.686 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:15:21.022 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:15:21.449 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:15:21.570 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:15:25.615 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:15:26.001 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:15:26.004 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:15:26.012 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:15:26.050 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:15:26.492 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:15:26.619 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:15:31.806 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:15:31.924 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:15:32.062 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:15:32.074 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:15:32.200 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:15:32.397 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:15:33.090 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:15:33.273 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:15:37.244 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:15:37.401 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:15:37.481 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:15:38.106 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:15:38.151 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:15:38.260 * MASTER <-> SLAVE sync: Loading DB in memory redis_10.200.101.90.log:35522:S 19:15:38.384 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.101.90.log:35522:S 19:15:38.424 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.88.log:21852:S 19:15:38.424 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.101.179.log:2261:S 19:15:38.425 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.49.log:646447:M 19:15:38.425 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.101.8.log:8114:S 19:15:38.430 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.5.91.log:15823:S 19:15:38.432 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.3.31.log:15815:S 19:15:38.435 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.55.log:21922:S 19:15:38.462 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:15:39.510 * MASTER <-> SLAVE sync: Finished with success redis_10.200.5.117.log:16009:S 19:15:39.563 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.5.117.log:16009:S 19:15:39.639 * Clear FAIL state for node aee685a8f8747185f5463ce0a61821e24b2de18b: slave is reachable again. redis_10.200.5.117.log:16009:S 19:15:39.876 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.72.log:22084:M 19:15:40.247 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.99.log:34158:M 19:15:40.453 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.2.171.log:1200:M 19:15:40.494 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.5.228.log:1205:M 19:15:41.226 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.187.log:22016:M 19:15:41.788 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.58.log:22037:M 19:15:41.894 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.194.log:10116:M 19:15:42.677 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.2.235.log:1200:M 19:15:44.263 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:15:44.363 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:15:44.686 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:15:44.703 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:15:44.863 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:15:46.600 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:15:49.770 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:15:49.792 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:15:49.800 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:15:49.815 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:15:49.883 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:15:51.618 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:15:54.916 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:15:55.403 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:15:56.287 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:15:57.088 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:15:57.669 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:16:00.518 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:16:01.257 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:16:01.290 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:16:02.114 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:16:02.869 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:16:08.302 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:16:08.428 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:16:08.807 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:16:08.941 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:16:09.012 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:16:10.418 * Marking node bb957a8b15470f47cb5b0008bbf7fc0474630213 as failing (quorum reached). redis_10.200.101.179.log:2261:S 19:16:10.419 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:16:10.435 # Cluster state changed: fail redis_10.200.5.117.log:16009:S 19:16:10.435 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.5.117.log:16009:S 19:16:10.438 * Marking node bb957a8b15470f47cb5b0008bbf7fc0474630213 as failing (quorum reached). redis_10.200.100.58.log:22037:M 19:16:10.440 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.194.log:10116:M 19:16:10.440 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.72.log:22084:M 19:16:10.441 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:16:10.441 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.101.8.log:8114:S 19:16:10.442 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:16:10.442 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.3.31.log:15815:S 19:16:10.444 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:16:10.444 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.5.91.log:15823:S 19:16:10.446 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.5.228.log:1205:M 19:16:10.449 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.88.log:21852:S 19:16:10.449 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.2.235.log:1200:M 19:16:10.450 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:16:10.456 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:16:10.456 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.55.log:21922:S 19:16:10.489 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.101.90.log:35522:S 19:16:10.519 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.49.log:646447:M 19:16:10.528 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.85.log:21636:S 19:16:10.540 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:16:11.407 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.2.171.log:1200:M 19:16:11.449 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:16:11.500 * FAIL message received from f24444696fb319fb6c85dd9e8e01d9b182852070 about bb957a8b15470f47cb5b0008bbf7fc0474630213 redis_10.200.100.99.log:34158:M 19:16:11.510 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:16:21.099 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.5.228.log:1205:M 19:16:21.100 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:16:21.463 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.5.91.log:15823:S 19:16:21.463 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:16:21.466 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.2.171.log:1200:M 19:16:21.466 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:16:21.469 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.72.log:22084:M 19:16:21.469 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:16:21.470 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.58.log:22037:M 19:16:21.470 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:16:21.473 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.5.117.log:16009:S 19:16:21.474 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.3.31.log:15815:S 19:16:21.474 # Cluster state changed: ok redis_10.200.5.117.log:16009:S 19:16:21.475 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.101.179.log:2261:S 19:16:21.475 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:16:21.475 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:16:21.477 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.2.235.log:1200:M 19:16:21.478 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.101.8.log:8114:S 19:16:21.478 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:16:21.478 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:16:22.294 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.99.log:34158:M 19:16:22.295 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:16:22.538 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.49.log:646447:M 19:16:22.538 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:16:22.753 # Cluster state changed: fail redis_10.200.100.85.log:21636:S 19:16:22.753 * MASTER <-> SLAVE sync: Finished with success redis_10.200.100.85.log:21636:S 19:16:22.753 # Start of election delayed for 874 milliseconds (rank #0, offset 2130333406460). redis_10.200.100.85.log:21636:S 19:16:22.806 * Clear FAIL state for node a8f9373511a75b9c97c1a6ffe017c31ca8725a48: slave is reachable again. redis_10.200.100.85.log:21636:S 19:16:22.822 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.88.log:21852:S 19:16:22.822 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:16:22.825 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.55.log:21922:S 19:16:22.826 # Cluster state changed: ok redis_10.200.100.55.log:21922:S 19:16:22.855 # Currently unable to failover: Waiting the delay before I can start a new failover. redis_10.200.100.85.log:21636:S 19:16:22.976 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.194.log:10116:M 19:16:22.976 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.100.85.log:21636:S 19:16:22.976 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:16:22.976 # Cluster state changed: ok redis_10.200.100.85.log:21636:S 19:16:28.193 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:16:28.741 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:16:28.761 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:16:28.914 # Cluster state changed: fail redis_10.200.100.58.log:22037:M 19:16:29.180 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:16:33.597 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:16:33.854 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:16:33.867 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:16:33.956 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:16:34.211 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:16:39.910 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:16:39.987 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:16:40.258 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:16:40.875 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:16:45.006 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:16:45.253 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:16:45.253 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.5.91.log:15823:S 19:16:45.257 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:16:45.257 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.187.log:22016:M 19:16:45.257 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.88.log:21852:S 19:16:45.257 * Marking node f24444696fb319fb6c85dd9e8e01d9b182852070 as failing (quorum reached). redis_10.200.100.49.log:646447:M 19:16:45.258 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:16:45.258 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:16:45.258 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.101.8.log:8114:S 19:16:45.265 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.2.235.log:1200:M 19:16:45.266 # Cluster state changed: fail redis_10.200.5.117.log:16009:S 19:16:45.266 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.5.117.log:16009:S 19:16:45.359 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.2.171.log:1200:M 19:16:45.363 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:16:45.363 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:16:45.363 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.101.179.log:2261:S 19:16:45.363 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.3.31.log:15815:S 19:16:45.364 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.55.log:21922:S 19:16:45.365 # Cluster state changed: fail redis_10.200.100.55.log:21922:S 19:16:45.366 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.85.log:21636:S 19:16:45.367 # Cluster state changed: fail redis_10.200.100.85.log:21636:S 19:16:45.392 # Start of election delayed for 999 milliseconds (rank #0, offset 2140265041927). redis_10.200.100.55.log:21922:S 19:16:45.472 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.101.90.log:35522:S 19:16:46.254 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.99.log:34158:M 19:16:46.255 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:16:46.390 # Failover auth granted to 5c61d77335632daf5762c946bccda3bb07b9fa60 for epoch 93 redis_10.200.100.99.log:34158:M 19:16:46.390 # Failover auth granted to 5c61d77335632daf5762c946bccda3bb07b9fa60 for epoch 93 redis_10.200.2.171.log:1200:M 19:16:46.392 # Starting a failover election for epoch 93. redis_10.200.100.55.log:21922:S 19:16:46.393 # Failover auth granted to 5c61d77335632daf5762c946bccda3bb07b9fa60 for epoch 93 redis_10.200.100.187.log:22016:M 19:16:46.394 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:16:46.394 # Failover auth granted to 5c61d77335632daf5762c946bccda3bb07b9fa60 for epoch 93 redis_10.200.100.49.log:646447:M 19:16:46.396 * Caching the disconnected master state. redis_10.200.100.55.log:21922:M 19:16:46.396 # Cluster state changed: ok redis_10.200.100.55.log:21922:M 19:16:46.396 # configEpoch set to 93 after successful failover redis_10.200.100.55.log:21922:S 19:16:46.396 # Connection with master lost. redis_10.200.100.55.log:21922:M 19:16:46.396 * Discarding previously cached master state. redis_10.200.100.55.log:21922:M 19:16:46.396 # Failover election won: I'm the new master. redis_10.200.100.55.log:21922:S 19:16:46.399 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:16:46.399 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.194.log:10116:M 19:16:46.401 # Failover auth granted to 5c61d77335632daf5762c946bccda3bb07b9fa60 for epoch 93 redis_10.200.2.235.log:1200:M 19:16:46.428 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:16:46.432 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:16:46.433 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:16:46.433 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:16:46.434 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:16:46.434 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:16:46.435 # Cluster state changed: ok redis_10.200.100.85.log:21636:S 19:16:46.435 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:16:46.438 # Cluster state changed: ok redis_10.200.5.117.log:16009:S 19:16:46.441 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:16:46.449 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:16:46.469 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:16:46.469 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.5.228.log:1205:M 19:16:46.475 # Failover auth granted to 5c61d77335632daf5762c946bccda3bb07b9fa60 for epoch 93 redis_10.200.100.194.log:10116:M 19:16:46.483 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:16:46.531 # Failover auth granted to 5c61d77335632daf5762c946bccda3bb07b9fa60 for epoch 93 redis_10.200.5.228.log:1205:M 19:16:46.532 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:16:46.755 # Connection with slave 10.200.100.55:6379 lost. redis_10.200.100.58.log:22037:M 19:16:47.190 # Configuration change detected. Reconfiguring myself as a replica of 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.58.log:22037:M 19:16:47.197 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.101.179.log:2261:S 19:16:47.272 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.100.85.log:21636:S 19:16:47.291 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.100.88.log:21852:S 19:16:47.415 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.101.8.log:8114:S 19:16:47.429 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.3.31.log:15815:S 19:16:47.446 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.5.91.log:15823:S 19:16:47.611 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.100.99.log:34158:M 19:16:47.640 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.5.117.log:16009:S 19:16:47.706 # Failover auth denied to 5c61d77335632daf5762c946bccda3bb07b9fa60: its master is up redis_10.200.100.72.log:22084:M 19:16:48.085 * FAIL message received from 34efe005d6275cb0045156baa277d90c07ff0284 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.72.log:22084:M 19:16:48.104 * Connecting to MASTER 10.200.100.55:6379 redis_10.200.100.58.log:22037:S 19:16:48.104 * MASTER <-> SLAVE sync started redis_10.200.100.58.log:22037:S 19:16:48.104 * Non blocking connect for SYNC fired the event. redis_10.200.100.58.log:22037:S 19:16:48.443 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.100.55.log:21922:M 19:16:48.862 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.5.228.log:1205:M 19:16:49.192 * Master replied to PING, replication can continue... redis_10.200.100.58.log:22037:S 19:16:49.235 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.100.49.log:646447:M 19:16:49.360 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.194.log:10116:M 19:16:50.150 * Partial resynchronization not possible (no cached master) redis_10.200.100.58.log:22037:S 19:16:50.585 * Full resync requested by slave 10.200.100.58:6379 redis_10.200.100.55.log:21922:M 19:16:50.585 * Slave 10.200.100.58:6379 asks for synchronization redis_10.200.100.55.log:21922:M 19:16:50.585 * Starting BGSAVE for SYNC with target: disk redis_10.200.100.55.log:21922:M 19:16:50.630 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.72.log:22084:M 19:16:50.682 * Full resync from master: 6b53caab628782e9d4b820622ddf8f450222ccd9:2140265123058 redis_10.200.100.58.log:22037:S 19:16:50.683 * Background saving started by pid 728685 redis_10.200.100.55.log:21922:M 19:16:51.298 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: master without slots is reachable again. redis_10.200.2.171.log:1200:M 19:16:51.308 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.187.log:22016:M 19:16:51.986 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.2.235.log:1200:M 19:16:52.043 * MASTER <-> SLAVE sync: Finished with success redis_10.200.101.90.log:35522:S 19:16:52.044 # Cluster state changed: fail redis_10.200.101.90.log:35522:S 19:16:52.058 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.101.90.log:35522:S 19:16:53.544 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:16:53.647 * Clear FAIL state for node bb957a8b15470f47cb5b0008bbf7fc0474630213: is reachable again and nobody is serving its slots after some time. redis_10.200.101.90.log:35522:S 19:16:53.647 # Cluster state changed: ok redis_10.200.101.90.log:35522:S 19:16:53.656 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:16:54.845 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:16:58.673 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:16:59.044 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:16:59.868 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:17:07.015 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:17:07.435 # Cluster state changed: fail redis_10.200.100.55.log:21922:M 19:17:07.725 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:17:12.015 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:17:12.453 # Cluster state changed: ok redis_10.200.100.55.log:21922:M 19:17:12.738 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:17:37.423 * DB saved on disk redis_10.200.100.55.log:728685:C 19:17:37.552 * RDB: 1016 MB of memory used by copy-on-write redis_10.200.100.55.log:728685:C 19:17:37.922 * Background saving terminated with success redis_10.200.100.55.log:21922:M 19:17:38.178 * MASTER <-> SLAVE sync: receiving 1436471994 bytes from master redis_10.200.100.58.log:22037:S 19:20:19.268 * Synchronization with slave 10.200.100.58:6379 succeeded redis_10.200.100.55.log:21922:M 19:20:19.490 * MASTER <-> SLAVE sync: Flushing old data redis_10.200.100.58.log:22037:S 19:20:26.903 * Marking node f24444696fb319fb6c85dd9e8e01d9b182852070 as failing (quorum reached). redis_10.200.2.235.log:1200:M 19:20:26.907 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.5.228.log:1205:M 19:20:26.911 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.99.log:34158:M 19:20:26.913 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.85.log:21636:S 19:20:26.914 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.187.log:22016:M 19:20:26.914 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.49.log:646447:M 19:20:26.914 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.72.log:22084:M 19:20:26.914 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.88.log:21852:S 19:20:26.914 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.5.91.log:15823:S 19:20:26.915 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.194.log:10116:M 19:20:26.916 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.55.log:21922:M 19:20:26.916 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.2.171.log:1200:M 19:20:26.916 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.3.31.log:15815:S 19:20:26.917 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.101.90.log:35522:S 19:20:26.917 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.5.117.log:16009:S 19:20:26.918 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.101.8.log:8114:S 19:20:26.923 * FAIL message received from b764740a2e801079da85235718905561d5b02576 about f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.101.179.log:2261:S 19:20:40.035 # Client id=41064325 addr=10.200.100.58:15208 fd=578 name= age=232 idle=1 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=33789 omem=536873296 events=rw cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits. redis_10.200.100.55.log:21922:M 19:20:40.125 # Connection with slave 10.200.100.58:6379 lost. redis_10.200.100.55.log:21922:M 19:21:01.403 * MASTER <-> SLAVE sync: Loading DB in memory redis_10.200.100.58.log:22037:S 19:21:01.534 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.3.31.log:15815:S 19:21:01.535 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.5.91.log:15823:S 19:21:01.536 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.5.117.log:16009:S 19:21:01.539 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.85.log:21636:S 19:21:01.540 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.99.log:34158:M 19:21:01.541 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.187.log:22016:M 19:21:01.541 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.88.log:21852:S 19:21:01.542 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.72.log:22084:M 19:21:01.542 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.101.8.log:8114:S 19:21:01.544 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.49.log:646447:M 19:21:01.544 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.55.log:21922:M 19:21:01.544 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.2.171.log:1200:M 19:21:01.545 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.101.90.log:35522:S 19:21:01.549 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.101.179.log:2261:S 19:21:01.551 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.100.194.log:10116:M 19:21:01.597 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.2.235.log:1200:M 19:21:01.732 * Clear FAIL state for node f24444696fb319fb6c85dd9e8e01d9b182852070: slave is reachable again. redis_10.200.5.228.log:1205:M 19:22:20.799 * MASTER <-> SLAVE sync: Finished with success redis_10.200.100.58.log:22037:S 19:22:21.310 * Caching the disconnected master state. redis_10.200.100.58.log:22037:S 19:22:21.310 # Connection with master lost. redis_10.200.100.58.log:22037:S 19:22:22.310 * Connecting to MASTER 10.200.100.55:6379 redis_10.200.100.58.log:22037:S 19:22:22.310 * MASTER <-> SLAVE sync started redis_10.200.100.58.log:22037:S 19:22:22.311 * Non blocking connect for SYNC fired the event. redis_10.200.100.58.log:22037:S 19:22:22.320 * Master replied to PING, replication can continue... redis_10.200.100.58.log:22037:S 19:22:22.328 * Trying a partial resynchronization (request 6b53caab628782e9d4b820622ddf8f450222ccd9:2140266433779). redis_10.200.100.58.log:22037:S 19:22:22.330 * Slave 10.200.100.58:6379 asks for synchronization redis_10.200.100.55.log:21922:M 19:22:22.331 * Starting BGSAVE for SYNC with target: disk redis_10.200.100.55.log:21922:M 19:22:22.331 * Unable to partial resync with slave 10.200.100.58:6379 for lack of backlog (Slave request was: 2140266433779). redis_10.200.100.55.log:21922:M 19:22:22.459 * Full resync from master: 6b53caab628782e9d4b820622ddf8f450222ccd9:2140718907018 redis_10.200.100.58.log:22037:S 19:22:22.460 * Background saving started by pid 728799 redis_10.200.100.55.log:21922:M 19:22:22.460 * Discarding previously cached master state. redis_10.200.100.58.log:22037:S 19:22:54.702 # Cluster state changed: fail redis_10.200.5.228.log:1205:M 19:22:54.702 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.2.235.log:1200:M 19:22:54.702 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.5.228.log:1205:M 19:22:54.703 # Cluster state changed: fail redis_10.200.2.235.log:1200:M 19:22:54.711 * Marking node 5c61d77335632daf5762c946bccda3bb07b9fa60 as failing (quorum reached). redis_10.200.100.187.log:22016:M 19:22:54.712 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.99.log:34158:M 19:22:54.713 # Cluster state changed: fail redis_10.200.100.187.log:22016:M 19:22:54.713 # Cluster state changed: fail redis_10.200.100.85.log:21636:S 19:22:54.713 # Cluster state changed: fail redis_10.200.100.99.log:34158:M 19:22:54.713 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.85.log:21636:S 19:22:54.714 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.49.log:646447:M 19:22:54.714 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.72.log:22084:M 19:22:54.715 # Cluster state changed: fail redis_10.200.100.49.log:646447:M 19:22:54.715 # Cluster state changed: fail redis_10.200.100.58.log:22037:S 19:22:54.715 # Cluster state changed: fail redis_10.200.100.72.log:22084:M 19:22:54.715 # Cluster state changed: fail redis_10.200.100.88.log:21852:S 19:22:54.715 # Cluster state changed: fail redis_10.200.2.171.log:1200:M 19:22:54.715 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.58.log:22037:S 19:22:54.715 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.88.log:21852:S 19:22:54.715 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.2.171.log:1200:M 19:22:54.716 # Cluster state changed: fail redis_10.200.101.179.log:2261:S 19:22:54.716 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.194.log:10116:M 19:22:54.716 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.101.179.log:2261:S 19:22:54.716 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.101.8.log:8114:S 19:22:54.717 # Cluster state changed: fail redis_10.200.101.8.log:8114:S 19:22:54.717 # Cluster state changed: fail redis_10.200.3.31.log:15815:S 19:22:54.717 # Cluster state changed: fail redis_10.200.5.91.log:15823:S 19:22:54.717 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.3.31.log:15815:S 19:22:54.717 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.5.91.log:15823:S 19:22:54.718 # Cluster state changed: fail redis_10.200.100.194.log:10116:M 19:22:54.718 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.101.90.log:35522:S 19:22:54.718 * FAIL message received from bb957a8b15470f47cb5b0008bbf7fc0474630213 about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.5.117.log:16009:S 19:22:54.719 # Cluster state changed: fail redis_10.200.101.90.log:35522:S 19:22:54.719 # Cluster state changed: fail redis_10.200.5.117.log:16009:S 19:22:54.805 # Start of election delayed for 686 milliseconds (rank #0, offset 0). redis_10.200.100.58.log:22037:S 19:22:55.495 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.2.235.log:1200:M 19:22:55.496 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.5.228.log:1205:M 19:22:55.505 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.100.99.log:34158:M 19:22:55.506 # Starting a failover election for epoch 94. redis_10.200.100.58.log:22037:S 19:22:55.508 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.100.187.log:22016:M 19:22:55.508 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.100.49.log:646447:M 19:22:55.508 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.100.72.log:22084:M 19:22:55.509 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.100.194.log:10116:M 19:22:55.509 # Failover election won: I'm the new master. redis_10.200.100.58.log:22037:S 19:22:55.510 # Cluster state changed: ok redis_10.200.100.58.log:22037:M 19:22:55.510 # configEpoch set to 94 after successful failover redis_10.200.100.58.log:22037:S 19:22:55.510 # Failover auth granted to f24444696fb319fb6c85dd9e8e01d9b182852070 for epoch 94 redis_10.200.2.171.log:1200:M 19:22:55.537 # Cluster state changed: ok redis_10.200.5.228.log:1205:M 19:22:55.539 # Cluster state changed: ok redis_10.200.2.235.log:1200:M 19:22:55.545 # Cluster state changed: ok redis_10.200.100.99.log:34158:M 19:22:55.546 # Cluster state changed: ok redis_10.200.100.85.log:21636:S 19:22:55.547 # Cluster state changed: ok redis_10.200.100.187.log:22016:M 19:22:55.547 # Cluster state changed: ok redis_10.200.100.72.log:22084:M 19:22:55.547 # Cluster state changed: ok redis_10.200.100.88.log:21852:S 19:22:55.548 # Cluster state changed: ok redis_10.200.101.179.log:2261:S 19:22:55.549 # Cluster state changed: ok redis_10.200.100.194.log:10116:M 19:22:55.549 # Cluster state changed: ok redis_10.200.100.49.log:646447:M 19:22:55.549 # Cluster state changed: ok redis_10.200.101.8.log:8114:S 19:22:55.549 # Cluster state changed: ok redis_10.200.3.31.log:15815:S 19:22:55.551 # Cluster state changed: ok redis_10.200.101.90.log:35522:S 19:22:55.551 # Cluster state changed: ok redis_10.200.5.117.log:16009:S 19:22:55.551 # Cluster state changed: ok redis_10.200.5.91.log:15823:S 19:22:55.552 # Cluster state changed: ok redis_10.200.2.171.log:1200:M 19:22:56.420 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.3.31.log:15815:S 19:22:56.420 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.5.91.log:15823:S 19:22:56.421 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.101.90.log:35522:S 19:22:56.421 # Failover auth denied to f24444696fb319fb6c85dd9e8e01d9b182852070: its master is up redis_10.200.100.55.log:21922:M 19:22:56.451 # Configuration change detected. Reconfiguring myself as a replica of f24444696fb319fb6c85dd9e8e01d9b182852070 redis_10.200.100.55.log:21922:M 19:22:56.451 # Connection with slave 10.200.100.58:6379 lost. redis_10.200.100.55.log:21922:S 19:22:56.458 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.100.58.log:22037:M 19:22:56.474 * Connecting to MASTER 10.200.100.58:6379 redis_10.200.100.55.log:21922:S 19:22:56.475 * MASTER <-> SLAVE sync started redis_10.200.100.55.log:21922:S 19:22:56.514 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.2.171.log:1200:M 19:22:56.522 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.2.235.log:1200:M 19:23:03.752 * Non blocking connect for SYNC fired the event. redis_10.200.100.55.log:21922:S 19:23:03.763 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.85.log:21636:S 19:23:03.767 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.5.117.log:16009:S 19:23:03.767 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.101.8.log:8114:S 19:23:03.772 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.101.179.log:2261:S 19:23:03.775 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.88.log:21852:S 19:23:03.787 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.5.228.log:1205:M 19:23:03.821 * Master replied to PING, replication can continue... redis_10.200.100.55.log:21922:S 19:23:03.837 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: master without slots is reachable again. redis_10.200.100.99.log:34158:M 19:23:03.838 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.49.log:646447:M 19:23:03.850 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.187.log:22016:M 19:23:03.852 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.194.log:10116:M 19:23:03.853 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.72.log:22084:M 19:23:03.958 * Partial resynchronization not possible (no cached master) redis_10.200.100.55.log:21922:S 19:23:03.965 * Full resync requested by slave 10.200.100.55:6379 redis_10.200.100.58.log:22037:M 19:23:03.965 * Slave 10.200.100.55:6379 asks for synchronization redis_10.200.100.58.log:22037:M 19:23:03.965 * Starting BGSAVE for SYNC with target: disk redis_10.200.100.58.log:22037:M 19:23:04.059 * Background saving started by pid 730116 redis_10.200.100.58.log:22037:M 19:23:04.061 * Full resync from master: a150fe527ecb5163e0c3537702e663a00f290f30:1 redis_10.200.100.55.log:21922:S 19:23:06.761 * DB saved on disk redis_10.200.100.55.log:728799:C 19:23:06.847 * RDB: 406 MB of memory used by copy-on-write redis_10.200.100.55.log:728799:C 19:23:07.067 * Background saving terminated with success redis_10.200.100.55.log:21922:S 19:23:46.156 * DB saved on disk redis_10.200.100.58.log:730116:C 19:23:46.228 * RDB: 433 MB of memory used by copy-on-write redis_10.200.100.58.log:730116:C 19:23:46.361 * Background saving terminated with success redis_10.200.100.58.log:22037:M 19:23:46.363 * MASTER <-> SLAVE sync: receiving 1428994543 bytes from master redis_10.200.100.55.log:21922:S 19:24:03.409 * Synchronization with slave 10.200.100.55:6379 succeeded redis_10.200.100.58.log:22037:M 19:24:03.672 * MASTER <-> SLAVE sync: Flushing old data redis_10.200.100.55.log:21922:S 19:24:10.187 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.5.228.log:1205:M 19:24:10.190 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.2.235.log:1200:M 19:24:10.200 * Marking node 5c61d77335632daf5762c946bccda3bb07b9fa60 as failing (quorum reached). redis_10.200.100.72.log:22084:M 19:24:10.201 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.85.log:21636:S 19:24:10.201 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.99.log:34158:M 19:24:10.203 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.88.log:21852:S 19:24:10.204 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.187.log:22016:M 19:24:10.204 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.49.log:646447:M 19:24:10.204 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.58.log:22037:M 19:24:10.204 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.101.179.log:2261:S 19:24:10.204 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.2.171.log:1200:M 19:24:10.205 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.100.194.log:10116:M 19:24:10.205 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.101.8.log:8114:S 19:24:10.206 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.101.90.log:35522:S 19:24:10.209 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.5.91.log:15823:S 19:24:10.211 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.3.31.log:15815:S 19:24:10.211 * FAIL message received from 118f4c0c5573e6b9a3fa889afc92476852820c6c about 5c61d77335632daf5762c946bccda3bb07b9fa60 redis_10.200.5.117.log:16009:S 19:24:44.645 * MASTER <-> SLAVE sync: Loading DB in memory redis_10.200.100.55.log:21922:S 19:24:44.762 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.5.228.log:1205:M 19:24:44.763 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.2.235.log:1200:M 19:24:44.776 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.85.log:21636:S 19:24:44.776 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.99.log:34158:M 19:24:44.778 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.187.log:22016:M 19:24:44.778 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.72.log:22084:M 19:24:44.778 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.88.log:21852:S 19:24:44.778 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.101.179.log:2261:S 19:24:44.779 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.49.log:646447:M 19:24:44.779 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.58.log:22037:M 19:24:44.779 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.101.8.log:8114:S 19:24:44.779 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.2.171.log:1200:M 19:24:44.780 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.100.194.log:10116:M 19:24:44.781 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.101.90.log:35522:S 19:24:44.787 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.5.91.log:15823:S 19:24:44.788 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.3.31.log:15815:S 19:24:44.789 * Clear FAIL state for node 5c61d77335632daf5762c946bccda3bb07b9fa60: slave is reachable again. redis_10.200.5.117.log:16009:S 19:26:14.296 * MASTER <-> SLAVE sync: Finished with success redis_10.200.100.55.log:21922:S