Strange behavior for Failover strategy

[Emanuele Bragagnolo]

Hello

I am simulating the drivers behavior when a down occurs in mongodb. 

I've notice that the Failover is not working as should be: using the following configuration

MongoConnectionOptions(failoverStrategy = FailoverStrategy(retries = 10, delayFactor = _ => 1))

I expect to get an error after about 1 second (10 retries * 100ms ), but the tests I did actually show that I get an exception after about 15 seconds.

The testcase in attachment execute these steps:

1. Create a MongoConnection
2. Perform a simple operation against mongo just to check everything is fine
3. Shutdown mongodb server with a "sudo killall mongod" !; ( change it depending on your OS );
4. Try to insert a document and wait for exception

The weird thing is that changing only the number of retries in the above configuration for FailoverStrategy shows that the error timing doesn't change in a linear way, but it seems to follow a quadratic growth

Checking the source code confirm this impression: there are two nested usage of the class Failover2 for handling the errors

1. One is in genericcollection row 362 (the external one)
2. One is in commands row 158 (the inner one)

To execute the test run sbt run. You have to run mongod on your localhost

Can you confirm this bug?

Best Regards, Emanuele

[Cédric Chantepie]

The unit and integration are not confirming that. On this other Akka doesn't provide an absolute guarantee on the scheduler timing under load issue/cpu starving.

You can increase logging for reactivemongo.api.Failover to watch the retries.

[Emanuele Bragagnolo]

Hi Cedric

I've checked the BSONCollectionSpec and the scenario I proposed is not covered in the test. 

Please download the attachment: inside there's already a log files that confirm a total of 120 retries instead of 10.

I've counted the rows with 

Got an error, retrying... (try #x is scheduled in 100 ms)

We didn't have this bug with the old version of the driver (0.10.5.0.akka23), but with 0,12.1 the behavior is changed

 

Emanuele

[Cédric Chantepie]

The retries and failover are coherent for me according the log :

2017-03-07 17:02:23,032 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:23,145 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:23,266 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:23,386 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:23,506 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:23,627 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:23,746 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:23,865 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:23,985 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:24,105 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:24,227 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:24,346 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:24,466 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:24,586 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:24,705 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:24,825 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:24,945 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:25,065 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:25,185 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:25,305 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:25,425 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:25,546 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:25,666 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:25,785 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:25,906 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:26,025 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:26,145 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:26,266 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:26,385 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:26,505 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:26,625 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:26,745 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:26,866 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:26,986 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:27,104 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:27,225 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:27,345 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:27,465 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:27,586 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:27,705 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:27,825 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:27,944 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:28,064 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:28,185 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:28,305 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:28,424 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:28,545 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:28,664 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:28,785 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:28,905 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:29,025 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:29,144 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:29,264 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:29,384 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:29,505 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:29,625 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:29,745 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:29,865 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:29,985 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:30,105 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:30,224 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:30,345 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:30,465 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:30,585 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:30,705 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:30,825 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:30,946 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:31,064 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:31,185 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:31,304 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:31,424 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:31,545 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:31,665 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:31,785 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:31,905 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:32,025 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:32,145 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:32,265 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:32,384 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:32,505 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:32,625 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:32,745 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:32,865 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:32,985 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:33,104 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:33,224 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:33,344 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:33,465 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:33,585 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:33,704 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:33,825 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:33,945 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:34,065 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:34,184 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:34,304 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:34,424 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:34,544 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:34,665 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:34,785 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:34,905 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:35,025 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:35,145 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:35,264 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:35,384 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:35,505 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:35,625 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:35,745 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:35,864 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:35,984 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:36,105 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-07 17:02:36,225 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-07 17:02:36,345 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-07 17:02:36,464 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-07 17:02:36,585 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-07 17:02:36,705 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-07 17:02:36,825 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-07 17:02:36,945 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-07 17:02:37,065 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-07 17:02:37,184 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-07 17:02:37,304 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:48,100 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:48,222 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:48,342 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:48,462 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:48,582 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:48,702 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:48,823 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:48,942 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:49,063 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:49,181 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:49,306 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:49,422 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:49,540 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:49,661 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:49,782 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:49,902 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:50,021 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:50,141 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:50,262 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:50,382 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:50,501 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:50,623 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:50,743 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:50,862 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:50,981 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:51,102 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:51,221 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:51,341 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:51,462 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:51,582 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:51,702 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:51,822 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:51,943 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:52,061 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:52,182 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:52,301 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:52,422 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:52,542 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:52,661 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:52,781 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:52,902 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:53,020 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:53,142 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:53,263 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:53,382 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:53,502 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:53,621 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:53,742 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:53,862 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:53,982 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:54,100 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:54,222 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:54,341 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:54,462 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:54,583 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:54,702 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:54,821 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:54,942 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:55,059 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:55,181 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:55,302 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:55,419 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:55,541 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:55,662 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:55,781 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:55,902 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:56,020 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:56,140 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:56,260 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:56,380 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:56,500 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:56,622 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:56,741 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:56,861 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:56,981 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:57,102 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:57,222 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:57,342 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:57,459 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:57,581 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:57,699 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:57,820 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:57,939 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:58,061 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:58,179 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:58,299 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:58,420 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:58,541 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:58,661 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:02:58,781 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:02:58,902 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:02:59,022 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:02:59,141 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:02:59,261 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:02:59,382 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:02:59,500 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:02:59,621 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:59,741 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:02:59,862 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:02:59,982 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:03:00,099 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:03:00,221 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:03:00,342 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:03:00,460 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:03:00,580 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:03:00,701 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:03:00,821 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:03:00,941 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:03:01,061 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:03:01,181 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:03:01,301 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:03:01,422 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:03:01,541 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:03:01,661 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:03:01,780 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:03:01,901 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:03:02,020 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:03:02,142 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:03:02,259 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:03:02,381 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...
2017-03-25 00:11:17,491 ... Got an error, retrying ... (try #1 is scheduled in 100 ms) ...
2017-03-25 00:11:17,608 ... Got an error, retrying ... (try #2 is scheduled in 100 ms) ...
2017-03-25 00:11:17,727 ... Got an error, retrying ... (try #3 is scheduled in 100 ms) ...
2017-03-25 00:11:17,848 ... Got an error, retrying ... (try #4 is scheduled in 100 ms) ...
2017-03-25 00:11:17,967 ... Got an error, retrying ... (try #5 is scheduled in 100 ms) ...
2017-03-25 00:11:18,086 ... Got an error, retrying ... (try #6 is scheduled in 100 ms) ...
2017-03-25 00:11:18,208 ... Got an error, retrying ... (try #7 is scheduled in 100 ms) ...
2017-03-25 00:11:18,328 ... Got an error, retrying ... (try #8 is scheduled in 100 ms) ...
2017-03-25 00:11:18,448 ... Got an error, retrying ... (try #9 is scheduled in 100 ms) ...
2017-03-25 00:11:18,567 ... Got an error, retrying ... (try #10 is scheduled in 100 ms) ...

As for the nesting Failover, that updated in snapshot.