--------------- MONGOS LOG START -----------------
Fri Mar 23 09:24:32 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:25:02 [Balancer] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:25:02 [Balancer] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:25:02 [Balancer] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:25:02 [Balancer] ~ScopedDbConnection: _conn != null
Fri Mar 23 09:25:02 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:25:15 [WriteBackListener-nodeA:27123] DBClientCursor::init call() failed
Fri Mar 23 09:25:15 [WriteBackListener-nodeA:27123] WriteBackListener exception : DBClientBase::findN: transport error: nodeA:27123 query: { writebacklisten: ObjectId('4f6b5d0f7c830556fb779771') }
Fri Mar 23 09:25:15 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed
Fri Mar 23 09:25:15 [ReplicaSetMonitorWatcher] trying reconnect to nodeA:27123
Fri Mar 23 09:25:15 [ReplicaSetMonitorWatcher] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:16 [WriteBackListener-nodeA:27123] WriteBackListener exception : socket exception
Fri Mar 23 09:25:16 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception nodeA:27123 socket exception
Fri Mar 23 09:25:16 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:25:16 [conn54768] ReplicaSetMonitor::_checkConnection: caught exception nodeA:27123 socket exception
Fri Mar 23 09:25:16 [conn54768] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:25:16 [conn54769] ReplicaSetMonitor::_checkConnection: caught exception nodeA:27123 socket exception
Fri Mar 23 09:25:16 [conn54769] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:25:17 [conn54768] trying reconnect to nodeA:27123
Fri Mar 23 09:25:22 [conn54768] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:22 [conn54768] trying reconnect to nodeA:27123
Fri Mar 23 09:25:27 [conn54768] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:27 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:25:32 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:32 [conn54770] trying reconnect to nodeA:27123
Fri Mar 23 09:25:37 [conn54770] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:37 [conn54769] trying reconnect to nodeA:27123
Fri Mar 23 09:25:42 [conn54769] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:42 [conn54769] trying reconnect to nodeA:27123
Fri Mar 23 09:25:47 [conn54769] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:47 [Balancer] trying reconnect to nodeA:27123
Fri Mar 23 09:25:52 [Balancer] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:52 [Balancer] trying reconnect to nodeA:27123
Fri Mar 23 09:25:57 [Balancer] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:25:57 [conn54771] trying reconnect to nodeA:27123
Fri Mar 23 09:25:58 [Balancer] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:25:58 [Balancer] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:25:58 [Balancer] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:25:58 [Balancer] ~ScopedDbConnection: _conn != null
Fri Mar 23 09:25:58 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:26:02 [conn54771] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:02 [conn54771] trying reconnect to nodeA:27123
Fri Mar 23 09:26:07 [conn54771] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:08 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:26:13 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:13 [conn54768] trying reconnect to nodeA:27123
Fri Mar 23 09:26:18 [conn54768] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:18 [conn54769] trying reconnect to nodeA:27123
Fri Mar 23 09:26:23 [conn54769] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:23 [conn54770] trying reconnect to nodeA:27123
Fri Mar 23 09:26:28 [conn54770] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:28 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:26:33 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:33 [conn54774] trying reconnect to nodeA:27123
Fri Mar 23 09:26:38 [conn54774] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:38 [conn54774] trying reconnect to nodeA:27123
Fri Mar 23 09:26:43 [conn54774] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:43 [conn54771] trying reconnect to nodeA:27123
Fri Mar 23 09:26:48 [conn54771] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:48 [conn54771] trying reconnect to nodeA:27123
Fri Mar 23 09:26:53 [conn54771] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:53 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:26:58 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:26:58 [conn54769] trying reconnect to nodeA:27123
Fri Mar 23 09:27:03 [conn54769] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:03 [ReplicaSetMonitorWatcher] trying reconnect to nodeA:27123
Fri Mar 23 09:27:08 [ReplicaSetMonitorWatcher] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:08 [conn54775] trying reconnect to nodeA:27123
Fri Mar 23 09:27:13 [conn54775] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:13 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:27:18 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:18 [conn54769] trying reconnect to nodeA:27123
Fri Mar 23 09:27:23 [conn54769] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:23 [conn54769] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:27:23 [Balancer] trying reconnect to nodeA:27123
Fri Mar 23 09:27:23 [conn54769] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:27:23 [conn54769] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:27:28 [Balancer] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:28 [conn54773] trying reconnect to nodeA:27123
Fri Mar 23 09:27:28 [Balancer] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:27:28 [Balancer] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:27:28 [Balancer] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:27:28 [Balancer] ~ScopedDbConnection: _conn != null
Fri Mar 23 09:27:28 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:27:33 [conn54773] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:33 [conn54773] trying reconnect to nodeA:27123
Fri Mar 23 09:27:38 [conn54773] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:38 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:27:43 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:43 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:27:48 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:48 [conn54771] trying reconnect to nodeA:27123
Fri Mar 23 09:27:53 [conn54771] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:53 [conn54775] trying reconnect to nodeA:27123
Fri Mar 23 09:27:58 [conn54775] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:27:58 [conn54775] trying reconnect to nodeA:27123
Fri Mar 23 09:28:03 [conn54775] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:03 [conn54770] trying reconnect to nodeA:27123
Fri Mar 23 09:28:08 [conn54770] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:08 [conn54770] trying reconnect to nodeA:27123
Fri Mar 23 09:28:13 [conn54770] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:13 [ReplicaSetMonitorWatcher] trying reconnect to nodeA:27123
Fri Mar 23 09:28:18 [ReplicaSetMonitorWatcher] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:18 [conn54769] trying reconnect to nodeA:27123
Fri Mar 23 09:28:23 [conn54769] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:23 [conn54769] trying reconnect to nodeA:27123
Fri Mar 23 09:28:27 [WriteBackListener-nodeA:27123] WriteBackListener exception : socket exception
Fri Mar 23 09:28:27 [WriteBackListener-nodeA:27123] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:28:27 [WriteBackListener-nodeA:27123] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:28:27 [WriteBackListener-nodeA:27123] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:28:28 [conn54769] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:28 [conn54771] trying reconnect to nodeA:27123
Fri Mar 23 09:28:33 [conn54771] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:33 [conn54771] trying reconnect to nodeA:27123
Fri Mar 23 09:28:38 [conn54771] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:38 [conn54773] trying reconnect to nodeA:27123
Fri Mar 23 09:28:43 [conn54773] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:43 [conn54772] trying reconnect to nodeA:27123
Fri Mar 23 09:28:48 [conn54772] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:48 [ReplicaSetMonitorWatcher] trying reconnect to nodeA:27123
Fri Mar 23 09:28:53 [ReplicaSetMonitorWatcher] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:53 [Balancer] trying reconnect to nodeA:27123
Fri Mar 23 09:28:58 [Balancer] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:28:58 [conn54776] trying reconnect to nodeA:27123
Fri Mar 23 09:28:58 [Balancer] ~ScopedDbConnection: _conn != null
Fri Mar 23 09:28:58 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:29:03 [conn54776] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:03 [conn54776] trying reconnect to nodeA:27123
Fri Mar 23 09:29:08 [conn54776] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:08 [conn54778] trying reconnect to nodeA:27123
Fri Mar 23 09:29:13 [conn54778] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:13 [conn54779] trying reconnect to nodeA:27123
Fri Mar 23 09:29:18 [conn54779] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:18 [conn54777] trying reconnect to nodeA:27123
Fri Mar 23 09:29:23 [conn54777] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:23 [conn54777] trying reconnect to nodeA:27123
Fri Mar 23 09:29:28 [conn54777] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:28 [conn54781] trying reconnect to nodeA:27123
Fri Mar 23 09:29:33 [conn54781] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:33 [conn54781] trying reconnect to nodeA:27123
Fri Mar 23 09:29:38 [conn54781] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:38 [conn54778] trying reconnect to nodeA:27123
Fri Mar 23 09:29:43 [conn54778] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:43 [conn54778] trying reconnect to nodeA:27123
Fri Mar 23 09:29:48 [conn54778] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:48 [conn54780] trying reconnect to nodeA:27123
Fri Mar 23 09:29:53 [conn54780] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:53 [conn54777] trying reconnect to nodeA:27123
Fri Mar 23 09:29:58 [conn54777] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:29:58 [conn54770] trying reconnect to nodeA:27123
Fri Mar 23 09:30:03 [conn54770] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:03 [conn54782] trying reconnect to nodeA:27123
Fri Mar 23 09:30:08 [conn54782] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:08 [conn54782] trying reconnect to nodeA:27123
Fri Mar 23 09:30:13 [conn54782] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:13 [conn54779] trying reconnect to nodeA:27123
Fri Mar 23 09:30:18 [conn54779] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:18 [conn54779] trying reconnect to nodeA:27123
Fri Mar 23 09:30:23 [conn54779] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:23 [conn54780] trying reconnect to nodeA:27123
Fri Mar 23 09:30:28 [conn54780] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:28 [conn54780] trying reconnect to nodeA:27123
Fri Mar 23 09:30:33 [conn54780] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:33 [conn54770] trying reconnect to nodeA:27123
Fri Mar 23 09:30:38 [conn54770] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:38 [conn54770] trying reconnect to nodeA:27123
Fri Mar 23 09:30:43 [conn54770] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:43 [conn54774] trying reconnect to nodeA:27123
Fri Mar 23 09:30:48 [conn54774] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:48 [conn54773] trying reconnect to nodeA:27123
Fri Mar 23 09:30:53 [conn54773] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:53 [ReplicaSetMonitorWatcher] trying reconnect to nodeA:27123
Fri Mar 23 09:30:58 [ReplicaSetMonitorWatcher] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:30:58 [conn54780] trying reconnect to nodeA:27123
Fri Mar 23 09:31:03 [conn54780] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:03 [conn54781] trying reconnect to nodeA:27123
Fri Mar 23 09:31:08 [conn54781] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:08 [conn54783] trying reconnect to nodeA:27123
Fri Mar 23 09:31:13 [conn54783] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:13 [conn54783] trying reconnect to nodeA:27123
Fri Mar 23 09:31:18 [conn54783] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:18 [conn54778] trying reconnect to nodeA:27123
Fri Mar 23 09:31:23 [conn54778] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:23 [Balancer] trying reconnect to nodeA:27123
Fri Mar 23 09:31:28 [Balancer] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:28 [conn54776] trying reconnect to nodeA:27123
Fri Mar 23 09:31:28 [Balancer] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:31:28 [Balancer] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:31:28 [Balancer] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:31:28 [Balancer] ~ScopedDbConnection: _conn != null
Fri Mar 23 09:31:28 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:31:33 [conn54776] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:33 [conn54780] trying reconnect to nodeA:27123
Fri Mar 23 09:31:38 [conn54780] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:38 [conn54780] trying reconnect to nodeA:27123
Fri Mar 23 09:31:39 [WriteBackListener-nodeA:27123] WriteBackListener exception : socket exception
Fri Mar 23 09:31:43 [conn54780] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:43 [conn54782] trying reconnect to nodeA:27123
Fri Mar 23 09:31:48 [conn54782] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:48 [conn54786] trying reconnect to nodeA:27123
Fri Mar 23 09:31:53 [conn54786] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:53 [conn54786] trying reconnect to nodeA:27123
Fri Mar 23 09:31:58 [conn54786] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:31:58 [conn54778] trying reconnect to nodeA:27123
Fri Mar 23 09:32:03 [conn54778] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:03 [conn54778] trying reconnect to nodeA:27123
Fri Mar 23 09:32:08 [conn54778] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:08 [conn54784] trying reconnect to nodeA:27123
Fri Mar 23 09:32:13 [conn54784] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:13 [ReplicaSetMonitorWatcher] trying reconnect to nodeA:27123
Fri Mar 23 09:32:18 [ReplicaSetMonitorWatcher] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:18 [conn54788] trying reconnect to nodeA:27123
Fri Mar 23 09:32:23 [conn54788] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:23 [conn54788] trying reconnect to nodeA:27123
Fri Mar 23 09:32:28 [conn54788] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:28 [conn54786] trying reconnect to nodeA:27123
Fri Mar 23 09:32:33 [conn54786] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:33 [conn54777] trying reconnect to nodeA:27123
Fri Mar 23 09:32:38 [conn54777] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:38 [conn54776] trying reconnect to nodeA:27123
Fri Mar 23 09:32:43 [conn54776] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:43 [conn54781] trying reconnect to nodeA:27123
Fri Mar 23 09:32:48 [conn54781] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:48 [conn54787] trying reconnect to nodeA:27123
Fri Mar 23 09:32:53 [conn54787] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:53 [conn54782] trying reconnect to nodeA:27123
Fri Mar 23 09:32:58 [conn54782] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:32:58 [conn54785] trying reconnect to nodeA:27123
Fri Mar 23 09:33:01 [conn54785] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:33:01 [conn54785] trying reconnect to nodeA:27123
Fri Mar 23 09:33:01 [conn54785] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:33:01 [Balancer] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:33:01 [Balancer] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:33:01 [Balancer] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:33:01 [Balancer] ~ScopedDbConnection: _conn != null
Fri Mar 23 09:33:01 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:33:02 [conn54776] DBException in process: socket exception
Fri Mar 23 09:33:02 [conn54776] SocketException handling request, closing client connection: 9001 socket exception [2] server [
127.0.0.1:53296]
Fri Mar 23 09:33:02 [conn54773] DBException in process: socket exception
Fri Mar 23 09:33:02 [conn54773] SocketException handling request, closing client connection: 9001 socket exception [2] server [
127.0.0.1:53199]
Fri Mar 23 09:33:02 [conn54774] DBException in process: socket exception
Fri Mar 23 09:33:02 [conn54774] SocketException handling request, closing client connection: 9001 socket exception [2] server [
127.0.0.1:53234]
Fri Mar 23 09:33:03 [conn54792] trying reconnect to nodeA:27123
Fri Mar 23 09:33:03 [conn54792] reconnect nodeA:27123 failed couldn't connect to server nodeA:27123
Fri Mar 23 09:33:05 [conn54795] trying reconnect to nodeA:27123
Fri Mar 23 09:33:05 [conn54795] reconnect nodeA:27123 ok
Fri Mar 23 09:33:12 [WriteBackListener-nodeB:27123] DBClientCursor::init call() failed
Fri Mar 23 09:33:12 [WriteBackListener-nodeB:27123] WriteBackListener exception : DBClientBase::findN: transport error: nodeB:27123 query: { writebacklisten: ObjectId('4f6b5d0f7c830556fb779771') }
Fri Mar 23 09:33:12 [conn54807] Socket recv() errno:104 Connection reset by peer
11.22.33.44:27123Fri Mar 23 09:33:12 [conn54807] DBClientCursor::init call() failed
Fri Mar 23 09:33:12 [conn54807] trying reconnect to nodeB:27123
Fri Mar 23 09:33:12 [conn54807] reconnect nodeB:27123 ok
Fri Mar 23 09:33:13 [conn54807] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], primary: "nodeB:27123", me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:13 [conn54807] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:13 [conn54808] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], primary: "nodeB:27123", me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:13 [conn54808] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:13 [conn54809] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], primary: "nodeB:27123", me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:13 [conn54809] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:14 [conn54810] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], primary: "nodeB:27123", me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:14 [conn54810] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54811] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54811] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54807] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54807] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54808] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54808] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54809] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:15 [conn54809] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:16 [WriteBackListener-nodeA:27123] WriteBackListener exception : socket exception
Fri Mar 23 09:33:16 [conn54810] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:16 [conn54810] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54811] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54811] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54807] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54807] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54808] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54808] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54809] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:17 [conn54809] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:18 [conn54810] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:18 [conn54810] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:18 [conn54807] Socket recv() errno:104 Connection reset by peer
11.22.33.44:27123Fri Mar 23 09:33:18 [conn54807] DBClientCursor::init call() failed
Fri Mar 23 09:33:18 [conn54808] Socket recv() errno:104 Connection reset by peer
11.22.33.44:27123Fri Mar 23 09:33:18 [conn54808] DBClientCursor::init call() failed
Fri Mar 23 09:33:18 [conn54809] Socket recv() errno:104 Connection reset by peer
11.22.33.44:27123Fri Mar 23 09:33:18 [conn54809] DBClientCursor::init call() failed
Fri Mar 23 09:33:19 [conn54810] Socket recv() errno:104 Connection reset by peer
11.22.33.44:27123Fri Mar 23 09:33:19 [conn54810] DBClientCursor::init call() failed
Fri Mar 23 09:33:19 [conn54811] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:19 [conn54811] ReplicaSetMonitor::_checkConnection: nodeB:27123 { setName: "repSet1", ismaster: false, secondary: true, hosts: [ "nodeB:27123", "nodeA:27123" ], arbiters: [ "arbiterAB:27321" ], me: "nodeB:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:20 [conn54811] Socket recv() errno:104 Connection reset by peer
11.22.33.44:27123Fri Mar 23 09:33:20 [conn54811] DBClientCursor::init call() failed
Fri Mar 23 09:33:21 [conn54812] ReplicaSetMonitor::_checkConnection: nodeA:27123 { setName: "repSet1", ismaster: true, secondary: false, hosts: [ "nodeA:27123", "nodeB:27123" ], arbiters: [ "arbiterAB:27321" ], primary: "nodeA:27123", me: "nodeA:27123", maxBsonObjectSize: 16777216, ok: 1.0 }
Fri Mar 23 09:33:31 [Balancer] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:33:31 [Balancer] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:33:31 [Balancer] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:33:31 [Balancer] ~ScopedDbConnection: _conn != null
Fri Mar 23 09:33:31 [Balancer] caught exception while doing balance: error checking clock skew of cluster config1:27123,config2:27123,config3:27123 :: caused by :: 13650 clock skew of the cluster config1:27123,config2:27123,config3:27123 is too far out of bounds to allow distributed locking.
Fri Mar 23 09:34:01 [Balancer] SyncClusterConnection connecting to [config1:27123]
Fri Mar 23 09:34:01 [Balancer] SyncClusterConnection connecting to [config2:27123]
Fri Mar 23 09:34:01 [Balancer] SyncClusterConnection connecting to [config3:27123]
Fri Mar 23 09:34:01 [Balancer] ~ScopedDbConnection: _conn != null
--------------- MONGOS LOG END -----------------
Thanks
V
Пʼятниця, 23 березня 2012 р., 17:31:06 UTC+2 користувач Kristina Chodorow написав: