consul instance 1 logs(x.x.x.x):
May 15 16:13:10 repo1 consul[20266]: 2018/05/15 16:13:10 [WARN] memberlist: Was able to connect to repo2 but other probes failed, network may be misconfigured
May 15 16:13:11 repo1 consul[20266]: 2018/05/15 16:13:11 [WARN] memberlist: Was able to connect to repo2.imsxxxx but other probes failed, network may be misconfigured
May 15 16:13:15 repo1 consul[20266]: 2018/05/15 16:13:15 [DEBUG] raft-net: x.x.x.x:5825 accepted connection from: y.y.y.y:25656
May 15 16:13:15 repo1 consul[20266]: 2018/05/15 16:13:15 [INFO] raft: Node at x.x.x.x:5825 [Follower] entering Follower state (Leader: "")
May 15 16:13:15 repo1 consul[20266]: 2018/05/15 16:13:15 [INFO] consul: New leader elected: repo2
May 15 16:13:16 repo1 consul[20266]: 2018/05/15 16:13:16 [DEBUG] raft-net: x.x.x.x:5825 accepted connection from: y.y.y.y:29296
May 15 16:13:16 repo1 consul[20266]: 2018/05/15 16:13:16 [INFO] serf: attempting reconnect to vconsul_float.imsxxxx z.z.z.z:5824
May 15 16:13:16 repo1 consul[20266]: 2018/05/15 16:13:16 [INFO] serf: attempting reconnect to vconsul_float z.z.z.z:5823
May 15 16:13:16 repo1 consul[20266]: 2018/05/15 16:13:16 [INFO] agent: Synced node info
May 15 16:13:17 repo1 consul[20266]: 2018/05/15 16:13:17 [INFO] agent: Synced service "vault:x.x.x.x:5819"
May 15 16:13:17 repo1 consul[20266]: 2018/05/15 16:13:17 [INFO] agent: Synced check "vault:x.x.x.x:5819:vault-sealed-check"
May 15 16:13:20 repo1 consul[20266]: 2018/05/15 16:13:20 [INFO] agent: Synced check "vault:x.x.x.x:5819:vault-sealed-check"
May 15 16:13:46 repo1 consul[20266]: 2018/05/15 16:13:46 [INFO] serf: attempting reconnect to vconsul_float.imsxxxx z.z.z.z:5824
May 15 16:13:48 repo1 consul[20266]: 2018/05/15 16:13:48 [INFO] agent: Deregistered service "vault:x.x.x.x:5819"
May 15 16:13:48 repo1 consul[20266]: 2018/05/15 16:13:48 [INFO] agent: Deregistered check "vault:x.x.x.x:5819:vault-sealed-check"
May 15 16:13:58 repo1 consul[20266]: 2018/05/15 16:13:58 [INFO] agent: Synced service "vault:x.x.x.x:5819"
May 15 16:13:58 repo1 consul[20266]: 2018/05/15 16:13:58 [INFO] agent: Synced check "vault:x.x.x.x:5819:vault-sealed-check"
May 15 16:14:02 repo1 consul[20266]: 2018/05/15 16:14:02 [INFO] agent: Synced check "vault:x.x.x.x:5819:vault-sealed-check"
May 15 16:14:02 repo1 consul[20266]: 2018/05/15 16:14:02 [INFO] agent: Synced service "vault:x.x.x.x:5819"
May 15 16:14:02 repo1 consul[20266]: 2018/05/15 16:14:02 [INFO] agent: Synced check "vault:x.x.x.x:5819:vault-sealed-check"
May 15 16:14:16 repo1 consul[20266]: 2018/05/15 16:14:16 [INFO] serf: attempting reconnect to vconsul_float z.z.z.z:5823
May 15 16:14:46 repo1 consul[20266]: 2018/05/15 16:14:46 [INFO] serf: attempting reconnect to vconsul_float z.z.z.z:5823
May 15 16:15:16 repo1 consul[20266]: 2018/05/15 16:15:16 [INFO] serf: attempting reconnect to vconsul_float.imsxxxx z.z.z.z:5824
May 15 16:15:16 repo1 consul[20266]: 2018/05/15 16:15:16 [INFO] serf: attempting reconnect to vconsul_float z.z.z.z:5823
consul instance 2 logs(y.y.y.y):
May 15 16:13:25 repo2 consul[13626]: 2018/05/15 16:13:25 [ERR] raft: Failed to heartbeat to z.z.z.z:5825: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
May 15 16:13:26 repo2 consul[13626]: 2018/05/15 16:13:26 [WARN] Unable to get address for server id f8591287-efee-97bb-95c0-2ecd39616cb5, using fallback address z.z.z.z:5825: Could not find address for server id f8591287-efee-97bb-95c0-2ecd39616cb5
May 15 16:13:26 repo2 consul[13626]: 2018/05/15 16:13:26 [ERR] raft: Failed to AppendEntries to {Voter f8591287-efee-97bb-95c0-2ecd39616cb5 z.z.z.z:5825}: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
May 15 16:13:30 repo2 consul[13626]: 2018/05/15 16:13:30 [WARN] Unable to get address for server id f8591287-efee-97bb-95c0-2ecd39616cb5, using fallback address z.z.z.z:5825: Could not find address for server id f8591287-efee-97bb-95c0-2ecd39616cb5
May 15 16:13:30 repo2 consul[13626]: 2018/05/15 16:13:30 [ERR] raft: Failed to heartbeat to z.z.z.z:5825: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
May 15 16:13:36 repo2 consul[13626]: 2018/05/15 16:13:36 [WARN] Unable to get address for server id f8591287-efee-97bb-95c0-2ecd39616cb5, using fallback address z.z.z.z:5825: Could not find address for server id f8591287-efee-97bb-95c0-2ecd39616cb5
May 15 16:13:36 repo2 consul[13626]: 2018/05/15 16:13:36 [ERR] raft: Failed to AppendEntries to {Voter f8591287-efee-97bb-95c0-2ecd39616cb5 z.z.z.z:5825}: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
May 15 16:13:40 repo2 consul[13626]: 2018/05/15 16:13:40 [WARN] Unable to get address for server id f8591287-efee-97bb-95c0-2ecd39616cb5, using fallback address z.z.z.z:5825: Could not find address for server id f8591287-efee-97bb-95c0-2ecd39616cb5
May 15 16:13:40 repo2 consul[13626]: 2018/05/15 16:13:40 [ERR] raft: Failed to heartbeat to z.z.z.z:5825: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
May 15 16:13:45 repo2 consul[13626]: 2018/05/15 16:13:45 [WARN] consul.kvs: Rejecting lock of kmfdata/core/lock due to lock-delay until 2018-05-15 16:14:00.347058073 +0530 IST
May 15 16:13:45 repo2 consul[13626]: 2018/05/15 16:13:45 [WARN] consul.kvs: Rejecting lock of kmfdata/core/lock due to lock-delay until 2018-05-15 16:14:00.347058073 +0530 IST
May 15 16:13:46 repo2 consul[13626]: 2018/05/15 16:13:46 [WARN] Unable to get address for server id f8591287-efee-97bb-95c0-2ecd39616cb5, using fallback address z.z.z.z:5825: Could not find address for server id f8591287-efee-97bb-95c0-2ecd39616cb5
May 15 16:13:46 repo2 consul[13626]: 2018/05/15 16:13:46 [ERR] raft: Failed to AppendEntries to {Voter f8591287-efee-97bb-95c0-2ecd39616cb5 z.z.z.z:5825}: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
May 15 16:13:48 repo2 consul[13626]: 2018/05/15 16:13:48 [INFO] agent: Deregistered service "vault:y.y.y.y:5819"
May 15 16:13:48 repo2 consul[13626]: 2018/05/15 16:13:48 [INFO] agent: Deregistered check "vault:y.y.y.y:5819:vault-sealed-check"
May 15 16:13:50 repo2 consul[13626]: 2018/05/15 16:13:50 [WARN] Unable to get address for server id f8591287-efee-97bb-95c0-2ecd39616cb5, using fallback address z.z.z.z:5825: Could not find address for server id f8591287-efee-97bb-95c0-2ecd39616cb5
May 15 16:13:50 repo2 consul[13626]: 2018/05/15 16:13:50 [ERR] raft: Failed to heartbeat to z.z.z.z:5825: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
May 15 16:13:56 repo2 consul[13626]: 2018/05/15 16:13:56 [WARN] Unable to get address for server id f8591287-efee-97bb-95c0-2ecd39616cb5, using fallback address z.z.z.z:5825: Could not find address for server id f8591287-efee-97bb-95c0-2ecd39616cb5
May 15 16:13:56 repo2 consul[13626]: 2018/05/15 16:13:56 [ERR] raft: Failed to AppendEntries to {Voter f8591287-efee-97bb-95c0-2ecd39616cb5 z.z.z.z:5825}: dial tcp y.y.y.y:0->z.z.z.z:5825: getsockopt: connection refused
consul instance 3 logs(z.z.z.z):
Consul service instance 3 wan't active at this timeframe.
vault instance 1 logs(x.x.x.x):
2018/05/15 16:13:39.493898 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:39.520097 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:39.821918 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:39.883150 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.047783 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.058892 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.204434 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.224893 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.395825 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.421182 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.550948 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.562523 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.781571 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.820538 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.908266 [DEBUG] forwarding: error sending echo request to active node: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:41.138147 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:41.165794 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:41.509211 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:41.525078 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
vault instance 2 logs(y.y.y.y):
2018/05/15 16:13:40.087007 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.103014 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.118278 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.136395 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.169163 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.194294 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.258137 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.294508 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.287678 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.337178 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.363053 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.372064 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.437709 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.479494 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.486514 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.502216 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
2018/05/15 16:13:40.539262 [ERROR] core: error during forwarded RPC request: error=rpc error: code = Unavailable desc = all SubConns are in TransientFailure
2018/05/15 16:13:40.549127 [ERROR] http/handleRequestForwarding: error forwarding request: error=error during forwarding RPC request
Please let us know when this issue could happen, and why vault returns success, when data hasn't been written to consul or not processed to the end step.
Thanks in advance.
Best Regards,
Bharath B