raft: Failed to contact

[David Birdsong]

I'm seeing a lot of messages like this:

2016/04/26 20:39:01 [WARN] raft: Failed to contact 10.2.7.25:8300 in 513.653662ms
2016/04/26 20:39:01 [WARN] raft: Failed to contact 10.2.8.90:8300 in 624.19334ms
2016/04/26 20:39:01 [WARN] raft: Failed to contact 10.2.8.25:8300 in 513.600026ms
2016/04/26 20:39:01 [WARN] raft: Failed to contact 10.2.8.215:8300 in 513.586349ms
2016/04/26 20:39:01 [WARN] raft: Failed to contact quorum of nodes, stepping down

Which is causing leadership to churn constantly. There are brief periods w/ no leader.

I'm running this in my own datacenter and our network guy has yet to find any reason why RTT should ever be more than a 0.5ms between these nodes. Does the measurement indicate network RTT or does it also include any transactions which might bring w/ them contention for mutex, I/O,  etc...?

[James Phillips]

Hi David,

It's the latter - it simply subtracts the last time it processed a message from the other node, so it could encompass all those different problems. Typically we've seen this when processes get starved for CPU and no longer respond in a soft real time fashion, or when there are periods of heavy packet loss between the servers.

-- James

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CABJfL5i2NMsAnhceYXq4538c4Di28xkSU6K0LWYsqjcaVeR_AA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[David Birdsong]

Great thanks for the clarification.