Changing node names of serve-continuous leader re-elections and no consistent leader

[hokie...@gmail.com]

I recently changed the node names of my consul client agents to match the ip addresses of my cluster nodes. I did this because the name nodes generated from the nodes would periodically change due to a race condition resulting form multiple network interfaces upon node startup. The client agent node name change worked well and all client agents rejoined the cluster.

When I attempted to change the node names of the server agents, this put the cluster in a bad state. There are almost continuous leader elections and the cluster UI is unresponsive.

There are continuous messages from the bootstrap server: raft: Rejecting vote request from <bootstrap server ip>:8300 since we have a leader:<server ip>:8300.

However, I don't think there is actually no cluster leader at the time of this message as evidenced by the following message:
error: No cluster leader from=<ip>

I figured that the node id was the key thing to remain constant, but, apparently, changing the node name put my Consul cluster in a bad state

Question: what's the fix? Any guidance is very much appreciated.

Thanks

--John

[hokie...@gmail.com]

More error messages:

[WARN] raft: not part of a stable configuration, aborting election
[ERR] agent: coordinate update error: No Cluster leader

Finally, consul operator raft -list-peers only list 3 out of the five servers, and the bootstrap server is not listed, even though it is running.

[hokie...@gmail.com]

Okay, I switched the node names back to original names, restarted the boot server and then restarted the other servers and now the cluster is in a good state.

On Tuesday, November 21, 2017 at 4:41:40 PM UTC-5, hokie...@gmail.com wrote: