ETCD Cluster Issue: Cluster has 5 nodes but showing only 3 as active(one leader, two followers)

[Ankur Sharma]

Hi,

We are using the a 5 node etcd cluster, created on bare metal servers and using CentOS Linux release 7.8.2003 (Core) and kernel version = 3.10.0-1127.13.1.el7.x86_64.

5 nodes:
192.168.11.11 (etcd Version: 3.2.22)

192.168.11.12 (etcd Version: 3.2.22)

192.168.11.13 (etcd Version: 3.2.22)

192.168.11.14 (etcd Version: 3.3.11)

192.168.11.15 (etcd Version: 3.3.11)

There is a virtual keepalived IP 192.168.11.10 on top of all 5 nodes.

Out of 5 nodes added to etcd cluster, only three are active and part of the cluster. The remaining two nodes not shown as part of cluster using below command:

etcdctl --endpoints http://192.168.11.10:2379 member list

2d9e663584f73: name=node1 peerURLs=http://192.168.11.11:2380 clientURLs=http://192.168.11.11:2379 isLeader=false

4d4d9565ba17f: name=node4 peerURLs=http://192.168.11.14:2380 clientURLs=http://192.168.11.14:2379 isLeader=true

33eb7ae514c37: name=node2 peerURLs=http://192.168.11.12:2380 clientURLs=http://192.168.11.12:2379 isLeader=false

Etcd instances, on all nodes, are started using following command by replacing only highlighted items specific to individual nodes:

/usr/bin/etcd --name node1 --initial-advertise-peer-urls http://192.168.11.11:2380 --listen-peer-urls http://192.168.11.11:2380 --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://192.168.11.11:2379 --initial-cluster-token x-cluster --initial-cluster node1=http://192.168.11.11:2380,node2=http://192.168.11.12:2380,node3=http://192.168.11.13:2380,node4=http://192.168.11.14:2380,node5=http://192.168.11.15:2380 --initial-cluster-state new --data-dir /home/testuser/etcd_data_dir

Following logs are seen on failing nodes but they all can reach each other properly:

Aug 19 22:55:51 node5 systemd: Started ETCD Service.

Aug 19 22:55:51 node5 etcd: etcd Version: 3.3.11

Aug 19 22:55:51 node5 etcd: Git SHA: 2cf9e51

Aug 19 22:55:51 node5 etcd: Go Version: go1.10.3

Aug 19 22:55:51 node5 etcd: Go OS/Arch: linux/amd64

Aug 19 22:55:51 node5 etcd: setting maximum number of CPUs to 8, total number of available CPUs is 8

Aug 19 22:55:51 node5 etcd: the server is already initialized as member before, starting as etcd member...

Aug 19 22:55:51 node5 etcd: listening for peers on http://192.168.11.15:2380

Aug 19 22:55:51 node5 etcd: listening for client requests on 0.0.0.0:2379

Aug 19 22:55:51 node5 etcd: name = node5

Aug 19 22:55:51 node5 etcd: data dir = /home/testuser/etcd_data_dir

Aug 19 22:55:51 node5 etcd: member dir = /home/testuser/etcd_data_dir/member

Aug 19 22:55:51 node5 etcd: heartbeat = 100ms

Aug 19 22:55:51 node5 etcd: election = 1000ms

Aug 19 22:55:51 node5 etcd: snapshot count = 100000

Aug 19 22:55:51 node5 etcd: advertise client URLs = http://192.168.11.15:2379

Aug 19 22:55:51 node5 etcd: restarting member 1add259db43f6 in cluster 6179e265d96e5 at commit index 0

Aug 19 22:55:51 node5 etcd: 1add259db43f6 became follower at term 0

Aug 19 22:55:51 node5 etcd: newRaft 1add259db43f6 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]

Aug 19 22:55:51 node5 etcd: simple token is not cryptographically signed

Aug 19 22:55:51 node5 etcd: starting server... [version: 3.3.11, cluster version: to_be_decided]

Aug 19 22:55:58 node5 etcd: publish error: etcdserver: request timed out

Aug 19 22:56:05 node5 etcd: publish error: etcdserver: request timed out

Could anyone help on this issue? Any pointers or help would be greatly appreciated.

[Mostafa Mahmoud]

Hello,

i'm facing same issue , did you resolve it?