Re: [kubernetes/kubernetes] etcd error in cluster downgrade: membership: cluster cannot be downgraded (current version: 3.0.17 is lower than determined cluster version: 3.1). (#54027)
Zihong Zheng
@kubernetes/sig-api-machinery-test-failures
/area etcd
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Zihong Zheng
Jing Ai
I investigated the issue a bit.
The etcd version for a k8s release is packed in cluster/saltbase/salt/etcd/etcd.manifest, e.g., the one from HEAD.
"env": [
{ "name": "TARGET_STORAGE",
"value": "{{ pillar.get('storage_backend', 'etcd3') }}"
},
{ "name": "TARGET_VERSION",
"value": "{{ pillar.get('etcd_version', '3.1.10') }}"
},
{ "name": "DATA_DIRECTORY",
"value": "/var/etcd/data{{ suffix }}"
}
],
On the other hand, etcd (https://github.com/coreos/etcd/blob/master/etcdserver/membership/cluster.go) has the following logic to detect downgrade. In case it happens, it will trigger a fatal error and log.
func mustDetectDowngrade(cv *semver.Version) {
lv := semver.Must(semver.NewVersion(version.Version))
// only keep major.minor version for comparison against cluster version
lv = &semver.Version{Major: lv.Major, Minor: lv.Minor}
if cv != nil && lv.LessThan(*cv) {
plog.Fatalf("cluster cannot be downgraded (current version: %s is lower than determined cluster version: %s).", version.Version, version.Cluster(cv.String()))
}
}
It sounds that it hits a lower version of etcd for the test.
Given this, I found etcd version was bumped up from 3.0.17 to 3.1.10 on 10/02/2017 (39e5a56#diff-0fa055cac191706bf4aff2385d1bc750). However it is not in any tagged k8s release yet.
cmluciano
Joe Betz
@xiangpengzhao Yes, etcd downgrades are not supported, see: https://coreos.com/etcd/docs/latest/upgrades/upgrade_3_1.html#downgrade.
Zihong Zheng
Closed #54027.
Zihong Zheng
Thanks all, let's use #57013 for tracking. Closing this.