Weird kubeadm upgrade apply issue

[Owen Ou]

Hi all,

I would like to understand why `kubeadm upgrade apply` behaves in a weird way. I have a running cluster at v1.10.3 and I'm trying to upgrade to the latest v1.10.4. I did the followings:

1. Downloaded kubeadm 1.10.3, ran `kubeadm upgrade apply v1.10.3` and it failed at upgrading control plane:

Static pod: kube-scheduler-ip-10-0-1-80.ec2.internal hash: ea66a171667ec4aaf1b274428a42a7cf

[upgrade/apply] FATAL: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: [timed out waiting for the condition]

2. Downloaded kubeadm 1.10.4, ran `kubeadm upgrade apply v1.10.3` and it also failed for the same error

3. Downloaded kubeadm 1.10.4, ran `kubeadm upgrade apply v1.10.4` and it worked

4. Downloaded kubeadm 1.10.4, after a successful run of `kubeadm upgrade apply v1.10.4`, consequent running of `kubeadm upgrade apply v1.10.3` always worked.

I expected 1 and 2 to work. 1 is an upgrade to its current version using a current version of kubeadm. 2 is an upgrade to its current version using a next version of kubeadm. 4 is weird that after a successful upgrade to the next version, consequent upgrade to its current version always work.

I also created a GH issue to with more details: https://github.com/kubernetes/kubernetes/issues/65071.

Thanks,

Owen

[Jason DeTiberus]

The issue you are hitting is that you are trying to upgrade from the current version to the current version, which should be a noop. It looks like we aren't checking for this in kubeadm and instead you are seeing an error the code that checks that the static manifest is updated. Since the manifest does not change kubeadm thinks that the manifest deploy failed rather than staying the same.

tldr; you are hitting a bug, but the bug is kubeadm is trying to perform an upgrade when it shouldn't.

--
Jason DeTiberus