[kubernetes/kubernetes] Failed to set statusUpdateNeeded to needed true because nodeName="node-v1-test" does not exist (#42158)
Maciej Szulik
Noticed that in here, this part:
W0227 04:39:11.830] E0227 04:39:11.830126 21448 actual_state_of_world.go:461] Failed to set statusUpdateNeeded to needed true because nodeName="node-v1-test" does not exist
W0227 04:39:11.830] E0227 04:39:11.830158 21448 actual_state_of_world.go:475] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node-v1-test" does not exist
I0227 04:39:11.931] node "node-v1-test" created
W0227 04:39:12.232] Error from server (Conflict): error when replacing "STDIN": Operation cannot be fulfilled on nodes "node-v1-test": the object has been modified; please apply your changes to the latest version and try again
W0227 04:39:12.239] !!! [0227 04:39:12] Call tree:
W0227 04:39:12.241] !!! [0227 04:39:12] 1: /go/src/k8s.io/kubernetes/hack/make-rules/test-cmd-util.sh:2917 run_pod_tests(...)
@kubernetes/sig-node-bugs || @kubernetes/sig-storage-bugs can you have a look at it?
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Andy Goldstein
Dupe of #41892
Andy Goldstein
Closed #42158.
YiChen Wang
I am also the same problem
dmnord
And me too! How to resolve this issue? Where can I get right documentation/description? :-(
Frank Solli
Same here.
YichenWong
@resouer Appears again in the 1.9.1 version.
[root@ip-10-101-2-181 ~]# tail -500f /data/logs/kubernetes/kube-controller-manager.ERROR Log file created at: 2018/01/30 18:38:59 Running on machine: ip-10-101-2-181 Binary: Built with gc go1.9.2 for linux/amd64 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E0130 18:38:59.082300 10461 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-101-2-91.cn-north-1.compute.internal" does not exist E0130 18:38:59.082841 10461 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-101-2-168.cn-north-1.compute.internal" does not exist E0130 18:38:59.082887 10461 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-101-2-204.cn-north-1.compute.internal" does not exist
Deven Fan
Kuberntes 1.9.1
root@ubuntu-105:/var/log/kubernetes# tail -f kube-controller-manager.ERROR
Log file created at: 2018/02/04 22:44:19
Running on machine: ubuntu-105
Binary: Built with gc go1.9.2 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0204 22:44:19.133862 2013 core.go:73] Failed to start service controller: WARNING: no cloud provider provided, services of type LoadBalancer will fail
E0204 22:44:29.316850 2013 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="ubuntu-64" does not exist
mritd
Same here.(1.10.0 when reboot controller manager)
Andy Goldstein
Sorry, just noticed the updates on this issue.
The history on this issue is a little confusing. It was opened because a test failed in CI. While it looks like the failure was this:
W0227 04:39:11.830] E0227 04:39:11.830158 21448 actual_state_of_world.go:475] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node-v1-test" does not exist
it was actually this:
W0227 04:39:12.232] Error from server (Conflict): error when replacing "STDIN": Operation cannot be fulfilled on nodes "node-v1-test": the object has been modified; please apply your changes to the latest version and try again
The test that failed was a kubectl replace call to replace a node. It failed because there was a conflict, and the apiserver was using stale data from its watch cache (filed as #41892). The fix was to fetch the latest version of the conflicted object, and then retry the operation (#43152).
As far as the Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true because nodeName="node-v1-test" does not exist error, I assume that is unrelated to the stale watch cache data issue that has since been fixed.
We can either reopen this and repurpose it as a storage issue, or we can create a new one. Either is fine with me.
mritd
On my 1.10.0 cluster (the three master nodes), when I try to restart the controller-manager component of a node, the controller-manager of other nodes will try to re-election. The controller-manager component logs after successful election will There is this warning message.
Apr 16 17:47:46 test37.node hyperkube[7518]: W0416 17:47:46.152050 7518 reflector.go:341] k8s.io/kubernetes/vendor/k8s.io/client-go/informers/factory.go:87: watch of *v1beta1.Event ended with: The resourceVersion for the provided watch is too old.
mritd
Gabriel Sousa
v.1.10.2 has the problem too
W0516 12:21:51.186408 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="app197" does not exist
W0516 12:21:51.186438 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="app198" does not exist
W0516 12:21:51.187172 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="app199" does not exist
Jing Xu
@gabrielfsousa could you please provide more information about the error? In what situation you saw this error? How long did it last? Any other problems except the log message?
Also, as @ncdc mentioned, the initial issue for this thread might be very different from what you have now. If it is the case, we can open another issue.
primeroz
@ncdc @jingxu97 I still see the same issue on v1.11.2 , exact same output
W0829 09:30:43.715195 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="worker-medium01" does not exist
W0829 09:30:43.715403 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="master01" does not exist
W0829 09:30:43.715781 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="master02" does not exist
W0829 09:30:43.715864 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="master03" does not exist
W0829 09:30:43.716009 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="worker-medium02" does not exist
W0829 09:30:43.716221 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="worker-medium03" does not exist
Everytime my controller-manager restart / failover
Bryan Carmichael
I also still see this in 1.13.3.
W0227 12:56:10.652298 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-4" does not exist W0227 12:56:10.652355 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-7" does not exist W0227 12:56:10.652391 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-13" does not exist W0227 12:56:10.652441 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-6" does not exist W0227 12:56:10.652477 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="master-2" does not exist W0227 12:56:10.652501 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-9" does not exist W0227 12:56:10.652532 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="master-0" does not exist W0227 12:56:10.652554 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-0" does not exist W0227 12:56:10.652608 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-5" does not exist W0227 12:56:10.652674 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="master-1" does not exist W0227 12:56:10.652712 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-12" does not exist W0227 12:56:10.653014 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-3" does not exist W0227 12:56:10.653038 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-14" does not exist W0227 12:56:10.653054 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-2" does not exist W0227 12:56:10.653075 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-11" does not exist W0227 12:56:10.653086 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-1" does not exist W0227 12:56:10.653097 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-8" does not exist W0227 12:56:10.653110 1 actual_state_of_world.go:491] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="minion-10" does not exist
Aisuko
Hi guys, I hit the same issue on v1.9. Is there more information can help us? We deployed operator which the reconcile conflict with kubernetes controller-manager? or the watch-list cache mechnism has some issue in some point can trigger?
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
George
I had the same issue,I used v1.21.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are on a team that was mentioned.