Re: [kubernetes/kubernetes] "kubectl rollout status" returns error message before rollout finish (#40224)
Michail Kargakis
@kubernetes/sig-cli-bugs @kubernetes/sig-api-machinery-bugs either fix watch.Until or remove
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Sam Powers
I'm getting this periodically when running my kubectl connection across a stateful firewall, where the connection may be periodically reset if idle for a period. I don't know what the idle period is. But I'm getting ready to make a bandaid shell script for our usage of kubectl rollout status. Just in case that motivates anyone to make watch.Until more robust.
I hope this was useful feedback. Maybe forcing a connection reset would help in reproducing the issue?
Michail Kargakis
@sampowers would you be interested in working on fixing rollout status rather than writting that script? :) Most people (myself included) seem to be fairly busy but we would love to spare review time.
Keyan Pishdadian
I was seeing the same error as OP when using kubectl v1.5.6 but upon upgrading to v1.7.0-beta.2 I'm seeing a different error:
Waiting for rollout to finish: 2 out of 4 new replicas have been updated...
error: watch closed before Until timeout
This is thrown by this line. I'm consistently seeing the error after kubectl has executed the status command for exactly 10 minutes.
The change I made in #47617 doesn't seem relevant anymore. Initially I thought I was somehow hitting the seemingly impossible select case from the after channel. But it is clear to me now that Until was failing at the same location in v1.5.6 except the error thrown is different.
The issue of timing out after 10 minutes remains the same.
Tomáš Nožička
/assign
Andrew Holway
+1
Skyler Layne
Christopher Hlubek
We are having the same issue when checking for deployments in our GitLab / K8s infrastructure:
Waiting for rollout to finish: 0 of 1 updated replicas are available...
E1207 11:03:06.765491 137 streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 5; INTERNAL_ERROR
error: watch closed before Until timeout
Bryan Lee
I am having the same issue
bryanleekw@nb1:/proj/kubernetes$ kubectl rollout status deployment/monitor-scale
Waiting for rollout to finish: 0 of 1 updated replicas are available...
Waiting for rollout to finish: 0 of 1 updated replicas are available...
error: watch closed before Until timeout
my kubectl version is:
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}
Bryan Lee
may I know how can I check rollout status by log files?
gavk34
Seeing this also
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T20:00:41Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"6e5b33a290a99c067003632e0fd6be0ead48b233", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}
Sample log output
07-Mar-2018 08:04:41 2018/03/07 08:04:41 Waiting for rollout to finish: 0 of 1 updated replicas are available...
07-Mar-2018 08:04:41 Waiting for rollout to finish: 0 of 1 updated replicas are available...
07-Mar-2018 08:04:41 error: watch closed before Until timeout
07-Mar-2018 08:04:41 2018/03/07 08:04:41 Error occurred when running 'apply' command: exit status 1
creinheimer
Same problem here.
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.10-rancher1", GitCommit:"66aaf7681d4a74778ffae722d1f0f0f42c80a984", GitTreeState:"clean", BuildDate:"2018-03-20T16:02:56Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Log output:
Waiting for rollout to finish: 1 old replicas are pending termination...
F0329 14:43:18.723237 9889 helpers.go:119] error: watch closed before Until timeout
Tomáš Nožička
Jinming Yue
Meet the same problem：
# kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.9", GitCommit:"19fe91923d584c30bd6db5c5a21e9f0d5f742de8", GitTreeState:"clean", BuildDate:"2017-10-19T17:09:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.9", GitCommit:"19fe91923d584c30bd6db5c5a21e9f0d5f742de8", GitTreeState:"clean", BuildDate:"2017-10-19T16:55:06Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
# kubectl rollout status deploy test
Waiting for rollout to finish: 0 out of 2 new replicas have been updated...
Waiting for rollout to finish: 0 out of 2 new replicas have been updated...
error: watch closed before Until timeout
root cause：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: test
namespace: default
spec:
spec:
paused: true # Should be false.
...
Solution：
Set .spec.paused as false.
.spec.paused is an optional boolean field for pausing and resuming a Deployment. It defaults to false (a Deployment is not paused).
xref: http://kubernetes.kansea.com/docs/user-guide/deployments/#paused
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
Tomáš Nožička
/remove-lifecycle stale
/milestone v1.12
brandshaide
/remove-lifecycle-stale
guineveresaenger
@tnozicka can you update on whether any work is being done on this?
As Code Slush begins this Tuesday, please make sure this issue is labeled as approved for the milestone by then.
https://github.com/kubernetes/sig-release/tree/master/releases/release-1.12#code-slush
Tomáš Nožička
@guineveresaenger there is a PR, #67817 linked just above your comment, fixing the issue; in review process, still targeting v1.12
guineveresaenger
@tnozicka awesome! can you get milestone approval from someone in sig/apps?
Tomáš Nožička
/priority critical-urgent
@guineveresaenger is this for status/approved-for-milestone? I though that was effectively canceled on lazy consensus last week here https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/MjyJzhBEgkM/64kNFcIACQAJ
guineveresaenger
@tnozicka gah, apologies for the noise. You are right. My bad.
Tomáš Nožička
@guineveresaenger np :)
ocofaigh
Any plans to backport this fix to v1.11?
Arockiasamy K
If you use an AWS Load balancer, you want to increase the timeout value.
Morinaga
You could add a timeout flag to the kubectl rollout status command.
kubectl rollout status deployment/app --namespace=app --timeout=60s
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-status-em-
Arockiasamy K
@zyfu90 Even if you add this, the rollout will still fail - because it's the load balancer timeout which kills the watch and not Kubernetes. The error message clearly says that Kubernetes watch failed even before kubectl timeout.
Jonathan McCall
@arocki7 You mean the ELB idle timeout? Which would imply that while running rollout status the connection could be idle for that time? Doesn't the rollout status command poll for the status of the deployment?
Arockiasamy K
@Jonnymcc Unfortunately, rollout status didn't do a poll for every few seconds - which is why it fails.
Hamza Y.
Scale up your deployment and apply your deployment and service files, then back to your desired state.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.