Re: [kubernetes/kubernetes] [test failed] Services should work after restarting apiserver (#60761)
Zihong Zheng
I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures:
Message: "an error on the server (\"apiserver is shutting down.\") has prevented the request from succeeding (post pods)
or
Post https://146.148.93.99/api/v1/namespaces/e2e-tests-services-7xhz8/pods: unexpected EOF
or
Post https://35.193.65.117/api/v1/namespaces/e2e-tests-services-bpk66/pods: EOF
To elaborate a bit more, a snippet of the test procedure is:
- ssh to master and
sudo kill $APISERVER_PID(sending a SIGTERM) - Keep polling on /healthz on apiserver until 200 is returned.
- Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating).
According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors.
@kubernetes/sig-api-machinery-test-failures Could anyone confirmed the SIGTERM theory?
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Kubernetes Submit Queue
[MILESTONENOTIFIER] Milestone Issue Needs Attention
@MrHohn @bowei @caseydavenport @dcbw @krzyzacy @rramkumar1 @thockin @kubernetes/sig-api-machinery-misc @kubernetes/sig-network-misc
Action required: During code freeze, issues in the milestone should be in progress.
If this issue is not being actively worked on, please remove it from the milestone.
If it is being worked on, please add the status/in-progress label so it can be tracked with other in-flight issues.
Note: This issue is marked as priority/critical-urgent, and must be updated every 1 day during code freeze.
Example update:
ACK. In progress
ETA: DD/MM/YYYY
Risks: Complicated fix required
sig/api-machinerysig/network: Issue will be escalated to these SIGs if needed.priority/critical-urgent: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels.kind/bug: Fixes a bug discovered during the current release.
Zihong Zheng
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test.
Zihong Zheng
Sent #60906.
Kubernetes Submit Queue
[MILESTONENOTIFIER] Milestone Issue Needs Attention
@MrHohn @bowei @caseydavenport @dcbw @krzyzacy @rramkumar1 @thockin @kubernetes/sig-api-machinery-misc @kubernetes/sig-network-misc
Action required: During code freeze, issues in the milestone should be in progress.
If this issue is not being actively worked on, please remove it from the milestone.
If it is being worked on, please add the status/in-progress label so it can be tracked with other in-flight issues.
Action Required: This issue has not been updated since Mar 8. Please provide an update.
Note: This issue is marked as priority/critical-urgent, and must be updated every 1 day during code freeze.
Example update:
ACK. In progress
ETA: DD/MM/YYYY
Risks: Complicated fix required
sig/api-machinerysig/network: Issue will be escalated to these SIGs if needed.priority/critical-urgent: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels.kind/bug: Fixes a bug discovered during the current release.
—
Zihong Zheng
Test is now passing on gce but failing on gke. Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
Kubernetes Submit Queue
[MILESTONENOTIFIER] Milestone Issue Needs Attention
@MrHohn @bowei @caseydavenport @dcbw @krzyzacy @rramkumar1 @thockin @kubernetes/sig-api-machinery-misc @kubernetes/sig-network-misc
Action required: During code freeze, issues in the milestone should be in progress.
If this issue is not being actively worked on, please remove it from the milestone.
If it is being worked on, please add the status/in-progress label so it can be tracked with other in-flight issues.
Note: This issue is marked as priority/critical-urgent, and must be updated every 1 day during code freeze.
Example update:
ACK. In progress
ETA: DD/MM/YYYY
Risks: Complicated fix required
sig/api-machinerysig/network: Issue will be escalated to these SIGs if needed.priority/critical-urgent: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels.kind/bug: Fixes a bug discovered during the current release.
—
Zihong Zheng
Test is now passing on gce but failing on gke. Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/