[kubernetes/kubernetes] Healthchecking of SSH Tunneler seem to be broken (#59347)
Wojciech Tyczynski
Healthchecking of Tunneler is part of healthz of apiserver. However, that seems to be a bit broken:
-
when we start, this is always ok:
https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go#L52
https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go#L131 -
If (because of some reason, e.g. user disabled compute API in GCE) we are not update SSHKeys, after 10 minutes, it will always turn into "failed" - this seems pretty bad to me.
-
If the call fails because of some reason, there is a big chance that we will fail health-check too. The problem is that:
- we are doing the installSSHKey sync loop every 5m: https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go#L163
- if we didn't finish the update for 10m+, we report failure: https://github.com/kubernetes/kubernetes/blob/master/pkg/master/tunneler/ssh.go#L62
This condition should be much less conservative.
@gmarek @kubernetes/sig-api-machinery-bugs
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
Daniel Smith
If this is configured to be on but not functioning, then proxy, port forward, exec, attach, some admission webhooks and some aggregated apiservers are all broken. The latter two are concerning as they are part of the control plane. I think calling the control plane unhealthy for this state is fair.
We don't really have a concept of "live but in degraded mode", but maybe we should invent one.
Wojciech Tyczynski
Let's be clear - I'm not saying this shouldn't be part of healthcheck.
I guess what I'm saying is:
- the installSSHKey should either be called more often or the timeout sholld be longer than 10m (single failure shouldn't cause going into non-health state)
- I think we should check both things at start, instead of entering the "healthy" state without setting up tunnelers.
Does that make sense?
Matt Liggett
@wojtek-t they both make sense, but we need to be careful that the apiserver pod spec's liveness intervals will play well with this since the apiserver will start out unhealthy and these can be somewhat high-latency operations.
Wojciech Tyczynski
Actually, I think that:
- (1) isn't very controversial
- (2) - agree that it's fairly non-trivial and I had the same concern that you have (otherwise I would have already done that). I'm not entirely sure how to solve it, but where we are now seems definitely like bad state to me.
Matt Liggett
Related: #55453
fejta-bot
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
Wojciech Tyczynski
/remove-lifecycle stale.
/lifecycle frozen
Mario Valderrama
I'm currently having issues with the 2. point you have mentioned in your initial message. I use an external CCM which doesn't implement the required function. I would propose to fix this by simply skipping the check if the interface doesn't support it.
Jordan Liggitt
ssh tunnel was removed in 1.22
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are on a team that was mentioned.
Jordan Liggitt
Closed #59347.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you are on a team that was mentioned.