kube-apiserver: Self-coordination

19 views
Skip to first unread message

Daniel Smith

unread,
Apr 3, 2019, 5:03:40 PM4/3/19
to K8s API Machinery SIG, kubernetes-sig-architecture
Inspired by a conversation Clayton & I had in #sig-api-machinery, here's an idea for how apiserver replicas could coordinate over things that really ought to be constant across replicas (e.g., what CR handlers are installed?): https://docs.google.com/document/d/1vHaEL6S6LSOjCt4HbXMm2El4lo5boNc5FZ8vIOQri6E/edit

Copy-paste below for people who have difficulty opening google docs:

Just Make API Objects!

lavalamp/dbsmith * 2019-04-03 * status: opinion piece * PUBLIC


Problem: kube-apiserver sometimes needs to coordinate with other replicas of itself. Today, we don’t do it (e.g., API Service / CRD status), or do it in an ad-hoc, non transparent way (e.g., IP allocations).


Suggested solution pattern:


Reasoning:

  • We need to handle forward/backwards compatibility. We already built a thing for that.

    • This needn’t slow us down; we don’t necessarily have to keep a deprecation policy as extreme as the usual one, since we discourage clients other than apiserver itself.

  • No up-front investment is necessary, all the pieces exist.

  • This permits external introspection into apiserver internal state.

    • Debugging!

    • On the off chance it’s needed, it’ll be possible to fix the state!


List of apiserver responsibilities that could benefit from this pattern:

  • Coordinating which CRDs are currently being served

  • Coordinating which APIServices are currently being served

  • Generally speaking, the expanded discovery doc.

  • Coordinating global apiserver settings / flag-flips

    • Serving certs

    • Storage versions & enabled apis

    • Feature gates (!)

    • ...without surveying, I expect 50%-80% of our flags ought to be consistent between apiserver replicas

  • Service IP allocations / Node port allocations

  • apiserver endpoint management (we sort of already do this by weirdly self-managing the apiserver endpoints record)

Brian Grant

unread,
Apr 3, 2019, 6:06:58 PM4/3/19
to Daniel Smith, K8s API Machinery SIG, kubernetes-sig-architecture
+1.



--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZfryvK547eAmKvgbfvET7RvMH2osdcwyS%3D%2BwN_jgGycw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Wojciech Tyczynski

unread,
Apr 4, 2019, 3:23:24 AM4/4/19
to Brian Grant, Daniel Smith, K8s API Machinery SIG, kubernetes-sig-architecture
 +1 - that sounds like a good option, especially given that in some cases (ipallocations) we already do almost exactly the same thing.

You received this message because you are subscribed to the Google Groups "kubernetes-sig-architecture" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-arch...@googlegroups.com.
To post to this group, send email to kubernetes-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAKCBhs69bakbnaJ0EXSu8UJzXqgfTj708F2vOos_48ksCs7yTQ%40mail.gmail.com.

David Eads

unread,
Apr 4, 2019, 8:23:09 AM4/4/19
to Wojciech Tyczynski, Brian Grant, Daniel Smith, K8s API Machinery SIG, kubernetes-sig-architecture
I'm ok with the idea in general (not as excited as others perhaps), but the kube-apiserver wasn't designed or built with dynamic config capabilities in mind for most values.  While some may be easy to retrofit, I don't know that I'd expect most of them to reallocate cleanly.  It's probably worth thinking about how much time is involved in such an effort and comparing it against other priorities we may have before ending up with config spread across multiple locations for a potentially significant period of time.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAPgfqyWWTaJ9FgTHDT7GpOc%3DQ9vS9zqQ26R2NCQ%2B27RZz%3DjvDQ%40mail.gmail.com.

Kenneth Owens

unread,
Apr 4, 2019, 9:07:41 AM4/4/19
to David Eads, Wojciech Tyczynski, Brian Grant, Daniel Smith, K8s API Machinery SIG, kubernetes-sig-architecture
+1. This is the use case etcd was originally designed for. Might want to think about doing this with controller manager as well. Agree with David that we need to decide what configuration is dynamic.
Thanks,
    -Ken

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-architecture/CAFS1MjLeBsasei-t1RTuQD6wMxY%3Dy_PAei5Y_C7AUDJdahXsOQ%40mail.gmail.com.

Daniel Smith

unread,
Apr 4, 2019, 12:20:08 PM4/4/19
to Kenneth Owens, David Eads, Wojciech Tyczynski, Brian Grant, K8s API Machinery SIG, kubernetes-sig-architecture
On Thu, Apr 4, 2019 at 6:07 AM Kenneth Owens <owe...@google.com> wrote:
+1. This is the use case etcd was originally designed for. Might want to think about doing this with controller manager as well. Agree with David that we need to decide what configuration is dynamic.
Thanks,
    -Ken

On Thu, Apr 4, 2019, 5:23 AM David Eads <de...@redhat.com> wrote:
I'm ok with the idea in general (not as excited as others perhaps), but the kube-apiserver wasn't designed or built with dynamic config capabilities in mind for most values.

Yeah, specifically, I'm not saying I'm going to make a bunch of changes here, the point is:
* Don't block people/features that need self coordination
* But don't do any work for them, either.

The burden of getting the coordination correct is going to be on the shoulders of people adding things that use it.

If I found time to spend on this, the thing I'd personally be tempted to fix is making the service IP allocation object into a real API object.

But mostly the point of this is give out a pattern that will solve the problem, so that no one is blocked. If someone wants to make a config value dynamically updatable, they have... a lot of other work to do to just make it changable in a *single* apiserver.

Chao Xu

unread,
Apr 4, 2019, 12:31:31 PM4/4/19
to Daniel Smith, Kenneth Owens, David Eads, Wojciech Tyczynski, Brian Grant, K8s API Machinery SIG, kubernetes-sig-architecture
I'll take a stab at making the discovery document consistent across apiservers. Otherwise the storage migrator does not work safely with HA masters.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bZAOjabtrJrVww5mOczPSX_2pwKLCnUyChCjQhnC-BtvQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.


--
- Chao

Brian Grant

unread,
Apr 4, 2019, 1:02:05 PM4/4/19
to Chao Xu, Daniel Smith, Kenneth Owens, David Eads, Wojciech Tyczynski, K8s API Machinery SIG, kubernetes-sig-architecture
Reply all
Reply to author
Forward
0 new messages