[Question] Bulk Server-side Apply

Austin Cawley-Edwards

unread,

Mar 20, 2023, 2:43:12 PM3/20/23

to K8s API Machinery SIG

Hey all,

Does anyone know if something like "bulk server-side apply" has been asked for or discussed in the past? I'm thinking of an endpoint that accepts multiple objects, and does a SSA on each. Moving `kubectl apply -f all-my-resources.yaml` closer to the server.

One step further into chaos would be allowing these bulk changes to be "all or nothing" -- not arbitrary read-modify-write transactions but still makes me uneasy. Perhaps SSA semantics help here if conflicts are always forced?

Best,

Austin

Han Kang

unread,

Mar 20, 2023, 2:47:13 PM3/20/23

to Austin Cawley-Edwards, K8s API Machinery SIG

K8s doesn't support multi-object transactions and the watch protocol basically forbids this from being implemented. The best we could do is serialize the operations server-side which doesn't seem to be what you are asking for in terms of changes being "all or nothing".

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/264dbe78-bf47-41f1-907d-f87b8fe9403dn%40googlegroups.com.

--

- Han

Daniel Smith

unread,

Mar 20, 2023, 2:51:30 PM3/20/23

to Austin Cawley-Edwards, K8s API Machinery SIG

apserver doesn't support any batch requests (unless you count DELETECOLLECTION which is its own discussion).

Specifically we don't want to expose all-or-nothing multi object TXs.

I realize this is a common request but it doesn't fit well with k8s choices re: what kind of distributed system it is.

We would have the same problems implementing this in apiserver that you would have implementing it out of apiserver. E.g., how to prevent conflicting changes while we put things in, or how to prevent controllers from actuating a state that might be reverted.

On Mon, Mar 20, 2023 at 11:43 AM Austin Cawley-Edwards <austin...@gmail.com> wrote:

--

Jordan Liggitt

unread,

Mar 20, 2023, 2:53:33 PM3/20/23

to Daniel Smith, Austin Cawley-Edwards, K8s API Machinery SIG

A few prior issues around batch multi-object endpoint handling:

A bulk write endpoint certainly wouldn't play nicely with authorization (which is resource / endpoint-level) or aggregation (where different servers are responsible for running admission/validation on the body of the request)

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bY5yB3B8XOZTzCRfh1Ag23%3DZZFth%2Bp6oq8Jj%2BB94CYzwg%40mail.gmail.com.

Karl Isenberg

unread,

Mar 20, 2023, 4:24:03 PM3/20/23

to Jordan Liggitt, Austin Cawley-Edwards, Daniel Smith, K8s API Machinery SIG

While it doesn’t sound like we’re going to get a server-side bulk apply any time soon, there has a been a lot of time and effort invested into client-side bulk apply.

The cli-utils project includes Applier and Destroyer bulk clients which include the following features:

- bulk apply/delete

- inventory object to track apply sets by name

- pruning by removal from apply set

- operations event stream

- phased apply grouping

- configurable adoption strategies

- waiting for reconciliation after apply/delete

- implicit and explicit object dependency ordering

- apply-time mutation with dynamic inputs

- apply namespaced objects and namespace at the same time

- apply custom resources and definitions at the same time

- apply resources without a typed client or client-side schema

- custom event handing CLI printers

- dynamic multi-resource watch library

- status interpretation library

- jsonpath library with read and write by field path

Check it out:

https://github.com/kubernetes-sigs/cli-utils

And check out Google’s GitOps operator, Config Sync, which uses cli-utils and adds more features on top, like automatic drift remediation:

https://github.com/GoogleContainerTools/kpt-config-sync

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAMBP-pJ%3D16n1L-1T4j-k%2BAAmT0RUbX9bsw-%3DezWVGsQtTBD2Vw%40mail.gmail.com.

Austin Cawley-Edwards

unread,

Mar 20, 2023, 4:47:26 PM3/20/23

to K8s API Machinery SIG

Thanks for all the feedback, everyone. To be clear: this isn't an ask, was just wondering what the implications would be as I've gotten some questions internally about it recently.

> The best we could do is serialize the operations server-side

Yeah, this is what I was thinking as well. Maybe the "all or nothing" desire goes away / can be mitigated if we can guarantee each individual update will go through (i.e. forcing conflicts) and ignore the cases where there's a network failure mid-apply, etc.

> I realize this is a common request but it doesn't fit well with k8s choices re: what kind of distributed system it is.

I completely agree with this, and that you can design resources in a way that doesn't rely on this with little overhead, but wondering if there's a good response to the "if we were backed by a data system that provided multirow transactions (e.g. postgres, dynamo), what's stopping us"? There are other good reasons listed in this thread, just wondering if there's one specifically around what it does to the underlying storage system -- performance nightmare w/ multirow locks?

---

The other arguments against all make sense -- thanks everyone for the input. Clients can indeed solve these problems themselves :)

Daniel Smith

unread,

Mar 20, 2023, 5:01:47 PM3/20/23

to Austin Cawley-Edwards, K8s API Machinery SIG

On Mon, Mar 20, 2023 at 1:47 PM Austin Cawley-Edwards <austin...@gmail.com> wrote:

Thanks for all the feedback, everyone. To be clear: this isn't an ask, was just wondering what the implications would be as I've gotten some questions internally about it recently.

> The best we could do is serialize the operations server-side

Yeah, this is what I was thinking as well. Maybe the "all or nothing" desire goes away / can be mitigated if we can guarantee each individual update will go through (i.e. forcing conflicts) and ignore the cases where there's a network failure mid-apply, etc.

> I realize this is a common request but it doesn't fit well with k8s choices re: what kind of distributed system it is.

I completely agree with this, and that you can design resources in a way that doesn't rely on this with little overhead, but wondering if there's a good response to the "if we were backed by a data system that provided multirow transactions (e.g. postgres, dynamo), what's stopping us"? There are other good reasons listed in this thread, just wondering if there's one specifically around what it does to the underlying storage system -- performance nightmare w/ multirow locks?

etcd *does* support transactions! It's the k8s paradigm that doesn't work well / at all with this. A different data store wouldn't change that.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/18373131-a33e-4a57-a623-4d3f1c35c7edn%40googlegroups.com.

Austin Cawley-Edwards

unread,

Mar 20, 2023, 5:05:21 PM3/20/23

to K8s API Machinery SIG

Ah, my mistake -- thanks :)

Han Kang

unread,

Mar 20, 2023, 5:05:46 PM3/20/23

to Daniel Smith, Austin Cawley-Edwards, K8s API Machinery SIG

Specifically, we'd have to rewrite the watch protocol (v2, anybody?) in order to support multi-object transactions. The current watch protocol only allows emitting a single object per watch event, which means if we have multiple objects which share an RV (a consequence of a multi-object transactions for etcd), then our protocol is broken, since we could get object at RV=X, disconnect our watch, and resume watch at RV X+1 (which could end up missing N events which share RV=X).

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bayn4Ad7Dt%2BFfSvZ3POg4DG%3DU%3D%3DL3sxcVm3yK8Jero6Gg%40mail.gmail.com.