Two Lane Queue in client-go/workqueue

[Haibing Zhou]

Hi,

I'm trying to get some feedback on the following problem statements as well as a tentative proposal:

## The problem

As the number of object grows, the controllers may speed a lot of time doing its processing (during resync or when controller restarts). Often times, it is desired to process some of the items with higher priority:

* When controller restarts, all the ADD/Update events after likely should have a higher priority (assuming the existing ones are already handled before the restart).

* When a global resync happens, those items to be resynced don't have as high priority as those new events.

In resourcequota controller, its implementation started with two queues ([primary queue][1] and [priority queue][2]) just to address this unnecessary lag in some of the situations.

In knative/pkg, they have this as their first class citizen which is called [two lane queue][3].

I think this might be a common problem and would be great to start a conversation here on how may address this issue in client-go and give the community an "official" (sort of) version of solution.

## The proposal

I remember Tim was talking about managing k8s complexity during last kubecon, and I totally agree with it. Any new changes are high bar and need more eyes and considerations. I'm proposing https://github.com/kubernetes/kubernetes/pull/123347 as the starting point for us to implement something similar to two lane queue and gather more data (or interest) out of k8s tree for people to try out and report back.

It is my first post in this group, let me know if I should add more information.

[1]: https://github.com/kubernetes/kubernetes/blob/3516bc6f49eee4ca2d45ad3a13385c22ea76c247/pkg/controller/resourcequota/resource_quota_controller.go#L112
[2]: https://github.com/kubernetes/kubernetes/blob/3516bc6f49eee4ca2d45ad3a13385c22ea76c247/pkg/controller/resourcequota/resource_quota_controller.go#L113

[3]: https://github.com/knative/pkg/blob/main/controller/two_lane_queue.go

[Haibing Zhou]

Thanks David. 

I did mention the twolanequeue from knative in the message earlier. It is great, but it is not a drop-in replacement since their API is not compatible: knative has another API called EnqueueSlow. It also doesn’t have preemption(based on my understanding): If consumer queue is full of items from the slow lane, any items in fast lane after are still put in the end of the consumer queue. That is not my point for this thread, though.

I feel preemption needs to be done at one layer below (which is workqueue itself).

Thanks,

Haibing

On Thu, Feb 22, 2024 at 08:45 David Protasowski <david.pr...@broadcom.com> wrote:

If you haven't seen Knative has implemented a two-lane queue in our controller framework - here's the original issue https://github.com/knative/pkg/issues/1505

It's meant as a drop in replacement for `workqueue.RateLimitingInterface` and the big idea is that certain events by default fall on the slow queue - eg. resync.

--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/9b62d5f6-b323-48a4-94eb-7559d3ed8been%40googlegroups.com.

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

[David Protasowski]

> Knative has another API called EnqueueSlow.

Yeah I think it's important to surface something that lets consumers of the API enqueue something with lower or higher priority.

> It also doesn’t have preemption(based on my understanding): If consumer queue is full of items from the slow lane, any items in fast lane after are still put in the end of the consumer queue. That is not my point for this thread, though.

Yeah this is a bit of quirk - I'd be speculating on why it was written this way - eg. avoids starvation, keys can be de-duped.

[David Protasowski]

If you haven't seen Knative has implemented a two-lane queue in our controller framework - here's the original issue https://github.com/knative/pkg/issues/1505

It's meant as a drop in replacement for `workqueue.RateLimitingInterface` and the big idea is that certain events by default fall on the slow queue - eg. resync.

On Thu, Feb 22, 2024 at 11:33 AM Haibing Zhou <zhouhai...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/9b62d5f6-b323-48a4-94eb-7559d3ed8been%40googlegroups.com.