--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
The resync period does more to compensate for problems in the controller code than to compensate for missed watch event. We have had very few "I missed an event bug" (I can think of one in recent memory), but we have had many controllers which use the resync as a big "try again" button.A resync is different than a relist. The resync plays back all the events held in the informer cache. A relist hits the API server to re-get all the data.Since we introduced the rate limited work queue a few releases ago, the need to wait for a resync to retry has largely disappeared since an error during processing gets requeued on an incrementing delay.Think of the resync as insurance. You probably want it set more than a few minutes, less than a few hours. If you're using requeuing correctly and avoiding panics, you aren't likely to benefit from rescanning all the data very often.
On Thu, Sep 7, 2017 at 5:24 AM, 'Timo Reimann' via K8s API Machinery SIG <kubernetes-sig-api-machinery@googlegroups.com> wrote:
Hello,I have a question regarding building controllers with the help of the informer framework (using Go, if that matters). Specifically, every informer must be given a resync period that defines the frequency by which all watched objects get a chance to be revisited by means of a full list. To my understanding, this is to guard against transient errors on the watch part of the ListWatch interface (though I'm not perfectly clear if this means one of client-side or server-side errors only).Recommendations found in the wild as to the right resync period seem to gravitate towards "keep it rather high to avoid burdening the API server too much", and I often see values on the order of multiple minutes or even hours. However, this advice seems to render the resync period unsuitable as a means to compensate for watch errors for somewhat latency-critical controllers.To add a real-world example: I am working on the Traefik Ingress controller which watches over a user-provided list of namespaces. For each namespace, Traefik retrieves Ingress, Service, and Endpoints objects to build up the routing table. Missing one of the objects means we cannot create a proper forwarding rule for the affected application(s) until the situation is remedied. So my tendency is to pick a rather low resync period (preferably no more than a minute) to make sure the routing gets fixed rather quickly, but that seems contrary to the common advice of what I should be doing.I guess my questions boil down to the following:1. Am I missing the point of the resync period somehow?2. Are there any guidelines on how to choose the right value?3. Is there maybe a way to measure the performance impact it has on the API server?Thanks in advance for any advice, suggestions, and help in general.Timo
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
When* you are writing a controller which interacts both with Kubernetes and some other outside system (like, say, some cloud's load balancer API), and* the outside system doesn't offer a watchable interface, so* you intend to poll the outside system (to make sure it stays correct / repair it), thenIt makes sense to use resync to trigger the polling behavior, and the resync period can be chosen based on how often you want to reconcile....you just can't forget to also have a system that periodically finds all the objects you've created in the past but need to be deleted (i.e., don't leak).
On Thu, Sep 7, 2017 at 5:08 AM, David Eads <de...@redhat.com> wrote:
The resync period does more to compensate for problems in the controller code than to compensate for missed watch event. We have had very few "I missed an event bug" (I can think of one in recent memory), but we have had many controllers which use the resync as a big "try again" button.A resync is different than a relist. The resync plays back all the events held in the informer cache. A relist hits the API server to re-get all the data.Since we introduced the rate limited work queue a few releases ago, the need to wait for a resync to retry has largely disappeared since an error during processing gets requeued on an incrementing delay.Think of the resync as insurance. You probably want it set more than a few minutes, less than a few hours. If you're using requeuing correctly and avoiding panics, you aren't likely to benefit from rescanning all the data very often.
On Thu, Sep 7, 2017 at 5:24 AM, 'Timo Reimann' via K8s API Machinery SIG <kubernetes-sig...@googlegroups.com> wrote:
Hello,I have a question regarding building controllers with the help of the informer framework (using Go, if that matters). Specifically, every informer must be given a resync period that defines the frequency by which all watched objects get a chance to be revisited by means of a full list. To my understanding, this is to guard against transient errors on the watch part of the ListWatch interface (though I'm not perfectly clear if this means one of client-side or server-side errors only).Recommendations found in the wild as to the right resync period seem to gravitate towards "keep it rather high to avoid burdening the API server too much", and I often see values on the order of multiple minutes or even hours. However, this advice seems to render the resync period unsuitable as a means to compensate for watch errors for somewhat latency-critical controllers.To add a real-world example: I am working on the Traefik Ingress controller which watches over a user-provided list of namespaces. For each namespace, Traefik retrieves Ingress, Service, and Endpoints objects to build up the routing table. Missing one of the objects means we cannot create a proper forwarding rule for the affected application(s) until the situation is remedied. So my tendency is to pick a rather low resync period (preferably no more than a minute) to make sure the routing gets fixed rather quickly, but that seems contrary to the common advice of what I should be doing.I guess my questions boil down to the following:1. Am I missing the point of the resync period somehow?2. Are there any guidelines on how to choose the right value?3. Is there maybe a way to measure the performance impact it has on the API server?Thanks in advance for any advice, suggestions, and help in general.Timo
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/1615adf8-cc0f-47af-8c27-d5fe705f7109%40googlegroups.com.
I believe we have quashed the bugs and insurance is not necessary or desirable any more.
I believe we have quashed the bugs and insurance is not necessary or desirable any more.I think there is value in tests to make sure the relisting code gets exercised, since networks do occasionally drop connections at inconvenient times.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/1615adf8-cc0f-47af-8c27-d5fe705f7109%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bbmM_bvc6R887WkLv-fdGOLxT%2BTZc_v_icAGbaaSJ-3dQ%40mail.gmail.com.
I believe we have quashed the bugs and insurance is not necessary or desirable any more.We've still been catching bugs in this space, but they've been mostly in the controllers. The PV controller relying on resync comes to mind. The insurance is largely for the controller code, not so much the list/watch (though we did find a bug in the current release there too).
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAB_J3bbmM_bvc6R887WkLv-fdGOLxT%2BTZc_v_icAGbaaSJ-3dQ%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjK_FsmkiEu2g_WGo__AxJ3o%3DbrhmeSL_nBH6Bquip%2BBSg%40mail.gmail.com.
I think there is value in tests to make sure the relisting code gets exercised, since networks do occasionally drop connections at inconvenient times.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
On Friday, September 8, 2017 at 6:33:07 PM UTC+2, lavalamp wrote:I think there is value in tests to make sure the relisting code gets exercised, since networks do occasionally drop connections at inconvenient times.Not exactly sure I'm understanding this point: Does it mean that I (as the controller implementor) am responsible for issuing a relist if I detect a connection issue with my client? In particular, does that hold for client-go?
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsubs...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/1615adf8-cc0f-47af-8c27-d5fe705f7109%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-api-machinery@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/3d4385ba-7522-4780-85cd-d13eeb59d623%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-machinery+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/6e446397-4fb6-4c95-afaa-97f8920fd95a%40googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/303e649c-b72d-43ef-b16e-e3e604f18964%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/8D326120-2918-4BCE-9BDF-E246701AD6E6%40google.com.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/c24cfffc-bc0a-4042-8631-26a2dcd40a61%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAFS1MjJOMzTBHNne2Qsq%3D1RtGyDHRjFN9YCVxT2VKYFsdWkEWg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/1615adf8-cc0f-47af-8c27-d5fe705f7109%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/6e446397-4fb6-4c95-afaa-97f8920fd95a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/303e649c-b72d-43ef-b16e-e3e604f18964%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/8D326120-2918-4BCE-9BDF-E246701AD6E6%40google.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "K8s API Machinery SIG" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-api-m...@googlegroups.com.
To post to this group, send email to kubernetes-sig...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-api-machinery/CAH16Sh%2B%3DqSz6WECONnzaQ_c%2Bm8p26BHycfUyzj2aoJ5bX0_N0g%40mail.gmail.com.