[kubernetes/kubernetes] TestClientGoCustomResourceExample flakes (#49956)

[Jordan Liggitt]

flaking on HEAD:
https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=ci-kubernetes-test-go&test=TestClientGoCustomResourceExample

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/8135#k8siokubernetesvendork8sioapiextensions-apiservertestintegration-testclientgocustomresourceexample

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/8118#k8siokubernetesvendork8sioapiextensions-apiservertestintegration-testclientgocustomresourceexample

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-test-go/8105#k8siokubernetesvendork8sioapiextensions-apiservertestintegration-testclientgocustomresourceexample

@kubernetes/sig-api-machinery-test-failures

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Jordan Liggitt]

these errors started occurring right after 74b9ba3 was merged

cc @deads2k @wojtek-t for oddities with watch, custom resources, and the watch cache

[Wojciech Tyczynski]

@smarterclayton

[Clayton Coleman]

Could this be interactions with the aggregator cache?

[Clayton Coleman]

E0728 21:35:49.536152   31354 customresource_discovery_controller.go:155] timed out waiting for caches to sync

Is that bad?

[Jordan Liggitt]

Could this be interactions with the aggregator cache?

no, TestClientGoCustomResourceExample tests a CRD server all alone, no aggregator involved

[Jordan Liggitt]

with lots of tracing added, I can reproduce:

controller lister/watcher:

1. lists before watch cache is ready, goes to etcd, gets empty list at resourceVersion=386
2. watches from resourceVersion=386 before watch is ready

client creates an object with resourceVersion 387

cacher lister/watcher:

1. lists, gets list with a single item at resourceVersion=387
2. watches etcd from resourceVersion=387

processEvent is never called with the 387 watch event. the next event it sees is the delete event at 388 when the test times out and is tearing itself down.

[Kubernetes Submit Queue]

Closed #49956 via #49992.