Re: [kubernetes/kubernetes] Flaky Tests: [sig-storage] 5000-node correctness tests (#66376)

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Issue Needs Approval

@shyamjvs @kubernetes/sig-storage-misc

Action required: This issue must have the status/approved-for-milestone label applied by a SIG maintainer. If the label is not applied within 4 days, the issue will be moved out of the v1.12 milestone.

Issue Labels

• sig/storage: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/bug: Fixes a bug discovered during the current release.

Help

• Additional instructions
• Commands for setting labels

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[Michelle Au]

Volume expansion tests have been fixed by #66324

[Michelle Au]

/assign @davidz627
can you investigate the remaining failures?

[Kubernetes Submit Queue]

[MILESTONENOTIFIER] Milestone Issue Needs Approval

@davidz627 @shyamjvs @kubernetes/sig-storage-misc

Action required: This issue must have the status/approved-for-milestone label applied by a SIG maintainer. If the label is not applied within 4 days, the issue will be moved out of the v1.12 milestone.

Issue Labels

• sig/storage: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/bug: Fixes a bug discovered during the current release.

Help

• Additional instructions
• Commands for setting labels

—

[Kubernetes Submit Queue]

Closed #66376 via #66405.

[Shyam JVS]

Reopening this issue, as a bunch of storage-related e2es started flaking in our large-scale jobs recently:

• https://k8s-testgrid.appspot.com/sig-scalability-gce#gce-large-correctness
• https://k8s-testgrid.appspot.com/sig-scalability-gce#gce-scale-correctness

/reopen

[k8s-ci-robot]

Reopened #66376.

[Michelle Au]

I think the latest failures are caused by gce API rate limiting: #67348

/assign @bowei @jingax10

[Michelle Au]

It seems to only be failing on scalability test suites. I wonder if it's due to some different cidr configuration

[Tim Pepper]

/sig scalability

[guineveresaenger]

/kind failing-test

[Shyam JVS]

/status approved-for-milestone

Friendly ping. This is affecting release-blocking scalability job.

[Davanum Srinivas]

hey @shyamjvs what needs to be done here? by whom?

[Shyam JVS]

@michelle192837 Could you comment on what needs to be done here and by whom?

[Michelle Au]

@bowei @jingax10 can you guys take a look at the API quota issue? This is happening consistently in the scale test environment