2026-04-01 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2026-04-01 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: Dylan White, dhiller, dollierp, ycui, fossedihelm, xpivarc

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• [ycui] 3 storage tests failing often? https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/16853/pull-kubevirt-e2e-k8s-1.35-sig-storage/2038936349674835968

• https://storage.googleapis.com/kubevirt-prow/reports/most-flaky-tests/kubevirt/kubevirt/index.html#index

• https://prow.ci.kubevirt.io/?job=*auto-q*

• Not see the failures in the flaky report. 

• [dhiller] https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flake-stats-14days-2026-04-01.html doesn’t show them actually, need to check

• Enhancement: https://github.com/kubevirt/project-infra/issues/4906

• [dhiller] so we do want to change the quarantine rules for 3 days?

• No, just want to show the warning if the cases failure > 10% 

• ci-health

• 

• 

• review and merge the quarantine PRs in 2 working days without SIG lgtm. 

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+quarantine+-label%3Ado-not-merge%2Fwork-in-progress+label%3Akind%2Fflake+-label%3Aneeds-rebase

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• [non-urgent]

• 
  

• Look at flakes

• most flaky tests

• 
  

• flake stats - create issues accordingly

• overall ( ∑=1434, 100.00% )

• periodic-kubevirt-e2e-test-S390X ( ∑=738, 51.46% )

• several s390x tests failing every time, need to notify wg-arch-s390x on that

• (new) https://github.com/kubevirt/kubevirt/issues/17371

• periodic-kubevirt-e2e-k8s-1.33-sig-compute ( ∑=117, 8.16% )

• this is a lane that has been deleted due to next release targeting 1.34-1.36, so it’s irrelevant

• periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=68, 4.74% )

• periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=59, 4.11% )

• periodic-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=51, 3.56% )

• periodic-kubevirt-e2e-k8s-1.35-sig-storage ( ∑=47, 3.28% )

• periodic-kubevirt-e2e-k8s-1.33-sig-storage ( ∑=44, 3.07% )

• pull-kubevirt-e2e-k8s-1.34-sig-operator ( ∑=44, 3.07% )

• periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=42, 2.93% )

• pull-kubevirt-e2e-k8s-1.35-sig-compute-serial ( ∑=32, 2.23% )

• pull-kubevirt-e2e-kind-1.35-sig-compute-arm64 ( ∑=21, 1.46% )

• pull-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=19, 1.32% )

• pull-kubevirt-e2e-k8s-1.35-sig-compute-migrations ( ∑=15, 1.05% )

Last updated: 2026-04-01 07:39:08.290956281 +0000 UTC m=+4.962316024

• Look at held tests:

• https://grafana.ci.kubevirt.io/d/uAoSeksSk/referee-retests?orgId=1&refresh=15m&from=now-1h&to=now

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+label%3Aapproved+label%3Algtm+-label%3Ado-not-merge%2Fhold+-label%3Aneeds-rebase

• dequarantine tests:

• look at list of quarantined tests

• Count: 18 tests in quarantine currently
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests

• misc topics

• [fossedihelm] Are 1.36 lanes already available? (missed recent updates, so this could be dumb q)

• https://github.com/kubevirt/kubevirt/pull/17361 1.36 beta bump

Action items

• 

  Daniel Hiller need to investigate why retest-required is not working: https://github.com/kubevirt/kubevirt/pull/17361

• 

  update: retest-required is only working for required checks that are actually reported on a commit - since the GitHub had issues those were missing on the commit, however from branch-protection (which required pull requests are added to) they are reported as required and pending, which is what the github ui is showing.

• 

  update/create issues with latest flakes spotted

• 

  communication

• send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty