2025-12-10 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2025-12-10 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: dollierp, dhiller, fossedihelm

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• 
  

• ci-health

• 

• 

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• [non-urgent]

• [dhiller] peribolos org updater was failing due to default branch misconfiguration: https://github.com/kubevirt/project-infra/issues/4575 - fixed now

• 
  

• Look at flakes

• flake stats - create issues accordingly

• count: 1000 failures overall

• 345 periodic-kubevirt-e2e-k8s-1.35-sig-compute

• (existing) https://github.com/kubevirt/kubevirt/issues/15976

• 179 periodic-kubevirt-e2e-k8s-1.34-sig-compute

• 84 attributed to a clustered failure: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.34-sig-compute/1995686986697936896

• (existing) https://github.com/kubevirt/kubevirt/issues/16275

• 117 periodic-kubevirt-e2e-k8s-1.32-sig-compute

• 100+ failures attributed to q’d tests

• 102 pull-kubevirt-e2e-k8s-1.34-sig-compute-serial

• three clustered failures accounting to the majority of failures

• PR https://github.com/kubevirt/kubevirt/pull/16073

• 52 https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/directory/pull-kubevirt-e2e-k8s-1.34-sig-compute-serial/1998578335822647296

• 45 https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/directory/pull-kubevirt-e2e-k8s-1.34-sig-compute-serial/1998220906551316480

• PR https://github.com/kubevirt/kubevirt/pull/15885

• 21 https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/directory/pull-kubevirt-e2e-k8s-1.34-sig-compute-serial/1998514334606561280

• PR authors and reviewers: please make sure that this is not a newly introduced instability!

• 92 periodic-kubevirt-e2e-k8s-1.33-sig-compute

• ~85 accounting towards q’d tests

• 26 periodic-kubevirt-e2e-k8s-1.34-sig-compute-np

• new: clustered failure https://github.com/kubevirt/kubevirt/issues/16320

• [dhiller] IIRC we agreed on enabling np for providers and versions that support it in the lanes and then remove the separate lanes?

• [fossedihelm] we released 1.7, so we should be good to enable this for the operator tests, who were still failing recently

• in the long run we will enable this for providers and releases supporting this, then we are good to drop the separate lanes

• [dhiller] sig-storage q PR: https://github.com/kubevirt/kubevirt/pull/16312

• cross live migration - since Alexander is OOO until Jan pinged Adam

• dequarantine tests:

• look at list of quarantined tests

• 13: tests in quarantine currently
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests 

• misc topics

• [fossedihelm] is it ok for the q tracker issues to get closed automatically?

• [dhiller] if we want to leave it open we can put /lifecycle frozen on it if need be, however when I encounter issues getting closed, I check whether the tests are still quarantined, and then reopen them. One of the issues I encountered yesterday was actually outdated, since the test was taken out of quarantine

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty