2025-11-26 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2025-11-26 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: nirdothan, brianmcarey, 

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• [bc] quarantine PR for live migration across ns: https://github.com/kubevirt/kubevirt/pull/16226

• in merge queue currently

• ci-health

• 

• 

• sig-failures

• AI summary

• sig-ci

• AI summary

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• [non-urgent]

• [dh per 0xfelix] windows2016 test fails when bumping qemu

• https://github.com/kubevirt/kubevirt/issues/15904

• update: there’s two issues that were found

• 1. the nfs server is outdated and needs to get updated - it reports wrong state about the windows image

• https://github.com/kubevirt/kubevirtci/issues/1578

• 2. there’s a regression in qemu that needs to get fixed

• https://github.com/kubevirt/kubevirt/issues/15904#issuecomment-3558224137

• more info to follow

• [dh] we would need to upgrade qemu, therefore we need to q those tests, since they are blocking the upgrade

• intel tdx relies on the upgrade

• security patches are not able to be brought in

• bc to take a look at whats going on with nfs

• Look at flakes

• flake stats - create issues accordingly

• count: 1049 failures overall

• apart from 1.35 lane failing the largest offenders are the periodic sig-compute lanes

• 333 failures periodic-kubevirt-e2e-k8s-1.35-sig-compute

• persisting issues as discussed in previous meetings

• 124 failures periodic-kubevirt-e2e-k8s-1.34-sig-compute

• 80% are caused by quarantined tests, remainder is caused by an older clustered failure

• 110 failures periodic-kubevirt-e2e-k8s-1.33-sig-compute

• 90+% are caused by quarantined tests

• 96 failures periodic-kubevirt-e2e-k8s-1.32-sig-compute

• 90+% are caused by quarantined tests

• 57 failures pull-kubevirt-e2e-k8s-1.34-sig-compute-serial

• 90% caused by a clustered failure, added to existing issue

• dequarantine tests:

• look at list of quarantined tests

• count: tests in quarantine currently
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• 

• see whether we can dequarantine tests 

• fossedihelm removed two tests from quarantine

• https://github.com/kubevirt/kubevirt/pull/16154

• https://github.com/kubevirt/kubevirt/pull/16153

• misc topics

• [bc] updates to k8s-1.35 kubernetes version blocked by failing kubernetes conformance tests 

• https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirtci/1576/check-provision-k8s-1.35/1992939314803118080 

Action items

• 
  
• 

  update/create issues with latest flakes spotted

• 

  communication

• send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty