2026-03-04 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2026-03-04 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: dhiller, dollierp, Dylan White

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• [dhiller] got a stuck publish-kubevirtci that we need to investigate

• https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/publish-kubevirtci/2028499263591288832

• blocking the kubevirtci bump on kubevirt

• we need to at least store some logs to find the root cause

• https://github.com/kubevirt/kubevirtci/pull/1636 should mitigate the issue

• [dollierp] Postponing Prow Control Plane upgrade to OCP-v4.20 for one week again

• ci-health

• 

• 

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• [non-urgent]

• [dhiller] code-freeze: restrict main branch merges to only allow approved-vep labelled PRs during code freeze

• [xpivarc] freeze-exception

• [dhiller] implement in 1.9

• probably unconference topic, maybe even for release retrospective

• will raise in community meeting

• Look at flakes

• most flaky tests

• note: related to clustered failures below - there’s a kubevirtci bump pending for syncing the cri-o version to the k8s version in 1.35 which is highly likely to improve the cluster stability

• publish job succeeded: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/publish-kubevirtci/2029133610103083008

• bump updated: https://github.com/kubevirt/kubevirt/pull/16972

• s390x issue

• https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flake-stats-14days-2026-03-04.html#%5bsig-compute%5dVM%20Affinity%20Updating%20VMs%20node%20affinity%20%5btest_id%3a11208%5dshould%20successfully%20update%20node%20selector

• (new) https://github.com/kubevirt/kubevirt/issues/17021

• 1.35 clustered failures:

• add as context prometheus

• https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flake-stats-14days-2026-03-04.html#%5bsig-compute%5d%20Infrastructure%20%5brfe_id%3a3187%5d%5bcrit%3amedium%5d%5bvendor%3acnv-qe%40redhat.com%5d%5blevel%3acomponent%5dPrometheus%20Endpoints%20should%20include%20metrics%20for%20a%20running%20VM%20%5btest_id%3a4556%5d%20vmi%20unused%20memory 

• above doesn’t make sense since it’s a follow up error

• flake stats - create issues accordingly

• overall ( ∑=3464, 100.00% )

• pull-kubevirt-e2e-k8s-1.35-sig-compute-serial ( ∑=990, 28.58% )

• (new) https://github.com/kubevirt/kubevirt/issues/17022

• periodic-kubevirt-e2e-k8s-1.33-sig-compute ( ∑=660, 19.05% )

• clustered failures

• periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=588, 16.97% )

• clustered failures

• periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=481, 13.89% )

• clustered failures

• pull-kubevirt-e2e-k8s-1.34-sig-compute-serial ( ∑=211, 6.09% )

• (existing) 1.34: https://github.com/kubevirt/kubevirt/issues/16688

• periodic-kubevirt-e2e-test-S390X ( ∑=87, 2.51% )

• (new) https://github.com/kubevirt/kubevirt/issues/17021

• periodic-kubevirt-e2e-k8s-1.35-sig-storage ( ∑=53, 1.53% )

• periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=51, 1.47% )

• periodic-kubevirt-e2e-k8s-1.34-sig-operator ( ∑=42, 1.21% )

• pull-kubevirt-e2e-k8s-1.35-sig-operator ( ∑=41, 1.18% )

• periodic-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=40, 1.15% )

• periodic-kubevirt-e2e-k8s-1.33-sig-storage ( ∑=36, 1.04% )

• dequarantine tests:

• look at list of quarantined tests

• count: tests in quarantine currently
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests 

• misc topics

• 
  

Action items

• 
  
• 

  update/create issues with latest flakes spotted

• 

  communication

• send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty