2026-01-14 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2026-01-14 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: dhiller, Igor Bezukh Federico Fossemo Nir Dothan

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• 
  

• ci-health

• 

• 

• sig-ci failures are caused by 1.5 403 download failure from gcs by bazel - probably a glitch on the gcs storage service

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• [non-urgent]

• [fossedihelm] which 1.35 provider version are we using currently?

• [dhiller] should be 1.35.0, however 1.35 publishing has failed a couple times, need to investigate what’s going on, probably start the publish manually again

• Look at flakes

• flake stats - create issues accordingly

• overall ( ∑=753, 100.00% )

• periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=253, 33.60% )

• (146 failures) [fossedihelm] three tests that are failing called “ContainerDisk Simulate an upgrade” - sync of target is failing, failed to find image, working on it

• (existing) https://github.com/kubevirt/kubevirt/issues/15976

• remaining ones attributed to quarantined test runs

• periodic-kubevirt-e2e-k8s-1.33-sig-compute ( ∑=194, 25.76% )

• clustered failure (106): https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.33-sig-compute/2008715372127064064

• (new) https://github.com/kubevirt/kubevirt/issues/16539

• remaining ones attributed to quarantined test runs and mild flakes

• periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=82, 10.89% )

• one clustered failure

• (existing) https://github.com/kubevirt/kubevirt/issues/16275#issuecomment-3701830107

• remaining ones attributed to quarantined test runs and mild flakes

• periodic-kubevirt-e2e-k8s-1.32-sig-compute ( ∑=75, 9.96% )

• two old clustered failures

• remaining ones attributed to quarantined test runs and mild flakes

• pull-kubevirt-e2e-k8s-1.33-sig-operator ( ∑=40, 5.31% )

• clustered failure (40): https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/directory/pull-kubevirt-e2e-k8s-1.33-sig-operator/2009262751758684160

• (new) https://github.com/kubevirt/kubevirt/issues/16538

• periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=13, 1.73% )

• failures attributed to quarantined test runs and mild flakes 

• pull-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=8, 1.06% )

• failures attributed to mild flakes

• 
  

• dequarantine tests:

• look at list of quarantined tests

• count: tests in quarantine currently
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests 

• misc topics

• [nir] windows PR - problem should have been fixed, need to reopen original PR and then upgrade qemu

• [fosseduhelm] there should be two open PRs bumping qemu already, let’s use those

Action items

• 
  
• 

  update/create issues with latest flakes spotted

• 

  communication

• 

  send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty