2026-05-06 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2026-05-06 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: dhiller,dywhite, 

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• [dhiller] we need to stabilize 1.36 periodic sig-compute lane, periodics still show a lot of failures around VSOCK

• screenshot shows 1.36 on the left and 1.35 on the right for comparison

• 

• issue: https://github.com/kubevirt/kubevirt/issues/17688

• fossedihelm is working on it

• pr https://github.com/kubevirt/kubevirt/pull/17690

• latest test-subset result is successful: https://github.com/kubevirt/kubevirt/pull/17690#issuecomment-4386015196

• ci-health

• 

• 

• review and merge the quarantine PRs in 2 working days without SIG lgtm. 

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+quarantine+-label%3Ado-not-merge%2Fwork-in-progress+label%3Akind%2Fflake+-label%3Aneeds-rebase

• n/a

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• n/a

• [non-urgent]

• 
  

• Look at flakes

• most flaky tests

• [sig-performance] Control Plane Performance Density Testing Density test [small] create a batch of 100 VMIs should successfully create all VMIS

• 🔎 💥 9% over 336h on

• pull-kubevirt-e2e-k8s-1.34-sig-performance: Failures: 27h0m0s 38h0m0s 44h0m0s

• Dylan is going to look into the latest failures

• flake stats - create issues accordingly

• overall ( ∑=1229, 100.00% )

• periodic-kubevirt-e2e-k8s-1.36-sig-compute ( ∑=379, 30.84% )

• VSOCK tests, see above

• periodic-kubevirt-e2e-test-S390X ( ∑=186, 15.13% )

• clustered failures are seen, overall seems to be stabilizing, need to check with wg-arch-s390x

• (new issue) https://github.com/kubevirt/kubevirt/issues/17714

• periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=131, 10.66% )

• seems to be stabilizing overall

• periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=123, 10.01% )

• seems to be stabilizing overall

• periodic-kubevirt-e2e-k8s-1.36-sig-storage ( ∑=64, 5.21% )

• datavolume migration test seems to be failing a lot recently: example: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.36-sig-storage/2051443091449057280

• periodic-kubevirt-e2e-k8s-1.34-sig-performance-kwok-100 ( ∑=61, 4.96% )

• (new issue) https://github.com/kubevirt/kubevirt/issues/17716

• periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=54, 4.39% )

• no suspicions

• periodic-kubevirt-e2e-k8s-1.34-sig-operator ( ∑=46, 3.74% )

• clustered failure: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.34-sig-operator/2047387619351859200

• (new issue) https://github.com/kubevirt/kubevirt/issues/17715

• periodic-kubevirt-e2e-k8s-1.35-sig-storage ( ∑=43, 3.50% )

• quarantined tests largely responsible 

• periodic-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=31, 2.52% )

• same as above

• periodic-kubevirt-e2e-k8s-1.35-ipv6-sig-network ( ∑=17, 1.38% )

• quarantined tests responsible

• 
  

• Look at held tests:

• https://grafana.ci.kubevirt.io/d/uAoSeksSk/referee-retests?orgId=1&refresh=15m&from=now-1h&to=now

• no action required

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+label%3Aapproved+label%3Algtm+-label%3Ado-not-merge%2Fhold+-label%3Aneeds-rebase

• two PR to be investigated afterwards

• dequarantine tests:

• look at list of quarantined tests

• Count: 17 tests in quarantine currently
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests

• misc topics

• [nirdothan] status of q test lifecycle discussion

• [dhiller] updated after feedback got in, need to chime in again to revive the discussion

• [dhiller] lubo suggested to have a collective q issue per sig for the test currently quarantined

Action items

• 

  Daniel Hiller fix recognition for QUARANTINE decorator: example: https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flake-stats-14days-2026-05-06.html#%5bsig-network%5d%20VirtualMachineInstance%20with%20passt%20network%20binding%20migration%20connectivity%20should%20be%20preserved%20%5bIPv6%5d https://github.com/kubevirt/project-infra/issues/5040

• 

  Daniel Hiller create collective q tracking issues per sig (lifecycle/frozen) for the test currently quarantined to aid in the transition before the q test lifecycle comes into action

• 

  update/create issues with latest flakes spotted

• 

  communication

• send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty