2026-05-13 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2026-05-13 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: dhiller, nirdothan, dywhite

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• 
  

• ci-health

• 

• 

• all external failures are

• either quay.io service failures (502)

• or github.com bazelbuild download failures (502)

• review and merge the quarantine PRs in 2 working days without SIG lgtm. 

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+quarantine+-label%3Ado-not-merge%2Fwork-in-progress+label%3Akind%2Fflake+-label%3Aneeds-rebase

• n/a

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• n/a

• [non-urgent]

• [dhiller] quarantined test lifecycle (PR)

• quarantine tracker issues per SIG:

• sig-compute: https://github.com/kubevirt/kubevirt/issues/17720

• sig-observability: https://github.com/kubevirt/kubevirt/issues/17721

• sig-network: https://github.com/kubevirt/kubevirt/issues/17722

• sig-storage: https://github.com/kubevirt/kubevirt/issues/17723

• [dhiller] re: last weeks periodic-kubevirt-e2e-k8s-1.36-sig-storage ( ∑=64, 5.21% )

• datavolume migration test seems to be failing a lot recently: example: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.36-sig-storage/2051443091449057280

• (new issue) https://github.com/kubevirt/kubevirt/issues/17724

• Look at flakes

• most flaky tests

• 
  

• flake stats - create issues accordingly

• overall ( ∑=1207, 100.00% )

• periodic-kubevirt-e2e-k8s-1.36-sig-compute ( ∑=348, 28.83% )

• VSOCK fixed, but virtiofs test failing every time (only on 1.36)

• (new issue) https://github.com/kubevirt/kubevirt/issues/17792

• periodic-kubevirt-e2e-test-S390X ( ∑=153, 12.68% )

• live migration test very flaky, also many others having a high rate of flakiness

• (existing issue) https://github.com/kubevirt/kubevirt/issues/17714

• new comment: https://github.com/kubevirt/kubevirt/issues/17714#issuecomment-4439342581

• periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=134, 11.10% )

• majority due to q’d tests, also some tests show minor flakiness levels

• periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=126, 10.44% )

• majority due to q’d tests, also some tests show minor flakiness levels

• periodic-kubevirt-e2e-k8s-1.36-sig-storage ( ∑=66, 5.47% )

• 

• unusual flakiness on 1.36 lane

• (new issue) https://github.com/kubevirt/kubevirt/issues/17791

• periodic-kubevirt-e2e-k8s-1.34-sig-performance-kwok-100 ( ∑=61, 5.05% )

• (existing issue) https://github.com/kubevirt/kubevirt/issues/17716

• periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=50, 4.14% )

• pull-kubevirt-e2e-k8s-1.35-sig-operator ( ∑=46, 3.81% )

• periodic-kubevirt-e2e-k8s-1.36-sig-operator ( ∑=44, 3.65% )

• periodic-kubevirt-e2e-k8s-1.35-sig-storage ( ∑=41, 3.40% )

• periodic-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=34, 2.82% )

• pull-kubevirt-e2e-k8s-1.35-sig-compute-serial ( ∑=20, 1.66% )

• pull-kubevirt-e2e-kind-1.35-sig-compute-arm64 ( ∑=15, 1.24% )

• pull-kubevirt-e2e-k8s-1.35-sig-compute-migrations ( ∑=14, 1.16% )

• 
  

• Look at held tests:

• https://grafana.ci.kubevirt.io/d/uAoSeksSk/referee-retests?orgId=1&refresh=15m&from=now-1h&to=now

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+label%3Aapproved+label%3Algtm+-label%3Ado-not-merge%2Fhold+-label%3Aneeds-rebase

• dequarantine tests:

• look at list of quarantined tests

• Count: 18 tests in quarantine currently (one network test not showing up)
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests

• misc topics

• 
  

Action items

• 
  
• 

  update/create issues with latest flakes spotted

• 

  communication

• send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)