2026-03-18 sig-ci quarantine catch-up - notes
Daniel Hiller
When: Weekly on Wed, 9:45 – 10:15am
Notes: KubeVirt CI SIG meeting notes
Attendees: dollierp, dhiller, nirdothan
Reminders:
we will create GitHub issues for tracking
GitHub issues and PRs
should be marked with /sig ci and /kind flake if applicable
should be marked with the target sig
Topics:
[urgent]
[dollierp] vGPU jobs are failing when scheduled on node bm11:
Issue seems to have started yesterday around 14h00 (UTC) although nothing should have changed about this node
Nothing clear/obvious from the logs on the node (dmesg + journal)
Will try to reset the PCI device and reboot the node if it does not help
[ycui] can we move it to the optional first?
cordon node, then no job will run on it.
[nirdothan] please approve https://github.com/kubevirt/kubevirtci/pull/1647
[dhiller] will take a look today
[ycui] Let’s review and merge the Quarantining case in 2 working days without SIG lgtm.
https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+quarantine
[dhiller] auto-quarantine needs a fix, need to work on that first
previous action items
state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci
[non-urgent]
[nirdothan] requesting review of https://github.com/kubevirt/kubevirtci/pull/1645
enabling interconnection between different nodes basically
[dhiller] will take a look
Look at flakes
flake stats - create issues accordingly
overall ( ∑=3119, 100.00% )
periodic-kubevirt-e2e-k8s-1.33-sig-compute ( ∑=589, 18.88% )
still not recovering from clustered failures, sig-compute needs to take a look
[xpivarc] to bring it to the team
periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=492, 15.77% )
recovering from clustered failures
pull-kubevirt-e2e-k8s-1.33-sig-compute ( ∑=492, 15.77% )
one huge clustered failure responsible for the majority of test failures: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/directory/pull-kubevirt-e2e-k8s-1.33-sig-compute/2031670244539371520
periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=446, 14.30% )
recovering from clustered failures
pull-kubevirt-e2e-k8s-1.35-sig-compute-serial ( ∑=365, 11.70% )
lane stability has decreased rapidly starting on 2026-03-13
sig-compute, please check the lane
periodic-kubevirt-e2e-test-S390X ( ∑=118, 3.78% )
pull-kubevirt-e2e-k8s-1.34-sig-compute-serial ( ∑=65, 2.08% )
periodic-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=60, 1.92% )
periodic-kubevirt-e2e-k8s-1.35-sig-storage ( ∑=55, 1.76% )
periodic-kubevirt-e2e-k8s-1.33-sig-storage ( ∑=53, 1.70% )
periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=48, 1.54% )
pull-kubevirt-e2e-k8s-1.33-sig-operator ( ∑=47, 1.51% )
pull-kubevirt-e2e-k8s-1.35-sig-operator ( ∑=44, 1.41% )
pull-kubevirt-e2e-kind-1.35-sig-compute-arm64 ( ∑=44, 1.41% )
periodic-kubevirt-e2e-k8s-1.35-sig-compute-migrations ( ∑=33, 1.06% )
Last updated: 2026-03-18 08:00:14.270246893 +0000 UTC
Look at held tests:
dequarantine tests:
look at list of quarantined tests
count: 18 tests in quarantine currently - no change to last week
check status, i.e. who is working on those
look at PRs that want to fix flakes
see whether we can dequarantine tests
misc topics
Action items
update quarantine query: https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+quarantine+-label%3Ado-not-merge%2Fwork-in-progress+label%3Akind%2Fflake+-label%3Aneeds-rebase
update/create issues with latest flakes spotted
communication
send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)
Kind regards,
Daniel Hiller
He / Him / His
Principal Software Engineer, KubeVirt CI, OpenShift Virtualization
 |
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty
Lee Yarwood
Lee