2026-07-22 sig-ci quarantine catch-up - notes
Daniel Hiller
When: Weekly on Wed, 9:45 – 10:15am
Notes: KubeVirt CI SIG meeting notes
Attendees: dhiller, dollierp, Nir Dothan
TL;DR we are aware of the KubeVirt CI cluster-up issue: tracker is https://github.com/kubevirt/kubevirtci/issues/1783
Reminders:
we will create GitHub issues for tracking
GitHub issues and PRs
should be marked with /sig ci and /kind flake if applicable
should be marked with the target sig
Topics:
[urgent]
[dollierp] s390x workloads clusters are down since this morning
stakeholders have been notified
[dhiller] we should do always_run: false
[dollierp] centos stream 9 bump
lot of occurrences of jobs failing to create clusters and ssh connection giving up after ~55m since ~5 days: query
[dhiller] will open an issue and see where we can get further insights
options: revert to old image, then we aren’t able to rebuild the base image any more as it’s gone from the mirror already
[dollierp] sig-compute-serial lane extended duration
mitigation on its way to get merged: PR#18523
[dollierp] sig-operator lane flakiness status
Lee Yarwood - IMHO it might be quicker to drop these from quarantine and allow us to collect more etcd data from failures, a 5% hit rate is extremely awkward to hit locally or in DNM retest runs.
[dhiller] one option to run a set of tests is /test-subset: https://prow.ci.kubevirt.io/command-help#test_subset
[dhiller] also change tests and run dequarantine (runs 500 times): https://github.com/kubevirt/kubevirt/blob/main/docs/quarantine.md#getting-tests-out-of-quarantine
release-blocker issues: issue#18465, issue#18490: label needs to be added by root approvers
[dhiller] suggestion: add sig-release and sig-ci as well to the group of people that can put release-blocker label on
g.a. release is planned for today
https://github.com/kubevirt/kubevirt/pull/18397 is ready for more review
https://github.com/kubevirt/kubevirt/pull/18355 is on hold pending more data being collected from https://github.com/kubevirt/kubevirt/pull/18487 or from actual CI runs if we remove from quarantine
review and merge the quarantine PRs in 2 working days without SIG lgtm.
previous action items
state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci
[non-urgent]
Look at flakes
flake stats - create issues accordingly
overall ( ∑=902, 100.00% )
periodic-kubevirt-e2e-test-S390X ( ∑=420, 46.56% )
[dhiller] will take a look
periodic-kubevirt-e2e-k8s-1.36-sig-compute ( ∑=77, 8.54% )
periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=73, 8.09% )
periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=65, 7.21% )
periodic-kubevirt-e2e-k8s-1.34-sig-performance-kwok ( ∑=32, 3.55% )
periodic-kubevirt-e2e-k8s-1.34-sig-performance-kwok-100 ( ∑=32, 3.55% )
periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=26, 2.88% )
pull-kubevirt-e2e-k8s-1.36-sig-compute-migrations ( ∑=24, 2.66% )
pull-kubevirt-check-tests-for-flakes ( ∑=20, 2.22% )
periodic-kubevirt-e2e-kind-1.36-sev ( ∑=14, 1.55% )
pull-kubevirt-e2e-k8s-1.36-sig-operator ( ∑=12, 1.33% )
Last updated: 2026-07-22 07:11:15.191598871 +0000 UTC m=+0.899788804
Look at held tests:
dequarantine tests:
look at list of quarantined tests
Count: 13 tests in quarantine currently
Lee Yarwood - No sig-operator count? This should likely also be sig-control-plane moving forward.
check status, i.e. who is working on those
look at PRs that want to fix flakes
see whether we can dequarantine tests
misc topics
Action items
update/create issues with latest flakes spotted
communication
send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)
Kind regards,
Daniel Hiller
He / Him / His
Principal Software Engineer, KubeVirt CI, OpenShift Virtualization
 |
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty