2025-11-12 sig-ci quarantine catch-up - notes
Daniel Hiller
When: Weekly on Wed, 9:45 – 10:15am
Notes: KubeVirt CI SIG meeting notes
Attendees: dhiller, brianmcarey, nirdothan
Reminders:
we will create GitHub issues for tracking
GitHub issues and PRs
should be marked with /sig ci and /kind flake if applicable
should be marked with the target sig
Topics:
[urgent]
sig-storage
[dhiller] probably caused by the network policy periodic lane
sig-monitoring
[bcarey] likely a k8s update was the reason, which didn’t trigger the run_if_changed
[dhiller] issue where we wanted to find the reason
sig-ci
tl;dr: large cluster of failures on one PR responsible for the majority of the number, related to missing bazel build file
previous action items
state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci
n/a
[non-urgent]
[ndothan] old issue windows2016 test fails when bumping qemu #15904
pinged felix and federico on the issue
Look at flakes
flake stats - create issues accordingly
count: 3529 failures overall
majority caused by earlier fallout of periodic network policy lanes – while those are fixed for a couple days already we still see the failures from before over the last 14 days
sig-compute
still failing periodic sig-compute 1.35: https://testgrid.k8s.io/kubevirt-periodics#periodic-kubevirt-e2e-k8s-1.35-sig-compute&width=20
pull 1.33 sig-operator clustered failures
new pull 1.32 sig-operator clustered failures
new pull-kubevirt-e2e-k8s-1.34-sig-compute-serial: clustered failure
new periodic-kubevirt-e2e-k8s-1.33-sig-compute: clustered failure
dequarantine tests:
look at list of quarantined tests
count: tests in quarantine currently
(insert screenshot from ci-health)check status, i.e. who is working on those
look at PRs that want to fix flakes
see whether we can dequarantine tests
misc topics
[fossedihelm] network policy lanes: is it worth it to add a presubmit that can be triggered manually
run_before_merge may make sense
[fosselihelm] long term install the network-policy by default
maybe have a subset of the tests to make sure that network policy works as expected
create a network-policy label that we attach to the subset
create a run_before_merge only for latest provider, run this for a fixed period to make sure nothing happens
the plan for now is to stick to the np periodics and after the ga set them to be installed by default, which will happen probably in two weeks
after the np install happens by default we can drop the periodics network policy lanes
Action items
Daniel Hiller create issue: update flake-stats to show lane failure summary sorted by number of failures descending: https://github.com/kubevirt/project-infra/issues/4511
update/create issues with latest flakes spotted
communication
send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)
Kind regards,
Daniel Hiller
He / Him / His
Principal Software Engineer, KubeVirt CI, OpenShift Virtualization
 |
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty