kubevirt-dev] 2026-07-27 sig-ci weekly meeting - notes

[doll...@redhat.com]

2026-07-27 sig-ci weekly meeting

When: Weekly on Mon, 10:00 – 10:30am

Notes: KubeVirt CI SIG meeting notes

Attendees: dollierp, Orel Misan, Ananya Banerjee, dywhite, Igor Bezukh

Reminders:

• we will create GitHub issues for tracking
• GitHub issues and PRs
• should be marked with /sig ci and /kind flake if applicable
• should be marked with the target sig

Topics:

• [urgent]
• [dollierp] CI testing clusters start up failures: [kubevirtci] issue#1783
• [dhiller] investigated resources starvation: [kubevirtci] PR#1784
• seems like it is a wrong lead
• [dollierp] investigated the qemu bump instead of the guest: [kubevirtci] PR#1691
• seems like it is the right direction
• qemu runner bumped from v10.1.5 (fedora 43) to v10.2.2 (fedora 44) 
• recent user space (qemu containerized) running on old kernel space (rhcos9 provided by CI BM nodes) might cause unexpected issues
• downgrading the qemu runner seems to solve the issue
• option 1: revert to fedora 43 (qemu v10.1.5) => [kubevirtci] PR#1787
• option 2: switching to centos 9 (qemu v10.1.0) => [kubevirtci] PR#1788
• will discuss the best option with [dhiller] once he is back
• [dollierp] s390x hardware has been replaced
• jobs will be re-enabled: [project-infra] PR#5324
• [dywhite] will approve
• kubeconfig certificate expired for the cluster prow-s390x-e2e-test
• requested a new one
• minor impact, very few jobs are scheduled on this cluster
• [dollierp] sig-operator lane flakiness status
• fixes still in progress: PR#18355, PR#18397
• ci-health
• 
• 
• most of SIG compute and storage failures occurred in release branches 
• ~60 SIG-CI failures related to bazel cache issue for the single PR kubevirt#18223 
• some noise is caused by the failures related to the kubevirtci node mage bump; we will have a clearer picture once this has been fixed
• quick look at the e2e job failures:
  https://grafana.ci.kubevirt.io/d/efpTS3t4z/e2e-jobs-overview-v2
• 
  
• [timeboxed: 5 mins] revisit previous action items
• 
  
• existing issues opened in last 7 days: https://github.com/search?q=repo%3Akubevirt%2Fproject-infra+is%3Aissue++-label%3Akind%2Fenhancement&type=issues&state=open
• [non-urgent]
• 
  
• [misc]
• Look at held tests:
• https://grafana.ci.kubevirt.io/d/uAoSeksSk/referee-retests?orgId=1&refresh=15m&from=now-1h&to=now
• 4 PRs with 3 retests
• is:pr is:open label:approved label:lgtm -label:do-not-merge/hold -label:needs-rebase 
• merge queue is currently empty
• is:pr is:open quarantine -label:do-not-merge/work-in-progress label:kind/flake -label:needs-rebase
• merge queue is currently empty
• recently merged PRs authored by SIG CI
• 11 recently merged PRs authored by SIG CI (query: is:pr is:merged merged:>=2026-07-20 author:dhiller author:dollierp author:whitedyl org:kubevirt”)
• kubevirt/kubevirtci#1784: feat(gocli): log resource diagnostics on SSH retry failures (by @dhiller)
• kubevirt/project-infra#5318: chore(jobs): consolidate GCS credentials preset (by @dollierp)
• kubevirt/project-infra#5316: s390x: disable lanes due to clusters being unreachable (by @dollierp)
• kubevirt/ci-health#168: fix: fetch PR timeline items individually to avoid GraphQL truncation (by @dhiller)
• kubevirt/project-infra#5314: fix(quarantine): map test SIG labels to valid Prow labels from labels.yaml (by @dhiller)
• kubevirt/project-infra#5313: chore(cleanup): remove unused Go code (by @dollierp)
• kubevirt/ci-health#167: fix(ci-failures): add patterns for uncategorized internal CI failures (by @dhiller)
• kubevirt/kubevirtci#1780: chore(cluster-up): update default provider to k8s-1.36 (by @dollierp)
• kubevirt/project-infra#5309: fix(external-plugins): handle github app bot name (by @dollierp)
• kubevirt/project-infra#5302: feat(peribolos): switch to GitHub App credentials (by @dollierp)
• kubevirt/project-infra#5301: feat(label-sync): switch to GitHub App credentials (by @dollierp)
• Igor Bezukh Bazel rules migration from rules_oci to rules_img:
• https://github.com/kubevirt/kubevirt/pull/17598
• VEP 258 : https://github.com/kubevirt/enhancements/pull/258
• [dywhite] will have a look

Action items

• 
  
• communication
• send meeting notes to kubevirt-dev (include meeting changes for upcoming instances)

--

Regards,

The KubeVirt CI maintenance team.