2026-03-23 sig-ci weekly - notes
Daniel Hiller
When: Weekly on Mon, 10:00 – 10:30am
Notes: KubeVirt CI SIG meeting notes
Attendees: Dylan White, Chandra Merla Federico Fossemo Ananya Banerjee
Reminders:
we will create GitHub issues for tracking
GitHub issues and PRs
should be marked with /sig ci and /kind flake if applicable
should be marked with the target sig
Topics:
[urgent]
[dhiller] e2e lane failures on main increased to 50% (see details below)
update: two bot-created PRs were the reason, failure rate is going down again
ci-failures
suspected corruption of images related to sharedimagescontroller
WARNING: Image 1f84bb6fd993 exists in local storage but may be corrupted (remove the image to resolve the issue): size for layer "c3289bf625ffda19fc793ede1d5b65eba5ad434441e4126ced22520c533af4ff" is unknown, failing getSize()
Error: unable to copy from source docker://quay.io/kubevirtci/gocli:2603192055-a9b102ab: copying system image from manifest list: trying to reuse blob sha256:0cd626c84f95aac340bddaaac2da4936a0d3028d4f457606dde6e600e95fd92c at destination: looking for layers with digest "sha256:0cd626c84f95aac340bddaaac2da4936a0d3028d4f457606dde6e600e95fd92c": deleting layer "87cfd9c8364cf13a2da5d3f5cafc64c9b40a5a38692e792c073fd2c48d57e844": failed to add to stage directory: rename /var/lib/shared-images/overlay/87cfd9c8364cf13a2da5d3f5cafc64c9b40a5a38692e792c073fd2c48d57e844 /var/lib/containers/storage/overlay/tempdirs/temp-dir-3593808655/1-87cfd9c8364cf13a2da5d3f5cafc64c9b40a5a38692e792c073fd2c48d57e844: invalid cross-device link
sriov kind cluster didn’t come up
remaining “taint noschedule not found” related to vgpu node issue
quick look at the e2e job failures:
https://grafana.ci.kubevirt.io/d/efpTS3t4z/e2e-jobs-overview-v2a stark overall increase in lane failures since around 2026-03-21 01:00 GMT+1
some failures:
ERROR: /root/go/src/kubevirt.io/kubevirt/images/winrmcli/BUILD.bazel:16:10: OCI Image //images/winrmcli:winrmcli-image failed: Failed to fetch blobs because they do not exist remotely.: Missing digest: 41952919fbf4e2bc332e41cf207afe978f755c4a356cff0a097d9274fadb9e0b/341 for bazel-out/k8-fastbuild/bin/images/winrmcli/winrmcli-image.0.descriptor.json
WARNING: Download from https://mirror.bazel.build/github.com/golang/tools/archive/refs/tags/v0.30.0.zip failed: class java.io.FileNotFoundException GET returned 404 Not Found
pull request candidates that may have caused this: https://github.com/search?q=repo%3Akubevirt%2Fkubevirt+is%3Apr+is%3Amerged+merged%3A%222026-03-20..2026-03-22%22+base%3Amain&type=pullrequests
found the offenders:
https://prow.ci.kubevirt.io/pr-history/?org=kubevirt&repo=kubevirt&pr=14804 - that PR which was created automatically was responsible for 88 failures in last 24h
https://prow.ci.kubevirt.io/pr-history/?org=kubevirt&repo=kubevirt&pr=17243 - PR was responsible for 64 failures in last 24h
conclusion: there seems to be something wrong with the bump automation inside k/kubeirt, need to take a look
[timeboxed: 5 mins] revisit previous action items
existing issues opened in last 7 days: https://github.com/search?q=repo%3Akubevirt%2Fproject-infra+is%3Aissue++-label%3Akind%2Fenhancement&type=issues&state=open
[non-urgent]
[misc]
Look at held tests:
https://grafana.ci.kubevirt.io/d/uAoSeksSk/referee-retests?orgId=1&refresh=15m&from=now-1h&to=now
is:pr is:open label:approved label:lgtm -label:do-not-merge/hold -label:needs-rebase
is:pr is:open quarantine -label:do-not-merge/work-in-progress label:kind/flake -label:needs-rebase
15 recently merged PRs authored by SIG CI (query: is:pr is:merged merged:>=2026-03-16 author:dhiller author:dollierp org:kubevirt”)
kubevirt/project-infra#4857: Add testgrid email alerts for deploy postsubmit jobs (by @dhiller)
kubevirt/project-infra#4856: fix(prow-deploy): use KUBECONFIG env var instead of --kubeconfig flag (by @dollierp)
kubevirt/project-infra#4849: feat(kubeconfigs): transition to git-crypt encryption (by @dollierp)
kubevirt/project-infra#4845: feat: add release branch lane filter to most-flaky-tests report (by @dhiller)
kubevirt/kubevirtci#1652: fix: skip Pod InPlace Resize Container conformance tests (by @dhiller)
kubevirt/ci-health#113: fix: add retry logic and correct cron schedule for badges-update (by @dhiller)
kubevirt/project-infra#4839: fix(git-pr.sh): fix option parsing and use repo-local script (by @dhiller)
kubevirt/project-infra#4831: Fix git pr and quarantine job (by @dhiller)
kubevirt/project-infra#4830: chore(deps): bump mikefarah/yq to version 4 (by @dollierp)
kubevirt/kubevirt#17164: [release-1.4] MDEV tests: fix checkAllMDEVCreated issue upon multiple GPUs (by @dollierp)
kubevirt/ci-health#109: Sort ci failures by date desc (by @dhiller)
kubevirt/kubevirt#17146: OWNERS: add sig-buildsystem reviewers/approvers for bazel files (by @dhiller)
kubevirt/project-infra#4813: fix(test-runtimes): change to select test lanes (by @dhiller)
kubevirt/project-infra#4748: docs: convert infrastructure SVG diagrams to Mermaid (by @dhiller)
kubevirt/project-infra#4596: chore(prow): sync manifests with k8s test-infra repository (by @dollierp)
Action items
communication
send meeting notes to kubevirt-dev (include meeting changes for upcoming instances)
Kind regards,
Daniel Hiller
He / Him / His
Principal Software Engineer, KubeVirt CI, OpenShift Virtualization
 |
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty