2026-04-20 sig-ci weekly meeting - notes

[Daniel Hiller]

2026-04-20 sig-ci weekly meeting

When: Weekly on Mon, 10:00 – 10:30am

Notes: KubeVirt CI SIG meeting notes

Attendees: dhiller, dollierp, fossedihelm, nirdothan

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

Topics:

• [urgent]

• [dollierp] AMD workloads API server certificate is expired

• Pinged AMD contact point

• PR to disable SEV jobs until fixed: https://github.com/kubevirt/project-infra/pull/4980

• [dhiller] PR approved

• how come the livez job was not failing? I’d have expected this?

• [dollierp] Some project-infra jobs use deleted tags for gcr.io/k8s-staging-test-infra/… images

• https://github.com/kubevirt/project-infra/issues/4948

• Should we use the latest tag or improve the bump script to handle them?

• [dhiller] improving bump script makes sense, Harshita is working on the cli conversion issue AFAIR

• Related PRs (needs review)

• https://github.com/kubevirt/project-infra/pull/4949

• https://github.com/kubevirt/project-infra/pull/4967 

• [dhiller] revisit flaky test fix pr: https://github.com/kubevirt/kubevirt/pull/17495

• ci-health

• 

• 

• quick look at the e2e job failures:
  https://grafana.ci.kubevirt.io/d/efpTS3t4z/e2e-jobs-overview-v2

• 
  

• [timeboxed: 5 mins] revisit previous action items

• 
  

• existing issues opened in last 7 days: https://github.com/search?q=repo%3Akubevirt%2Fproject-infra+is%3Aissue++-label%3Akind%2Fenhancement&type=issues&state=open

• [non-urgent]

• [dollierp] *-alpine-with-tooling jobs are having issues

• https://github.com/kubevirt/kubevirtci/issues/1336#issuecomment-4254500524

• Frequent deadlock of extlinux command during the build:

/usr/bin/conmon

\_ [entrypoint]

    \_ [alpine-make-vm-]

        \_ extlinux --install /boot

• This leads to Pod of the Jobs stuck in Terminating state and dangling processes

• It might have an impact on the nodes of the CI workloads cluster (worst offender):

kubevirt-worker-bm12 | load average: 433.80, 433.08, 433.74

• Draining the node didn’t reduce the load which seems to confirm it is a side effect of the immortal extlinux processes

• Back to normal after rebooting the node:
  kubevirt-worker-bm12 | load average: 2.69, 5.66, 13.36

• [dhiller] how was this noticed? should we somehow add alerts for some basic prow-workloads node metrics?

• [fossedihelm] feedback request on https://github.com/kubevirt/enhancements/pull/234

• [dhiller] will take a look

• [misc]

• Look at held tests:

• https://grafana.ci.kubevirt.io/d/uAoSeksSk/referee-retests?orgId=1&refresh=15m&from=now-1h&to=now

• is:pr is:open label:approved label:lgtm -label:do-not-merge/hold -label:needs-rebase 

• is:pr is:open quarantine -label:do-not-merge/work-in-progress label:kind/flake -label:needs-rebase 

• recently merged PRs authored by SIG CI

• 13 recently merged PRs authored by SIG CI (query: is:pr is:merged merged:>=2026-04-12 author:dhiller author:dollierp author:whitedyl org:kubevirt”)

• kubevirt/project-infra#4971: fix: use commenter-oauth-token for dependabot skip-review periodic (by @dhiller)

• kubevirt/project-infra#4962: feat(prow): bootstrap transition to GitHub App credentials (by @dollierp)

• kubevirt/project-infra#4961: feat(flakes): make check flakes lane required (by @dhiller)

• kubevirt/project-infra#4960: Fix(coverage): Hardcode toolchain and coverage paths (by @Whitedyl)

• kubevirt/project-infra#4958: chore(webhooks): rotate secrets after GitHub notice (by @dollierp)

• kubevirt/project-infra#4946: fix: update prow-debug kubeconfig script to use current cluster contexts (by @dhiller)

• kubevirt/project-infra#4945: fix: use /test all instead of /retest-required for dependabot PRs (by @dhiller)

• kubevirt/project-infra#4939: Add README documentation for release-blocker external plugin (by @Whitedyl)

• kubevirt/project-infra#4937: chore(prow-kustomize): remove duplicate entries and use anchors (by @dollierp)

• kubevirt/project-infra#4933: Fix coverage plugin ProwJobs spec with Agent and Decoration config (by @Whitedyl)

• kubevirt/ci-health#118: feat: add analyze-build command for single prow job failure analysis (by @dhiller)

• kubevirt/project-infra#4867: chore(livez): also monitor external clusters (by @dollierp)

• kubevirt/kubevirt#17166: [release-1.2] MDEV tests: fix checkAllMDEVCreated issue upon multiple GPUs (by @dollierp)

Action items

• 
  
• 

  communication

• send meeting notes to kubevirt-dev (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty