2026-04-22 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2026-04-22 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: dhiller, fossedihelm, dollierp

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• [dollierp] Quay outage over night https://status.redhat.com/incidents/rgf6sx9lcjyd

• lasted from 12:50 AM CEST - 2:09 AM CEST

• need to check for impacted pull requests

• https://github.com/kubevirt/kubevirt/pull/16815: retested and merged

• https://github.com/kubevirt/kubevirt/pull/17542: retested and merged

• https://github.com/kubevirt/kubevirt/pull/17555: retesting

• ci-health

• 

• 

• review and merge the quarantine PRs in 2 working days without SIG lgtm. 

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+quarantine+-label%3Ado-not-merge%2Fwork-in-progress+label%3Akind%2Fflake+-label%3Aneeds-rebase

• [dhiller] https://github.com/kubevirt/kubevirt/pull/17454 still open, since an automation issue needs to be investigated

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• n/a

• [non-urgent]

• [dollierp] vdpa-network-binding-plugin repository onboarding has been split in 2 PRs

• Repository creation PR#4944 (merged)

• Admin team creation PR#4986

• The repository was created but Peribolos is still complaining for the last one

• [dhiller] will take a look approved the PR, should be merged soon

• [fossedihelm] Dequarantine lane & process:

• https://github.com/kubevirt/kubevirt/pull/17542 

• https://github.com/kubevirt/kubevirt/pull/17391#issuecomment-4204618960

• [dhiller] lane still exists, but it got cancelled due to a new push on the PR which probably hid then the github status by showing the next commit in tide https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/17391/pull-kubevirt-check-dequarantine-test/2041783746444660736

• Look at flakes

• most flaky tests

• [sig-performance] Control Plane Performance Density Testing Density test [small] create a batch of 100 VMIs should successfully create all VMIS

• 🔎 💥 9% over 336h on

• pull-kubevirt-e2e-k8s-1.34-sig-performance: Failures: 2h0m0s 7h0m0s 9h0m0s 10h0m0s 13h0m0s 18h0m0s 19h0m0s 96h0m0s 144h0m0s 168h0m0s 168h0m0s 168h0m0s 216h0m0s 288h0m0s 288h0m0s 288h0m0s 288h0m0s 288h0m0s

• ask lyarwood whether he has time to investigate

• flake stats - create issues accordingly

• overall ( ∑=541, 100.00% )

• periodic-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=76, 14.05% )

• periodic-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=68, 12.57% )

• periodic-kubevirt-e2e-test-S390X ( ∑=63, 11.65% )

• bunch of tests failing every time, add or create an issue for wg-arch-s390x

• periodic-kubevirt-e2e-k8s-1.34-sig-monitoring ( ∑=55, 10.17% )

• needs inspection by sig-observability

• pull-kubevirt-e2e-k8s-1.35-sig-operator ( ∑=44, 8.13% )

• fossedihelm to check whether related to recent release

• periodic-kubevirt-e2e-k8s-1.34-sig-storage ( ∑=41, 7.58% )

• recently improved in stability overall

• periodic-kubevirt-e2e-k8s-1.35-sig-storage ( ∑=40, 7.39% )

• nothing obviously wrong here

• pull-kubevirt-e2e-kind-1.35-sig-compute-arm64 ( ∑=23, 4.25% )

•  pull-kubevirt-e2e-k8s-1.35-sig-compute-serial ( ∑=20, 3.70% )

• periodic-kubevirt-e2e-k8s-1.35-ipv6-sig-network ( ∑=19, 3.51% )

• pull-kubevirt-e2e-k8s-1.35-sig-compute-migrations ( ∑=13, 2.40% )

• pull-kubevirt-e2e-k8s-1.34-sig-performance-kwok ( ∑=11, 2.03% )

• pull-kubevirt-e2e-k8s-1.35-sig-network ( ∑=9, 1.66% )

• periodic-kubevirt-e2e-k8s-1.34-sig-performance-kwok-100 ( ∑=7, 1.29% )

• periodic-kubevirt-e2e-k8s-1.35-sig-compute-migrations ( ∑=7, 1.29% )

• pull-kubevirt-e2e-k8s-1.35-sig-compute ( ∑=7, 1.29% )

• pull-kubevirt-e2e-k8s-1.34-sig-compute ( ∑=6, 1.11% )

Last updated: 2026-04-22 07:03:29.785597498 +0000 UTC m=+12.392696882

• Look at held tests:

• https://grafana.ci.kubevirt.io/d/uAoSeksSk/referee-retests?orgId=1&refresh=15m&from=now-1h&to=now

• https://github.com/kubevirt/kubevirt/pulls?q=is%3Apr+is%3Aopen+label%3Aapproved+label%3Algtm+-label%3Ado-not-merge%2Fhold+-label%3Aneeds-rebase

• dequarantine tests:

• look at list of quarantined tests

• Count: 20 tests in quarantine currently
  

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests

• misc topics

• [dhiller] working on creating the lanes for 1.36

• https://github.com/kubevirt/project-infra/pull/4987#issuecomment-4290795077

• there’s work from lyarwood in the pipeline for centosstream10, but that is for kubevirtci, we need to consume another bump for that

• also follow up required due to run_before_merge for 1.35

• [xpivarc] please prepare a report for the cnv sprint that ci is a mess and that people need to work on this

• periodics horrible

• presubmits failing

• high profile flakes

• [dhiller] will do

Action items

• 
  
• 

  update/create issues with latest flakes spotted

• 

  communication

• send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty