2025-08-27 sig-ci quarantine catch-up - notes

[Daniel Hiller]

2025-08-27 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Attendees: ffossemo, dhiller, ndothan, omisan, brianmcarey,

Reminders:

• we will create GitHub issues for tracking

• GitHub issues and PRs

• should be marked with /sig ci and /kind flake if applicable

• should be marked with the target sig

• kubevirt quarantine process documentation

Topics:

• [urgent]

• [bc] Bad flake in the SRIOV lane - https://search.ci.kubevirt.io/?search=single+SRIOV+network+migration+should+be+successful+with+a+running+VMI+on+the+target&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job 

• Quarantine PR - https://github.com/kubevirt/kubevirt/pull/15525 

• tracker issue to be created by Brian

• ci-health

• 

• 

• [bc] high ci failure rate due to release-1.1 PR failures

• [bc] also build failures in 1.1 due to bazel dependency missing, fixed though -https://github.com/kubevirt/kubevirt/pull/15446

• [fossedihelm] asked Igor to backport fix from 1.1 to avoid hitting this again 

• previous action items

• state of existing issues: https://github.com/kubevirt/kubevirt/issues?q=is%3Aissue+is%3Aopen+label%3Akind%2Fflake+sort%3Aupdated-asc+label%3Asig%2Fci

• [non-urgent]

• [bc] s390x cluster experiencing DNS issues causing s390x unit test lane to fail constantly

• https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/15516/pull-kubevirt-unit-test-s390x/1960601493836926976 

• Moving lane to optional - https://github.com/kubevirt/project-infra/pull/4326 

• Look at flakes

• flake stats - create issues accordingly

• count: 436 failures overall

• some clustered failures on 1.33 periodic sig-compute lane, need to get investigated:

• https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.33-sig-compute/1957377094186962944

• https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.33-sig-compute/1956289916493107200

• GH issue: https://github.com/kubevirt/kubevirt/issues/15526

• 1.34 periodic sig-compute clustered failures

• https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.34-sig-compute/1957877903723073536

• https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/logs/periodic-kubevirt-e2e-k8s-1.34-sig-compute/1953891487208968192

• GH issue: https://github.com/kubevirt/kubevirt/issues/15527

• serial lane timeouts

• not sure whether caused by PR or infra, [bc] to take a look

• https://testgrid.k8s.io/kubevirt-presubmits#pull-kubevirt-e2e-k8s-1.33-sig-compute-serial&width=20

• dequarantine tests:

• look at list of quarantined tests

• count: 13 tests in quarantine currently

• check status, i.e. who is working on those

• look at PRs that want to fix flakes

• see whether we can dequarantine tests 

• misc topics

• 
  

Action items

• 

  Daniel Hiller most-flaky-tests sometimes broken: https://storage.googleapis.com/kubevirt-prow/reports/most-flaky-tests/kubevirt/kubevirt/index.html

• 

  sig-storage etc should show up 

• 

  Brian Carey to take a look - confirm whether only cancelled job runs due to pushes and no infra failures https://testgrid.k8s.io/kubevirt-presubmits#pull-kubevirt-e2e-k8s-1.33-sig-compute-serial&width=20

• 

  update/create issues with latest flakes spotted

• 

  communication

• send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty