sig-ci failure rate

[Daniel Hiller]

Hey folks,

we are currently looking at a very high rate of CI failures (~14%, >100 test lanes failing) right now.

What we can say so far:

* the large majority of failures was caused by a vanished Centos RPM, likely in turn caused by an rpm upload not having happened in time (http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/libvirt-daemon-driver-storage-core-10.10.0-13.el9.x86_64.rpm)

* majority of affected lanes are for release-1.6 nearly all sig lanes of sig-compute, sig-storage and sig-network, across all Kubernetes versions

We are working on getting a better insight in how to mitigate the situation and will keep you updated.

Thanks!

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty  

[Federico Fossemo]

Hi Daniel,

Regarding the high rate of CI failures, I wanted to provide an update on the fixes:

- For the main branch, this PR will solve the issue: https://github.com/kubevirt/kubevirt/pull/16516 (https://github.com/kubevirt/kubevirt/pull/16516)

- For release-1.6, https://github.com/kubevirt/kubevirt/pull/16564 (https://github.com/kubevirt/kubevirt/pull/16564) was recently merged and has fixed that branch.

Best regards,

-F

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAK%2BeyL4vSRzxJT8O-M63PkkUrsOvpm3DTWpMk4CwHq25fXfdHg%40mail.gmail.com.