pull-kubevirt-check-tests-for-flakes - remove or make required

[Daniel Hiller]

Hey all,

it was brought to our attention that the pull-kubevirt-check-tests-for-flakes lane seems to be largely ignored, as it is not required (example [1][2]). *

If people are actually ignoring it it's value is questionable, since it doesn't require the PR author to fix anything - they can get away with retesting and the lane will eventually succeed. As is the case in the example.

You all know my opinion of automated retesting, so I won't go into details - however I believe for the project quality it would be favorable if a test is noticed as being flaky, the least action should be to create a flaky test GitHub issue.

My broader question is - should we finally make the lane required? IMHO we should, so that people are forced to AT LEAST create the issue, and then we might override it if there's no bandwidth to fix it right away.

Or we remove the lane entirely since people are ignoring it anyway.

WDYT?

* In the example a test is failing every time, so this seems to be a case of test order dependency

[1]: https://github.com/kubevirt/kubevirt/pull/17307#issuecomment-4151651754

[2]: https://prow.ci.kubevirt.io/pr-history/?org=kubevirt&repo=kubevirt&pr=17307

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty  

[Alexander Wels]

For me the check-test-for-flakes lane is very important to me; it's my primary signal for determining if any new tests I added are reliable or not. IMO it should be a voting lane.

Alexander

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAK%2BeyL4uLLD29eyVH7Z5sY2G4M%2BQEiiWhUxz1Egm%3DEoQTTL8Sg%40mail.gmail.com.

[Alvaro Romero]

I agree with Alexander. At least in my experience this lane is pretty reliable and has consistently helped catch bugs or issues that might have been missed otherwise. I think it should be required.

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAKDnqXZuxASEkqWsEcG0eQTwM69w3zh--bbHyMCNEcT2xhMJLw%40mail.gmail.com.

[Alex Kalenyuk]

As a maintainer I don't normally dismiss this lane when it fails, I think it is valuable. However, I understand that anything that's not required can fly under the radar;
The issue is that it's a one-size-fits-all lane (KUBEVIRT_ENABLE_EVERYTHING=true) and thus incapable of running certain tests,
and would require an occasional override which I am not certain sig-* maintainer could grant.

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAOZDZyHOmsu11CgwyVC2g%3D7RUYMCC_xNXzO7gRYu5hswwDPsow%40mail.gmail.com.

[Daniel Hiller]

On Mon, Mar 30, 2026 at 4:01 PM Alex Kalenyuk <akal...@redhat.com> wrote:

As a maintainer I don't normally dismiss this lane when it fails, I think it is valuable. However, I understand that anything that's not required can fly under the radar;
The issue is that it's a one-size-fits-all lane (KUBEVIRT_ENABLE_EVERYTHING=true) and thus incapable of running certain tests,
and would require an occasional override which I am not certain sig-* maintainer could grant.

Fair point.

We can mark tests unfit to be run in a lane using the `NoFlakeCheck` decorator NoFlakeCheck [1] - that can be done directly inside the same PR.

[1]: https://github.com/kubevirt/kubevirt/blob/6ee3f5060763038b93284adb93a317b61400fd47/tests/decorators/decorators.go#L122

--

-- 

Best,

Daniel