no-flake-check decorator - a band-aid to avoid running on check-tests-for-flakes

[Daniel Hiller]

Hey all,

This (CEST) morning we had a lively discussion about the decorator `no-flake-check` affecting the check-tests-for-flakes lane.

So if a test is not fit for the check-tests-for-flakes lane it can receive the no-flake-check decorator, which will exclude it from being run on that lane.

The docs say:
// NoFlakeCheck decorates tests that are not compatible with the check-tests-for-flakes test lane.
// This should only be used for legitimate purposes, like on tests that have a flake-checker-friendly clone.

Note: this clearly needs updating; see below

So basically for a test that can't run on the lane, there's a couple solutions available:
* Configure the lane to have the infra/reqs to make the test succeed

* Add the decorator in as a band aid to signal that the test is not supported yet

* Update the lane script to specifically exclude the test(s) being run

To be clear: this decorator must not be put on tests that are flaky!

The desired long-term outcome here IMHO is that as many of the tests running on the “standard” required sig lanes which exist for all three kubernetes versions are also being covered on the check-tests-for-flake lane(s) - this should lead to more overall test stability on the required lanes which will be an achievement that every contributor benefits from.

To increase the flake-check coverage we could create more specific lanes that have the infrastructure requirements so that those tests can be supported, i.e. specific sig-network flake-check lanes.

Furthermore it's not fully clear how and when to apply the decorator. Therefore we agreed that when the author applies the decorator they must add the reason why it was applied. It is suggested to use a separate commit with an explaining commit message for this change.

An idea was that we should make `make lint` fail if there’s no separate commit for applying that decorator to a test - however I am not sure how easy that is to implement though.

Another suggestion was to create automation marking the PR with a specific label kind/no-flake-check-added, where the label color is red, so that it is immediately visible to the reviewer.

Any thoughts and comments on this matter are highly appreciated - I'll also take this topic to the community meeting this afternoon.

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty  

[Edward Haas]

On Wed, Jun 3, 2026 at 12:27 PM 'Daniel Hiller' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hey all,

This (CEST) morning we had a lively discussion about the decorator `no-flake-check` affecting the check-tests-for-flakes lane.

So if a test is not fit for the check-tests-for-flakes lane it can receive the no-flake-check decorator, which will exclude it from being run on that lane.

The docs say:
// NoFlakeCheck decorates tests that are not compatible with the check-tests-for-flakes test lane.
// This should only be used for legitimate purposes, like on tests that have a flake-checker-friendly clone.

Note: this clearly needs updating; see below

So basically for a test that can't run on the lane, there's a couple solutions available:
* Configure the lane to have the infra/reqs to make the test succeed

* Add the decorator in as a band aid to signal that the test is not supported yet

* Update the lane script to specifically exclude the test(s) being run

To be clear: this decorator must not be put on tests that are flaky!

The desired long-term outcome here IMHO is that as many of the tests running on the “standard” required sig lanes which exist for all three kubernetes versions are also being covered on the check-tests-for-flake lane(s) - this should lead to more overall test stability on the required lanes which will be an achievement that every contributor benefits from.

To increase the flake-check coverage we could create more specific lanes that have the infrastructure requirements so that those tests can be supported, i.e. specific sig-network flake-check lanes.

Furthermore it's not fully clear how and when to apply the decorator. Therefore we agreed that when the author applies the decorator they must add the reason why it was applied. It is suggested to use a separate commit with an explaining commit message for this change.

An idea was that we should make `make lint` fail if there’s no separate commit for applying that decorator to a test - however I am not sure how easy that is to implement though.

Linter are usually used to check the code as is, not to check the commit message context.
Unless you share some other tool or project that uses such checks, I prefer not to go this path.

Another suggestion was to create automation marking the PR with a specific label kind/no-flake-check-added, where the label color is red, so that it is immediately visible to the reviewer.

I find the approver responsible for accepting such changes. If an approver or the SIG that owns the tests is not following best practices, it is their prerogative and (arguably) their burden.
I do not think it requires more visibility, but this is very opinionated.

Another option is to collect reports every release (including RC ones) detailing which tests are marked as such.
This could be part of a wider report that includes the number of tests, how many are quarantined, etc.

Any thoughts and comments on this matter are highly appreciated - I'll also take this topic to the community meeting this afternoon.

--

Kind regards,

Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty  

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAK%2BeyL6ZCpZh7dioA3jYkNwxgWZ7phzoGZMsDj1Xgz6joEDsMg%40mail.gmail.com.