2026-06-24 sig-ci quarantine catch-up - notes

3 views
Skip to first unread message

Daniel Hiller

unread,
Jun 24, 2026, 4:42:04 AM (7 days ago) Jun 24
to kubevirt-dev, Krzysztof Majcher, Adam Litke, Tal Nisan
2026-06-24 sig-ci quarantine catch-up

When: Weekly on Wed, 9:45 – 10:15am

Notes: KubeVirt CI SIG meeting notes

Attendees: Dylan White, dhiller, dollierp, fossedihelm


Reminders:

Topics:

Last updated: 2026-06-24 07:31:36.476144709 +0000 UTC m=+14.427368304

Action items

  • unchecked

    Daniel Hiller investigate how batch in tide works in depth

  • unchecked

    update/create issues with latest flakes spotted

  • unchecked

    communication

    • send meeting notes to kubevirt-dev, bcc sig people for spotted flakes (include meeting changes for upcoming instances)



--

Kind regards,


Daniel Hiller

He / Him / His

Principal Software Engineer, KubeVirt CI, OpenShift Virtualization

Red Hat

dhi...@redhat.com   

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty

Lee Yarwood

unread,
Jun 24, 2026, 6:13:26 AM (7 days ago) Jun 24
to Daniel Hiller, kubevirt-dev, Krzysztof Majcher, Adam Litke, Tal Nisan
On Wed, 24 Jun 2026 at 09:42, 'Daniel Hiller' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Topics:

  • [urgent]

    • CI is overloaded (400+ jobs yesterday). As a result no PRs got merged yesterday against kubevirt/main and the merge queue is not shrinking.

      • It would be great to not concentrate that much PRs a few days before the code freeze (and to be more CI friendly in general).


I really don't think asking folks to be more CI friendly is enough, the system itself needs to enforce this in the week leading up to CF.

What about moving all existing and new kubevirt/kubevirt main PRs to draft a week before CF if they lack a vep-approved, high-priority or some other new maintainer-only required-before-cf label?

That should reduce the load on CI ahead of CF and hopefully provide a more predictable merge window before we cut the initial release branch. We would then have enough time after this during RC to land and backport bugfixes that may have been moved to draft ahead of CF.

Cheers,

Lee

Daniel Hiller

unread,
Jun 24, 2026, 6:15:21 AM (7 days ago) Jun 24
to Lee Yarwood, kubevirt-dev, Krzysztof Majcher, Adam Litke, Tal Nisan
On Wed, Jun 24, 2026 at 12:13 PM Lee Yarwood <lyar...@redhat.com> wrote:
On Wed, 24 Jun 2026 at 09:42, 'Daniel Hiller' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Topics:

  • [urgent]

    • CI is overloaded (400+ jobs yesterday). As a result no PRs got merged yesterday against kubevirt/main and the merge queue is not shrinking.

      • It would be great to not concentrate that much PRs a few days before the code freeze (and to be more CI friendly in general).


I really don't think asking folks to be more CI friendly is enough, the system itself needs to enforce this in the week leading up to CF.

What about moving all existing and new kubevirt/kubevirt main PRs to draft a week before CF if they lack a vep-approved, high-priority or some other new maintainer-only required-before-cf label?


Some months ago I've proposed enforcing code freeze by changing tide rules to exclude any PR that has bug/flake/failing-test/approved-vep labels here, but there was pushback against this approach AFAIR. Maybe we can give it a try next release.

 
That should reduce the load on CI ahead of CF and hopefully provide a more predictable merge window before we cut the initial release branch. We would then have enough time after this during RC to land and backport bugfixes that may have been moved to draft ahead of CF.

Cheers,

Lee


--
-- 
Best,
Daniel

Daniel Hiller

unread,
Jun 24, 2026, 6:18:55 AM (7 days ago) Jun 24
to Lee Yarwood, kubevirt-dev, Krzysztof Majcher, Adam Litke, Tal Nisan
On Wed, Jun 24, 2026 at 12:15 PM Daniel Hiller <dhi...@redhat.com> wrote:


On Wed, Jun 24, 2026 at 12:13 PM Lee Yarwood <lyar...@redhat.com> wrote:
On Wed, 24 Jun 2026 at 09:42, 'Daniel Hiller' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Topics:

  • [urgent]

    • CI is overloaded (400+ jobs yesterday). As a result no PRs got merged yesterday against kubevirt/main and the merge queue is not shrinking.

      • It would be great to not concentrate that much PRs a few days before the code freeze (and to be more CI friendly in general).


I really don't think asking folks to be more CI friendly is enough, the system itself needs to enforce this in the week leading up to CF.

What about moving all existing and new kubevirt/kubevirt main PRs to draft a week before CF if they lack a vep-approved, high-priority or some other new maintainer-only required-before-cf label?


Some months ago I've proposed enforcing code freeze by changing tide rules to exclude any PR that has bug/flake/failing-test/approved-vep labels here, but there was pushback against this approach AFAIR. Maybe we can give it a try next release.

Apologies I meant "any PR that lacks bug/flake/failing/test/approved-veg labels should be excluded from being merged during that period"

Here's the PR: https://github.com/kubevirt/enhancements/pull/221

 
That should reduce the load on CI ahead of CF and hopefully provide a more predictable merge window before we cut the initial release branch. We would then have enough time after this during RC to land and backport bugfixes that may have been moved to draft ahead of CF.

Cheers,

Lee


--
-- 
Best,
Daniel


--
-- 
Best,
Daniel

Lee Yarwood

unread,
Jun 24, 2026, 6:20:47 AM (7 days ago) Jun 24
to Daniel Hiller, kubevirt-dev, Krzysztof Majcher, Adam Litke, Tal Nisan
On Wed, 24 Jun 2026 at 11:18, Daniel Hiller <dhi...@redhat.com> wrote:
> On Wed, Jun 24, 2026 at 12:15 PM Daniel Hiller <dhi...@redhat.com> wrote:
>> On Wed, Jun 24, 2026 at 12:13 PM Lee Yarwood <lyar...@redhat.com> wrote:
>>> On Wed, 24 Jun 2026 at 09:42, 'Daniel Hiller' via kubevirt-dev <kubevi...@googlegroups.com> wrote:
>>>>
>>>> Topics:
>>>>
>>>> [urgent]
>>>>
>>>> CI is overloaded (400+ jobs yesterday). As a result no PRs got merged yesterday against kubevirt/main and the merge queue is not shrinking.
>>>>
>>>> It would be great to not concentrate that much PRs a few days before the code freeze (and to be more CI friendly in general).
>>>
>>>
>>> I really don't think asking folks to be more CI friendly is enough, the system itself needs to enforce this in the week leading up to CF.
>>>
>>> What about moving all existing and new kubevirt/kubevirt main PRs to draft a week before CF if they lack a vep-approved, high-priority or some other new maintainer-only required-before-cf label?
>>>
>>
>> Some months ago I've proposed enforcing code freeze by changing tide rules to exclude any PR that has bug/flake/failing-test/approved-vep labels here, but there was pushback against this approach AFAIR. Maybe we can give it a try next release.
>
>
> Apologies I meant "any PR that lacks bug/flake/failing/test/approved-veg labels should be excluded from being merged during that period"
>
> Here's the PR: https://github.com/kubevirt/enhancements/pull/221

ACK yeah apologies for forgetting about this, lets definitely discuss
it again at the unconference for v1.10.0!

Cheers,

Lee

Daniel Hiller

unread,
Jun 24, 2026, 6:44:52 AM (7 days ago) Jun 24
to Lee Yarwood, kubevirt-dev, Krzysztof Majcher, Adam Litke, Tal Nisan
Admittedly the PR is still not ready though, but I think we can all agree that something has to change for next release.

One thing to note is that we recently have put approved-vep PR into higher priority for tide: https://github.com/kubevirt/project-infra/blob/f0a95917dd413d402cb3d81468d06c5c21f27065/github/ci/prow-deploy/kustom/base/configs/current/config/config.yaml#L332

 

Cheers,

Lee



--
-- 
Best,
Daniel
Reply all
Reply to author
Forward
0 new messages