[kubevirt-dev] Removal of ARM64 test lanes from KubeVirt

[Brian Carey]

Hi everyone,

As some of you may be aware, the arm64 test lanes in KubeVirt had been failing consistently for over 6 months now. We moved to stop reporting these failures on pull requests at the end of October 2024 as there wasn't a sign of improvement and to reduce noise for contributors on PRs[1].

Recently we have learned that ARM plans on decommissioning the hardware that we were using for this testing so there is no plan for fixing these failures in the near future. Following this we have decided to remove the ARM test lanes from KubeVirt[2] as we don't have a working environment for these tests. 

Unfortunately this is quite a step backwards for the KubeVirt on ARM efforts and we currently don't know when we will have a reliable ARM environment for CI testing.

These test lanes can be reintroduced when we have a stable ARM environment and the test lanes themselves pass consistently.

Thanks,

Brian

[1] https://github.com/kubevirt/project-infra/pull/3526

[2] https://github.com/kubevirt/project-infra/pull/4097

[Itamar Holder]

This is indeed very unfortunate.

I wonder what this means regarding ARM-related features.

How can we support / develop those without a proper test environment? Does it make sense to create an ARM lane from virtualized ARM kubevirtCI nodes?

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CA%2B3oYq77p9FdWEvPPkdwS3Kuynp2K9BEyc%3DHvD6gSmH5SbYQuQ%40mail.gmail.com.

[Luboslav Pivarc]

On Wed, Apr 23, 2025 at 6:02 PM 'Itamar Holder' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

This is indeed very unfortunate.

I wonder what this means regarding ARM-related features.

How can we support / develop those without a proper test environment? Does it make sense to create an ARM lane from virtualized ARM kubevirtCI nodes?

Emulation would be most likely too slow if you meant emulation.

-Lubo

 

On Wed, 23 Apr 2025 at 12:21, 'Brian Carey' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

Hi everyone,

As some of you may be aware, the arm64 test lanes in KubeVirt had been failing consistently for over 6 months now. We moved to stop reporting these failures on pull requests at the end of October 2024 as there wasn't a sign of improvement and to reduce noise for contributors on PRs[1].

Recently we have learned that ARM plans on decommissioning the hardware that we were using for this testing so there is no plan for fixing these failures in the near future. Following this we have decided to remove the ARM test lanes from KubeVirt[2] as we don't have a working environment for these tests. 

Unfortunately this is quite a step backwards for the KubeVirt on ARM efforts and we currently don't know when we will have a reliable ARM environment for CI testing.

These test lanes can be reintroduced when we have a stable ARM environment and the test lanes themselves pass consistently.

Thanks,

Brian

[1] https://github.com/kubevirt/project-infra/pull/3526

[2] https://github.com/kubevirt/project-infra/pull/4097

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CA%2B3oYq77p9FdWEvPPkdwS3Kuynp2K9BEyc%3DHvD6gSmH5SbYQuQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CALpNySxJsu9bRYuSg7JzB0VBGuw4Uysu2K1wLZYzYqEHp8VB5g%40mail.gmail.com.

[Lee Yarwood]

Are we able to request any ARM based compute resources from the CNCF
to power these lanes? I know various companies donate hardware or
credits into the CNCF, with Oracle for example donating Ampere
Arm®-based compute credits for their OCI cloud, but it looks like
these specific credits have already been exhausted for the year haha
[1].

[1] https://blogs.oracle.com/developers/post/cncf-reports-arm-credit-use-up-precipitously

[Itamar Holder]

On Thu, 24 Apr 2025 at 09:11, Luboslav Pivarc <lpi...@redhat.com> wrote:

On Wed, Apr 23, 2025 at 6:02 PM 'Itamar Holder' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

This is indeed very unfortunate.

I wonder what this means regarding ARM-related features.

How can we support / develop those without a proper test environment? Does it make sense to create an ARM lane from virtualized ARM kubevirtCI nodes?

Emulation would be most likely too slow if you meant emulation.

Yes, this is what I meant.

Maybe we can use emulation for basic configuration tests?

I just don't see how we can commit to supporting ARM without a test environment.

[Luboslav Pivarc]

Hey,

On Thu, Apr 24, 2025 at 10:18 AM Itamar Holder <iho...@redhat.com> wrote:

On Thu, 24 Apr 2025 at 09:11, Luboslav Pivarc <lpi...@redhat.com> wrote:

On Wed, Apr 23, 2025 at 6:02 PM 'Itamar Holder' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

This is indeed very unfortunate.

I wonder what this means regarding ARM-related features.

How can we support / develop those without a proper test environment? Does it make sense to create an ARM lane from virtualized ARM kubevirtCI nodes?

Emulation would be most likely too slow if you meant emulation.

Yes, this is what I meant.

Maybe we can use emulation for basic configuration tests?

I just don't see how we can commit to supporting ARM without a test environment.

+1 unfortunately.

-Lubo

[Luboslav Pivarc]

I will open a ticket with CNCF to explore what is possible.

-Lubo

 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/kubevirt-dev/CAPkJ9DtfhektdtLeP4iXCnM8xeOo8wx60J1USJUmJGc6qgaA-A%40mail.gmail.com.

[Brian Carey]

On Thu, Apr 24, 2025 at 11:13 AM Luboslav Pivarc <lpi...@redhat.com> wrote:

On Thu, Apr 24, 2025 at 10:15 AM 'Lee Yarwood' via kubevirt-dev <kubevi...@googlegroups.com> wrote:

On Wed, 23 Apr 2025 at 10:21, 'Brian Carey' via kubevirt-dev
<kubevi...@googlegroups.com> wrote:
>
> Hi everyone,
>
> As some of you may be aware, the arm64 test lanes in KubeVirt had been failing consistently for over 6 months now. We moved to stop reporting these failures on pull requests at the end of October 2024 as there wasn't a sign of improvement and to reduce noise for contributors on PRs[1].
>
> Recently we have learned that ARM plans on decommissioning the hardware that we were using for this testing so there is no plan for fixing these failures in the near future. Following this we have decided to remove the ARM test lanes from KubeVirt[2] as we don't have a working environment for these tests.
>
> Unfortunately this is quite a step backwards for the KubeVirt on ARM efforts and we currently don't know when we will have a reliable ARM environment for CI testing.
>
> These test lanes can be reintroduced when we have a stable ARM environment and the test lanes themselves pass consistently.
>
> Thanks,
> Brian
>
>
> [1] https://github.com/kubevirt/project-infra/pull/3526
> [2] https://github.com/kubevirt/project-infra/pull/4097

Are we able to request any ARM based compute resources from the CNCF
to power these lanes? I know various companies donate hardware or
credits into the CNCF, with Oracle for example donating Ampere
Arm®-based compute credits for their OCI cloud, but it looks like
these specific credits have already been exhausted for the year haha
[1].

[1] https://blogs.oracle.com/developers/post/cncf-reports-arm-credit-use-up-precipitously

I will open a ticket with CNCF to explore what is possible.

Thanks Lubo. 

We are investigating what different options are available to us for ARM testing but we don't have any concrete options yet. 

Regarding ARM support, as far as I remember we don't officially support KubeVirt on ARM yet. The builds for ARM are just made available without any guarantees. 

Thanks,

Brian