Invitation for brainstorming: delivering IPA images via OCI artefacts

[Dmitry Tantsur]

Hi folks!

This is a continuation of our discussion in the last team meeting. It concerns the future of delivering IPA images for Metal3 and is independent of the recent issues around caching downloads from tarballs.opendev.org.

# Problem space

For years, we have relied on the IPA-downloader [1] initContainer to download IPA images from the upstream location [2] where they're built each time a commit happens in ironic-python-agent. This approach has numerous issues:

1. Tarballs.opendev.org is not a real CDN and can be quite slow, especially from certain locations (like github runners, unfortunately). We're working on a transparent proxy to mitigate this for our CI, but the consumers of Metal3 may experience the same issue.

2. Official images that we use are x86-64 only. There are also images for aarch64 built at a different location [3] which we don't yet support.

3. Official images are only built for major OpenStack releases and the main branch. ARM images are only built for the main branch. We have zero influence on the images our users consume.

4. Using custom images is confusing and error prone: you need to set several variables in an exact way. Multi-arch support makes this issue even worse.

5. Using a local cache (think: disconnected case) requires a separate HTTP server.

6. Since Quay is our official distribution site, users expect all our official artifacts to come from it. The OpenDev location may not even be allowed on firewalls.

# Proposal

Ironic 29.0 and newer supports [4] downloading images from oci:// URLs referring to OCI Artefacts [5]. These artefacts behave pretty similarly to container images: they are essentially manifests that point to binary blobs. They can have tags and meta-information, such as CPU architecture.

As an aside, using OCI artefacts as instance images works today, as long as the registry does not require authentication. Just set image.url to oci://host/namespace/container:tag (or better use the hash).

We could have a periodic job in our Jenkins that takes the upstream images and uploads them as artefacts to our Quay namespace. Then we could update ironic-image to default to something like (subject to further bikeshedding on what becomes a part of the name and what of the tag):

deploy_kernel = oci://quay.io/metal3-io/ipa-kernel-centos9:latest

deploy_ramdisk = oci://quay.io/metal3-io/ipa-initramfs-centos9:latest

For stable branches it will become

deploy_kernel = oci://quay.io/metal3-io/ipa-kernel-centos9:release-32.0

deploy_ramdisk = oci://quay.io/metal3-io/ipa-initramfs-centos9:release-32.0

even if the tag points to the same manifest as latest.

## Limitations

The Ironic documentation claims that tags are not supported for deploy_kernel/ramdisk. We had a quick discussion yesterday, and nobody could remember the reason, nor find evidence in the code that it's actually the case. Needs double-checking and potentially fixing.

I'm not sure if the code is multi-arch aware, i.e. it will fetch the right manifest depending on the architecture of the BMH. Again, something to investigate and potentially fix. Worst case scenario, fall back to different tags and use deploy_kernel_by_arch.

Finally, we need to investigate how Ironic's cache behaves with such images. Normal HTTP images are cached locally and validated each time using a HEAD request to the server. Ideally, Ironic needs to check if the tag still resolves to the same hash. I don't know if it does. We can check it and fix it if necessary.

## How it solves the problems above

1. Quay is designed to distribute artefacts to users. High-scale users can easily set up a mirror.

2. Path forward for multi-arch support. Our Jenkins job can pull images from different locations or even build them.

3. Eventually, we may start building IPA images ourselves and provide different images for different Ironic branches.

4. Custom images will be easy: upload your own artefacts to your registry, point ironic-image to it. I expect IrSO to make it even easier.

5. Disconnected users will already have a registry for containers, no extra service is needed for IPA Images.

6. Everything comes from quay.io/metal3-io/*, less confusion.

Bonus: IPA downloader becomes entirely optional and may even eventually be phased out.

Thoughts?

Thanks for reading so far!

Dmitry

[1] https://github.com/metal3-io/ironic-ipa-downloader

[2] https://tarballs.opendev.org/openstack/ironic-python-agent/dib/

[3] https://tarballs.opendev.org/openstack/ironic-python-agent-builder/dib/

[4] https://docs.openstack.org/ironic/latest/admin/oci-container-registry.html

[5] https://oras.land/docs/concepts/artifact/

--

Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty 

[Ádám Rozmán]

Hi Dmitry

I am in support of the OCI initiative from the beginning maybe even earlier, I think that should be the #1 way of transfering artifacts in a K8s environment.

I think we have the CI capacity and we have the knwoledge to build custom IPAs as we do it already via Jenkins, it is easy to extend our pipeline to to build tarballs and OCI images at the same time.

Hosting OCI images in Quay also makes sense as the Metal3 project already does that and it works for all our use cases.

The only thing I would say is that even though OCI would be the choice in most scenarios I think the tarballs have their place too so IMO we will need to provide both options when we start officially offering our own Metal3 IPA images, but I see this as very inexpensive process in terms of CI resources.

I think we already have an issue for this related to caching in IrSO https://github.com/metal3-io/ironic-standalone-operator/issues/27 and in BMO related to node images https://github.com/metal3-io/baremetal-operator/issues/2332  so we might need to start an organization level GH discussion thread to link all the isses of different repos to improve coordination ( I can create it, if you are ok with that). 

From next community meeting I will announce the "Artifact" working group, or you can propose other name I but I think there is enough substance here to facilitate a working group.

As we have discussed during the community meeting I am also fine elevating this topic as one of the Metal3 Team Meetup topics.

Br,
Adam

---

From: 'Dmitry Tantsur' via Metal3 Development List <metal...@googlegroups.com>
Sent: Thursday, September 4, 2025 4:52 PM
To: Metal3 Development List <metal...@googlegroups.com>
Subject: [metal3-dev] Invitation for brainstorming: delivering IPA images via OCI artefacts

 

--
You received this message because you are subscribed to the Google Groups "Metal3 Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metal3-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/metal3-dev/CACNgkFyjRBEXiO9KK43TEND30x7tWgiQqOG%2BYbMJoaD%3Dyr9gSQ%40mail.gmail.com.

[Nicolas Belouin]

Hi Dmitry,

I think it would be great to support this using builtin Ironic features, but I don't think this would be doable in the near future, as it will require some new features in Ironic and we probably don't want to drop support of older versions of Ironic.

First I checked in Ironic codebase and the OciImageService that is responsible for pulling the OCI artifacts, when used in the IPA kernel/ramdisk context it will use the download method (ironic/common/image_service.py line 450 [1]) that first checks that the given URL is specific (i.e points directly to a manifest using it's hash)

But I think that since it would need Ironic development anyway, we could also try to get the architecture awareness there (the deploy image with oci is architecture aware, but not the IPA kernel/ramdisk part).

I have the other linked question, do we want to stop at IPA kernel and ramfs this way, or should we extend it to EFI bootloader as well ?

Another limitation is that while with the IPA downloader one can "easily" inject additional files into it, we do it to inject CA files to be able to have ipa-insecure to false [2], using builtin OCI may make this harder.

Regarding still distributing tarballs, I'm not opposed to it, but I think that we should get the IPA downloader to be able to pull from oci artifacts as well (in the same way), to account for older Ironic versions, so this may make the tarballs unneeded.

Regards,

Nicolas

[1] https://opendev.org/openstack/ironic/src/branch/master/ironic/common/image_service.py#L450

[2] https://src.opensuse.org/suse-edge/Factory/src/branch/main/ironic-ipa-downloader-image/get-resource.sh#L98