RFC: revisiting ironic-image and IrSO version support policy

22 views
Skip to first unread message

Dmitry Tantsur

unread,
Sep 9, 2025, 4:00:08 AMSep 9
to Metal3 Development List
Hi folks,

Prompted by the recent discussion about the OCI artifacts, I started pondering $subj, and I think we may need to tweak it (again?).

Currently per [1] we support 6 ironic releases (26.0 - 31.0, the latter not yet added to the docs), with 32.0 coming within a couple of weeks, making it 7. This is despite [1] explicitly saying that we support "two or three releases". Per Ironic policy [2], 26.0-28.0 are not supposed to be supported any more, we just did not bother EOL them there. I think we should take a more proactive position in Metal3.

There are two more angles to this problem:

1) BMO support. In BMO, our support for Ironic goes a long way back. The minimum API version is currently 1.81 from Ironic 21.3. This Ironic version has been decidedly unmaintained for quite a while. All these historical versions create a small but non-negligible maintenance burden and result in largely untested code (e.g. still lingering Inspector support).

2) IrSO support. IrSO supports several versions of Ironic by design and also has stable branches. We cannot EOL a version of ironic-image until a version of IrSO [3] exists that supports it. The oldest branch 0.4 supports Ironic 27.0-30.0.

So, here is a draft proposal, which mostly reiterates what seems to have been previously agreed, adding IrSO to the picture:

1) We always support *two* stable branches of IrSO (plus main). Meaning, once 0.6.0 is released, 0.4.0 is no longer supported. (As an aside, versions 0.1.0-0.3.0 did not have a branch and are not supported any more in any sense.) This matches the BMO policy.

2) Each version of IrSO supports *three* versions of ironic-image (plus latest). Meaning, right now IrSO should support 29.0-31.0. When adding 32.0, we'll remove 29.0. This rule won't be applied retroactively to 0.4 and 0.5 since we already promised support for them.

3) We support all versions of ironic-image that are supported by a still supported branch of IrSO.  This diverges from the current policy that does not account for IrSO. 0.4 and 0.5 support 27.0+, which means that this version stays for us for a bit longer. Once IrSO 0.7.0 is released and 0.5.0 is abandoned, we drop support for Ironic 27.0 and 28.0.

4) BMO supports *at least* all supported ironic-image versions and all API versions that come with them. Here is where I'm not sure and would like input from the community: we could go down a harsher path and stop supporting versions of Ironic that are not directly supported by ironic-image/IrSO. However, I know that some Metal3 consumers have Ironic installations that are tied to OpenStack stable releases, not ironic-image.

So here is a softer proposal: BMO also supports the newest API version that is available in the oldest maintained stable release per [4]. This is currently the 2024.1 "Caracal" series. Matching BMO features [5] against the API history [6], I get 1.89 as the oldest API version we should support now, dropping 1.81, 1.85 and 1.87.

5) Finally, the Inspector question. The new inspection was already available in 2024.1 releases of Ironic. It is also used by all currently supported versions of ironic-image. Thus, per the suggested BMO policy, I suggest we remove any support for Inspector from BMO and testing code.

On a related topic, should we start removing branches we no longer support to prevent people from accidentally relying on them?

Opinions?

Dmitry


-- 
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany  
Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty

Tuomo Tanskanen

unread,
Sep 10, 2025, 6:10:37 AMSep 10
to Metal3 Development List, Dmitry Tantsur
Hi Dmitry,

Good timing with this. I was discussing this with Kashif the other day that we need to bring this up as it is getting quite confusing. Let's discuss this also in community meeting today, but here is my 2 cents:

Proposal 1: Two stable + one more as "tested" in IRSO would match the CAPM3/IPAM/BMO policy.

Proposals 2 and 3: In theory this seems fine, but one problem that comes to mind is that IRSO/ironic-image has more frequent releases than CAPM3/IPAM/BMO. For the e2e, after making it use IRSO instead of BMO manifests for Ironic deployments, it would mean that supported branches of CAPM3/IPAM/BMO would either need to be uplifted with newer IRSO or try stay afloat with unsupported one. This could lead to interested changes required in case we have incompatibilities in IRSO.

Proposal 4:  Second problem that will eventually fade away is this BMO + Ironic-image "marriage" via manifests existing in BMO stable branches, but we need to some plan how to manage transition. And as initial gut reaction, I'd say we could drop support for Ironic API versions we don't support in Ironic-image. I'm fine with hard option, but I'm not against if someone has reasons to do it soft way.

Proposal 5: I thought we were done with it already, lol. +1 for cleanup.

As for the bonus question, we don't want to remove branches. We just remove CI from them and lock them so no further edits can be done.

Cheers, Tuomo

From: 'Dmitry Tantsur' via Metal3 Development List <metal...@googlegroups.com>
Sent: 09 September 2025 10:59
To: Metal3 Development List <metal...@googlegroups.com>
Subject: [metal3-dev] RFC: revisiting ironic-image and IrSO version support policy
 
--
You received this message because you are subscribed to the Google Groups "Metal3 Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metal3-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/metal3-dev/CACNgkFzOzywsqgjSSGwDgA2oth7piCmRorhYwudThp5g3TP5gw%40mail.gmail.com.

Dmitry Tantsur

unread,
Sep 10, 2025, 11:30:23 AMSep 10
to Metal3 Development List
I've created an ascii diagram to illustrate the problem about overlapping releases (one unit here is one month; use a small fixed width font to make any sense out of it):

IrSO 0.5   <##|###|                          has Ironic 27 - 31

IrSO 0.6  |###|###|###|###|                  has Ironic 30, 31, 32

IrSO 0.7  |   |   |###|###|###|###|          has Ironic 31, 32, 33

IrSO 0.8  |   |   |   |   |###|###|###|###|  has Ironic 32, 33, 34

Ironic 30  <##|###|###|###|                          EOL with IrSO 0.6

Ironic 31  <##|###|###|###|###|###|                  EOL with IrSO 0.7

Ironic 32 |###|###|###|###|###|###|###|###|          EOL with IrSO 0.8

Ironic 33 |   |   |###|###|###|###|###|###|###|###|  

BMO 0.10   <##|###|###|###|

BMO 0.11  |###|###|###|###|###|###|###|###|                  starts with IrSO 0.6, Ironic 32

BMO 0.12  |   |   |   |   |###|###|###|###|###|###|###|###|  starts with IrSO 0.8, Ironic 34


So yeah, a few issues:
1. Ironic Image releases are supported 8 months (upstream only 6)
2. Three IrSO releases happen during the lifetime of a single BMO release.
3. Only one Ironic version is supported throughout the entire lifetime of a single BMO release.
On the positive side, Ironic Image schedules match the BMO's one.

Potential ways out:
1. Upgrade IrSO on BMO stable branches. E.g. 0.11 will start* with IrSO 0.6 and will be upgraded to 0.8, using Ironic 32. This requires that IrSO release 0.N keeps compatibility with 0.N-2, so the deprecation period will be 3 cycles or 8 months. It's quite annoying for a young project but doable.
2. Release IrSO less frequently, i.e. match BMO. Then we need to decide what to do about the version of Ironic that is released in-between IrSO releases. E.g. if we skip 0.7, then support for Ironic 33 will only come together with 34, so realistically 33 will only be used for upgrades to 34. Basically, IrSO consumers would only use even releases** of Ironic, jumping over odd releases while they upgrade. The upgrade process would need polishing.
3. Backport Ironic support to stable branches of IrSO. This may involve changes to accommodate any ironic-images changes on 'main'. This is doable, but this idea changes the definition of a stable branch. Also requires a longer deprecation period in ironic-image itself to minimize changes to stable branches of IrSO.
4. Release BMO more frequently, i.e. match Ironic. Get new features out more quickly. It's unclear which impact it has on co-releasing with CAPM3. In any case, BMO will have to follow a two-release deprecation policy since CAPM3 consumers will only see every other BMO release.
5. Support IrSO releases for 8 months like BMO. Given the release frequency, it means supporting 4 stable branches at the same time. On top of that, it will increase Ironic releases lifetime to unsustainable values, unless we decide that IrSO is allowed to drop Ironic version support on a stable branch (which is.. not exactly stable?).

IrSO 0.5   <##|###|###|###|                          has Ironic 27 - 31

IrSO 0.6  |###|###|###|###|###|###|###|###|          has Ironic 30, 31, 32

IrSO 0.7  |   |   |###|###|###|###|###|###|###|###|  has Ironic 31, 32, 33

IrSO 0.8  |   |   |   |   |###|###|###|###|###|##>   has Ironic 32, 33, 34

Ironic 30  <##|###|###|###|???|???|???|???|          EOL 4 months into IrSO 0.6?

Ironic 31  <##|###|###|###|###|###|???|???|???|???|  EOL 4 months into IrSO 0.7?

Ironic 32 |###|###|###|###|###|###|###|###|???|??>   EOL 4 months into IrSO 0.8?

Ironic 33 |   |   |###|###|###|###|###|###|###|##>  


Thoughts on these options? Any options I'm missing?

* 0.11 will not actually have IrSO in its CI probably; I'm ignoring it for the sake of the big picture
** I'm assuming major version bumps, in reality we may have 31.1, etc.

Tuomo Tanskanen

unread,
Sep 18, 2025, 12:47:39 PM (8 days ago) Sep 18
to Metal3 Development List, Dmitry Tantsur
Hi Dmitry, all,

I've been super busy this weke, so I will think about this for a sec further, but will also pitch in two more things into the mix:

  1. We need to start doing patch releases for IRSO and Ironic-image as well

For IRSO, it should be nice and clean operation with the current release workflow, but for Ironic-image we haven't had more than one patch release over any minors so far and that was for CVE fix, yet it would benefit greatly as Ironic-image is pulling both Centos and Ironic during build time and is not pinned. The latest tags are thus full of very old packages, and we'll receive fixes beyond the merged PRs in the branches when released. There should be no issue with this one, its just not happening right now.

  1. Since it kinda affects releasing, I'll also re-insert the "monorepo" idea into the discussion

We have so much duplication with our controller repos, codewise but especially CI wise that I'd love to see our controllers merge into one repo Kubernetes style, to share the code/CI machinery and simplify things a lot (and complicate some for sure). This would lead of course into coordinated release between IRSO/BMO/CAPM3/IPAM and could be called just Metal3 release.  ...  That would be a major change in organizing the code, and the releases as well, so before we lock in the releasing I wanted to get the idea out. Maybe it is a bad one, maybe it is a good one, give it a through from your POV.

Cheers, Tuomo

From: 'Dmitry Tantsur' via Metal3 Development List <metal...@googlegroups.com>
Sent: 10 September 2025 18:30

To: Metal3 Development List <metal...@googlegroups.com>
Subject: Re: [metal3-dev] RFC: revisiting ironic-image and IrSO version support policy
 

Dmitry Tantsur

unread,
Sep 19, 2025, 8:18:23 AM (7 days ago) Sep 19
to Tuomo Tanskanen, Metal3 Development List
On Thu, Sep 18, 2025 at 6:47 PM Tuomo Tanskanen <tuomo.t...@est.tech> wrote:
Hi Dmitry, all,

I've been super busy this weke, so I will think about this for a sec further, but will also pitch in two more things into the mix:

  1. We need to start doing patch releases for IRSO and Ironic-image as well

For IRSO, it should be nice and clean operation with the current release workflow, but for Ironic-image we haven't had more than one patch release over any minors so far and that was for CVE fix, yet it would benefit greatly as Ironic-image is pulling both Centos and Ironic during build time and is not pinned. The latest tags are thus full of very old packages, and we'll receive fixes beyond the merged PRs in the branches when released. There should be no issue with this one, its just not happening right now.

+1
 

  1. Since it kinda affects releasing, I'll also re-insert the "monorepo" idea into the discussion

We have so much duplication with our controller repos, codewise but especially CI wise that I'd love to see our controllers merge into one repo Kubernetes style, to share the code/CI machinery and simplify things a lot (and complicate some for sure). This would lead of course into coordinated release between IRSO/BMO/CAPM3/IPAM and could be called just Metal3 release.  ...  That would be a major change in organizing the code, and the releases as well, so before we lock in the releasing I wanted to get the idea out. Maybe it is a bad one, maybe it is a good one, give it a through from your POV.

Keep in mind that not everyone is using the same set of components.

Furthermore, does it really solve the problem? We can have the same release cadence even without a monorepo, I've listed two possible options. But do you really want to tie CAPM3 releases to Ironic ones or Ironic-Images releases to CAPI?

Dmitry

Lennart Jern

unread,
Sep 19, 2025, 8:48:09 AM (7 days ago) Sep 19
to Tuomo Tanskanen, Dmitry Tantsur, Metal3 Development List
Hi!

Regarding monorepo, I am torn. I can see the value in having release workflows centralized, not having to create PRs in multiple repositories just to update versions and release branches. On the other hand, it is nice to have separate release branches for separate components so that each can have their own cadence.

I think we would still need separate tags, e.g. for CAPM3 and BMO since their APIs are changing separately. Otherwise we would end up with a new "major" Metal3 version all the time. First CAPM3 goes to v1beta2, then BMO releases v1beta1, for example.
Kustomize is one project that has this way of releasing multiple tags from one repo and honestly it feels confusing to me.

Maybe before we go as far as monorepo, we should try to simplify the CI in other ways.

My pet peeve is metal3-dev-env. It needs constant updating with all the release branches in all the sub-projects. A constant hassle. It breaks quite often and since all the CAPM3 e2e tests depend on metal3-dev-env still, all of that is then broken.
Furthermore, because of the central metal3-dev-env way of running the tests, we have project-infra that defines the Jenkins pipelines. Also these need constant updating with variables and defaults, sometimes even going through the jenkins job builder repo.

Where am I going? If we get rid of metal3-dev-env (at least for CI use), we no longer need to have Jenkins pipelines in project-infra, they can be directly in the relevant repo. Suddenly we don't have the three level indirection jjb -> project-infra -> metal3-dev-env -> CAPM3 e2e. With that we should have much less to update. But we will then need proper e2e tests for each repo. For CAPM3 I think it is already in progress. For IPAM we need to start. Ironic-image is tested with IrSO, and BMO has its own e2e so in that regard we are good.

Br,
Lennart
From: 'Dmitry Tantsur' via Metal3 Development List <metal...@googlegroups.com>
Sent: Friday, September 19, 2025 15:18
To: Tuomo Tanskanen <tuomo.t...@est.tech>
Cc: Metal3 Development List <metal...@googlegroups.com>

Tuomo Tanskanen

unread,
Sep 24, 2025, 9:55:28 AM (2 days ago) Sep 24
to Lennart Jern, Dmitry Tantsur, Metal3 Development List
OK, I guess these are the only immediate comments. 

  • I think we would still need separate tags, e.g. for CAPM3 and BMO since their APIs are changing separately. Otherwise we would end up with a new "major" Metal3 version all the time. First CAPM3 goes to v1beta2, then BMO releases v1beta1, for example.
  • Kustomize is one project that has this way of releasing multiple tags from one repo and honestly it feels confusing to me.

Do they need to be, and is it a really problem, even if they do? For example, we have Kubernetes with way more components and API versions than us, and they can manage it just fine?

  • My pet peeve is metal3-dev-env. It needs constant updating with all the release branches in all the sub-projects. A constant hassle. It breaks quite often and since all the CAPM3 e2e tests depend on metal3-dev-env still, all of that is then broken.
  • Furthermore, because of the central metal3-dev-env way of running the tests, we have project-infra that defines the Jenkins pipelines. Also these need constant updating with variables and defaults, sometimes even going through the jenkins job builder repo.

Having most of the code in one repo, we could get rid of dev-env as well, as that is mostly integration glue between repos. We're already duplicating the code in CAPM3 for CAPM3 e2e, which shows it could in the Metal3 repo all along. We have now separate e2e environments for dev-env, CAPM3, BMO and IRSO, which is MAJOR pain in the buttocks. Dmitry is on-record in CAPM3 e2e PR asking why are we duplicating things. Having per component e2e is what we need, but doing it by duplicating half of dev-env in each repo is not the way IMO.

  • Where am I going? If we get rid of metal3-dev-env (at least for CI use), we no longer need to have Jenkins pipelines in project-infra, they can be directly in the relevant repo. Suddenly we don't have the three level indirection jjb -> project-infra -> metal3-dev-env -> CAPM3 e2e. With that we should have much less to update. But we will then need proper e2e tests for each repo. For CAPM3 I think it is already in progress. For IPAM we need to start. Ironic-image is tested with IrSO, and BMO has its own e2e so in that regard we are good.

Indeed this is the monster we need to get rid of. Each minor release and each change that goes cross-repo is painful, hard to coordinate and often hard to even test (in CI) as you need to coordinate multiple repos.

  • Keep in mind that not everyone is using the same set of components.

How is that an issue? Consumer is free to take the pieces they want, even if components share a codebase or a release number. Note: I'm not suggesting we put ironic-image to that repo, but 4 of our controllers (CAPM3, BMO, IPAM and IRSO). I'm also not suggesting we're slamming them in one controller or one Docker image, I'm suggesting we put the code from our controller repos into one and use the synergy the reuse the codebase and the CI for major simplification of project structure.

  • Furthermore, does it really solve the problem? We can have the same release cadence even without a monorepo, I've listed two possible options. But do you really want to tie CAPM3 releases to Ironic ones or Ironic-Images releases to CAPI?

From my POV, it would solve quite a few problems. Some are mentioned above (CI), but we'd also get to share controller code and not implement same things four times over, have some bump PRs merged four times over, scan same things four times over etc. It would have a lot of synergies, both CI wise, but also code wise, and day to day operation wise. We at ESJ feel this pain a lot, and it is wasting a lot of time and effort.

It would also solve some perception/visibility issues as we don't have a clear main repo. We have somewhere CAPM3, somewhere BMO, etc. Whether we want it or not, but were monitored and scored in many places based on the main repo alone. CNCF project health, LFX insights, etc. This might not seem much to some of you, but community health and diversity and contributor experience etc are major things for the project when we aim to for the eventual graduation.

For the downsides, we definitely have the issue of Ironic release schedule, but there are many ways to deal with that. We can still release that each time upstream Ironic releases. In monorepo, we can just take the latest or most stable, or whatever we deem most fit during the development cycle. Or we can just add support to IRSO in a patch release (yes, its un-semver like, but hey, CAPI adds new k8s support in .1 release too). That should not be a blocker for re-structuring Metal3 IMO.

That said, this is not urgent matter, and does not need nor can't be decided over email or even in community meeting. But please give it a honest thought from a wider perspective.

Cheers, Tuomo

From: Lennart Jern <lennar...@est.tech>
Sent: 19 September 2025 15:48
To: Tuomo Tanskanen <tuomo.t...@est.tech>; Dmitry Tantsur <dtan...@redhat.com>
Reply all
Reply to author
Forward
0 new messages