Brainstorming: preventing UX issues on forced deletion of BMH
24 views
Skip to first unread message
Dmitry Tantsur
Jun 6, 2025, 2:54:16 AMJun 6
to Metal3 Development List
Hi team,
I'm not sure about everyone here, but I regularly get complaints from customers that use forced deletion of BMH objects as a way to "fix" problems like stuck cleaning. As explained in our troubleshooting FAQ [1], it may cause issues on re-registration because of the stale node record in Ironic.
The FAQ suggests deleting and re-creating the Ironic pod. But this "solution" is not just pretty inconvenient, it won't work when a real database backed by a persistent volume is used. I'd like to open a brainstorming session on solving this issue.
My immediate idea is to introduce a new internal CRD IronicNode that is created for any node that a BMH is backed by. It will be linked to the BMH by an ownerReference and will also have a finalizer. If the IronicNode controller finds that the BMH is gone, it will try to get rid of the corresponding Node as soon as possible. This will include aborting cleaning/inspection/deployment to get the Node into some deletable state.
This addition may have a positive side effect of splitting the BMH controller into two controllers, one handling high-level and one low-level provisioning logic. It does provide an easy way to bypass cleaning, which I don't particularly like. But so does the current workaround of removing the pod.
Does anyone have any better ideas? And what do you think about this idea?
Dmitry
--
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty
Zane Bitter
Jun 6, 2025, 8:19:52 AMJun 6
to metal...@googlegroups.com
On 6/06/25 18:53, 'Dmitry Tantsur' via Metal3 Development List wrote:
> Hi team,
>
> I'm not sure about everyone here, but I regularly get complaints from
> customers that use forced deletion of BMH objects as a way to "fix"
> problems like stuck cleaning. As explained in our troubleshooting FAQ
> [1], it may cause issues on re-registration because of the stale node
> record in Ironic.
>
> The FAQ suggests deleting and re-creating the Ironic pod. But this
> "solution" is not just pretty inconvenient, it won't work when a real
> database backed by a persistent volume is used. I'd like to open a
> brainstorming session on solving this issue.
>
> My immediate idea is to introduce a new internal CRD IronicNode that is
> created for any node that a BMH is backed by. It will be linked to the
> BMH by an ownerReference and will also have a finalizer. If the
> IronicNode controller finds that the BMH is gone, it will try to get rid
> of the corresponding Node as soon as possible. This will include
> aborting cleaning/inspection/deployment to get the Node into some
> deletable state.
Users will just delete the finalizer on the IronicNode.
> Hi team,
>
> I'm not sure about everyone here, but I regularly get complaints from
> customers that use forced deletion of BMH objects as a way to "fix"
> problems like stuck cleaning. As explained in our troubleshooting FAQ
> [1], it may cause issues on re-registration because of the stale node
> record in Ironic.
>
> The FAQ suggests deleting and re-creating the Ironic pod. But this
> "solution" is not just pretty inconvenient, it won't work when a real
> database backed by a persistent volume is used. I'd like to open a
> brainstorming session on solving this issue.
>
> My immediate idea is to introduce a new internal CRD IronicNode that is
> created for any node that a BMH is backed by. It will be linked to the
> BMH by an ownerReference and will also have a finalizer. If the
> IronicNode controller finds that the BMH is gone, it will try to get rid
> of the corresponding Node as soon as possible. This will include
> aborting cleaning/inspection/deployment to get the Node into some
> deletable state.
> This addition may have a positive side effect of splitting the BMH
> controller into two controllers, one handling high-level and one low-
> level provisioning logic. It does provide an easy way to bypass
> cleaning, which I don't particularly like. But so does the current
> workaround of removing the pod.
>
> Does anyone have any better ideas? And what do you think about this idea?
having some sort of process that periodically scans the ironic nodes and
deletes any that aren't associated with a BMH?
Or even simpler, store the UID of the BMH somewhere in the ironic Node
(which you'd probably have to do to safely implement the above anyway),
and then when someone creates a new BMH with the same name delete any
existing node that references a different UID.
(I want to say that some version of this was first suggested by Mahnoor,
but I can't find where so I assume that any good parts in the idea were
hers and the bad parts were mine.)
- ZB
Dmitry Tantsur
Jun 6, 2025, 11:27:44 AMJun 6
to Zane Bitter, metal...@googlegroups.com
Thank you for chiming in!
On Fri, Jun 6, 2025 at 2:19 PM 'Zane Bitter' via Metal3 Development List <metal...@googlegroups.com> wrote:
On 6/06/25 18:53, 'Dmitry Tantsur' via Metal3 Development List wrote:
> Hi team,
>
> I'm not sure about everyone here, but I regularly get complaints from
> customers that use forced deletion of BMH objects as a way to "fix"
> problems like stuck cleaning. As explained in our troubleshooting FAQ
> [1], it may cause issues on re-registration because of the stale node
> record in Ironic.
>
> The FAQ suggests deleting and re-creating the Ironic pod. But this
> "solution" is not just pretty inconvenient, it won't work when a real
> database backed by a persistent volume is used. I'd like to open a
> brainstorming session on solving this issue.
>
> My immediate idea is to introduce a new internal CRD IronicNode that is
> created for any node that a BMH is backed by. It will be linked to the
> BMH by an ownerReference and will also have a finalizer. If the
> IronicNode controller finds that the BMH is gone, it will try to get rid
> of the corresponding Node as soon as possible. This will include
> aborting cleaning/inspection/deployment to get the Node into some
> deletable state.
Users will just delete the finalizer on the IronicNode.
I'm not really trying to protect us against really malicious users, more about ignorant ones.
The IronicNode resource would not do any retries or anything that is not on the shortest path to deletion. So it will be somewhat different from BareMetalHost now. On the other hand, it won't be that visible to users.
> This addition may have a positive side effect of splitting the BMH
> controller into two controllers, one handling high-level and one low-
> level provisioning logic. It does provide an easy way to bypass
> cleaning, which I don't particularly like. But so does the current
> workaround of removing the pod.
>
> Does anyone have any better ideas? And what do you think about this idea?
Since it doesn't appear we will ever try to do discovery, what about
having some sort of process that periodically scans the ironic nodes and
deletes any that aren't associated with a BMH?
Upstream Metal3 has never made it a requirement that no other nodes are enrolled in Ironic. And without Keystone support, we don't even have tenant separation in Ironic.
Or even simpler, store the UID of the BMH somewhere in the ironic Node
(which you'd probably have to do to safely implement the above anyway),
and then when someone creates a new BMH with the same name delete any
existing node that references a different UID.
The problem is not with a duplicated name, it's with a duplicated MAC. And we need to be able to distinguish between actually duplicated MACs (e.g. because two BMHs have the same MAC) and the situation in question.
Dmitry
(I want to say that some version of this was first suggested by Mahnoor,
but I can't find where so I assume that any good parts in the idea were
hers and the bad parts were mine.)
- ZB
--
You received this message because you are subscribed to the Google Groups "Metal3 Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metal3-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/metal3-dev/e1c30d9d-e823-4dd7-9668-5ab935b431ea%40redhat.com.
Zane Bitter
Jun 8, 2025, 8:31:15 PMJun 8
to metal...@googlegroups.com
MAC and we should reject it (possibly the webhook does this already?).
If the BMH with that UID doesn't exist, it's a stale Node that should be
cleared out.
> Dmitry
>
>
> (I want to say that some version of this was first suggested by
> Mahnoor,
> but I can't find where so I assume that any good parts in the idea were
> hers and the bad parts were mine.)
>
> - ZB
>
> --
> You received this message because you are subscribed to the Google
> Groups "Metal3 Development List" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to metal3-dev+...@googlegroups.com
> To view this discussion visit https://groups.google.com/d/msgid/
> metal3-dev/e1c30d9d-e823-4dd7-9668-5ab935b431ea%40redhat.com
> <https://groups.google.com/d/msgid/metal3-dev/e1c30d9d-
> metal3-dev/e1c30d9d-e823-4dd7-9668-5ab935b431ea%40redhat.com
> e823-4dd7-9668-5ab935b431ea%40redhat.com>.
>
>
>
> --
>
> Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany
Dmitry Tantsur
Jun 16, 2025, 11:23:26 AMJun 16
to Zane Bitter, metal...@googlegroups.com
On Mon, Jun 9, 2025 at 2:31 AM 'Zane Bitter' via Metal3 Development List <metal...@googlegroups.com> wrote:
On 7/06/25 03:27, Dmitry Tantsur wrote:
> Thank you for chiming in!
>
> On Fri, Jun 6, 2025 at 2:19 PM 'Zane Bitter' via Metal3 Development List
> <metal...@googlegroups.com <mailto:metal...@googlegroups.com>> wrote:
>
> On 6/06/25 18:53, 'Dmitry Tantsur' via Metal3 Development List wrote:
> > Hi team,
> >
> > I'm not sure about everyone here, but I regularly get complaints
> from
> > customers that use forced deletion of BMH objects as a way to "fix"
> > problems like stuck cleaning. As explained in our troubleshooting
> FAQ
> > [1], it may cause issues on re-registration because of the stale
> node
> > record in Ironic.
> >
> > The FAQ suggests deleting and re-creating the Ironic pod. But this
> > "solution" is not just pretty inconvenient, it won't work when a
> real
> > database backed by a persistent volume is used. I'd like to open a
> > brainstorming session on solving this issue.
<snip>
How do we know this node belongs to Metal3 at all?
Dmitry
> Dmitry
>
>
> (I want to say that some version of this was first suggested by
> Mahnoor,
> but I can't find where so I assume that any good parts in the idea were
> hers and the bad parts were mine.)
>
> - ZB
>
> --
> You received this message because you are subscribed to the Google
> Groups "Metal3 Development List" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to metal3-dev+...@googlegroups.com
> <mailto:metal3-dev%2Bunsu...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/
> metal3-dev/e1c30d9d-e823-4dd7-9668-5ab935b431ea%40redhat.com
> <https://groups.google.com/d/msgid/metal3-dev/e1c30d9d-
> e823-4dd7-9668-5ab935b431ea%40redhat.com>.
>
>
>
> --
>
> Red Hat GmbH <https://www.redhat.com/de/global/dach>, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany
> Commercial register: Amtsgericht Muenchen/Munich, HRB 153243,
> Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty
>
--
You received this message because you are subscribed to the Google Groups "Metal3 Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metal3-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/metal3-dev/dc8a3063-545e-4703-95ce-c0357da879f1%40redhat.com.
--
Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Avril Crosse O'Flaherty
Zane Bitter
Jun 16, 2025, 4:39:05 PMJun 16
to Dmitry Tantsur, metal...@googlegroups.com
On 17/06/25 03:23, Dmitry Tantsur wrote:
> > And we need to be able to distinguish between actually duplicated
> MACs
> > (e.g. because two BMHs have the same MAC) and the situation in
> question.
>
> We can check the UID. If it belongs to another BMH, it's a duplicated
> MAC and we should reject it (possibly the webhook does this already?).
> If the BMH with that UID doesn't exist, it's a stale Node that
> should be
> cleared out.
>
>
> How do we know this node belongs to Metal3 at all?
Presumably nothing else will be annotating the Node with a BMH UUID. So:
> > And we need to be able to distinguish between actually duplicated
> MACs
> > (e.g. because two BMHs have the same MAC) and the situation in
> question.
>
> We can check the UID. If it belongs to another BMH, it's a duplicated
> MAC and we should reject it (possibly the webhook does this already?).
> If the BMH with that UID doesn't exist, it's a stale Node that
> should be
> cleared out.
>
>
> How do we know this node belongs to Metal3 at all?
No UUID present -> not our Node; reject as duplicated MAC
UUID matches extant BMH -> Node belongs to that BMH; reject as
duplicated MAC
UUID doesn't match any extant BMH -> Stale Node; delete it
I guess this doesn't handle the case where there are multiple Metal³
installations pointed at the same Ironic though. If that's a real use
case I'm not sure how you handle it, because it's difficult to think of
a stable, unique identifier for a Metal³ deployment.
- ZB
Reply all
Reply to author
Forward
0 new messages