On cleaning
60 views
Skip to first unread message
Zane Bitter
Nov 13, 2020, 3:59:51 PM11/13/20
to Metal3 Development List
I recently had cause to start looking a bit more deeply into how
cleaning works in Metal³. This is something I had been confused about
for a while, and now that I figured it out I think it has some
implications for how we implement some future features, so I wanted to
take a moment to document here for everyone.
It's actually not obvious what's happening from the state machine
diagram[1] because there is some hidden state - the next state sometimes
depends on the previous state as well as the current state.
The basic idea of the ironic state machine is that when a node is not
provisioned it is in the "available" state, and when it is provisioned
it is in the "active" state. In normal use a node flips between these
two states. Automated cleaning happens when flipping from "active" to
"available". There's one additional subtlety: when the node is first
created automated cleaning also happens when moving from "manageable" to
"available". Basically automated cleaning always happens before moving
into the "available" state:
[Manageable]
|
V
Automated
Cleaning <---------+
| |
V |
[Available] | Deprovision
| |
| Provision |
V |
[Active] ---------+
How Metal³ currently works is... not like that. On deprovisioning we
wait for the automated cleaning to happen, then when the node reaches
the "available" state we move it back to "manageable". This means that
if we provision the host again we have to clean it a second time, even
though we just cleaned it.
I submit that this is not what we want, and I have proposed a patch[2]
to change it in this PR:
https://github.com/metal3-io/baremetal-operator/pull/716
The final subtlety is that there also exists "manual cleaning",[3] and
this is required to change some settings including and possibly(?) not
limited to RAID. And manual cleaning can only be triggered from the
"manageable" state. If you want to change the RAID setup between
deployments, you have to run 3 cleanings (automated, then manual, then
automated). There doesn't appear to be a way around this.
In light of this, what if we implemented a new Preparing state in Metal³
that occurs *before* the Ready state:
Inspecting -> Preparing -> Ready -> Provisioning -> Provisioned
^ | ^ |
| | | |
+--------+ +----Deprovisioning <------+
In this Preparing state we would copy over the RAID (and BIOS?) settings
and note them in the status. From the Ready state we would return to
Preparing if and only if the settings in the spec differed from the
status. Provisioning would begin only once the status matches the spec.
This differs from the current RAID proposal, which has a Preparing state
*after* the Ready state (immediately before Provisioning). If you always
provision a host immediately whenever it is available, this makes no
difference. But if you don't then it means that an available host is
always ready to be provisioned immediately without waiting for cleaning.
In this scenario, if the RAID settings don't change then it also saves
an extra manual + automated cleaning. However, that may be a bad thing:
there was some discussion that we are concerned that a user could mess
up the RAID configuration while provisioned, and that therefore we
should always reconfigure it after deprovisioning. Dmitry suggested[4]
that ironic could allow us to delete the current config during
deprovisioning. When that becomes possible, we could implement it by
clearing the RAID data in the status during the Deprovisioning state.
I think this also has the effect of eliminating any dependencies between
the RAID implementation and Dmitry's proposal[5] to add provisioning
sub-states for progress reporting. We could still potentially merge the
RAID patch as-is and change the state machine later, subject to
confirming that no API changes would be needed.
Thoughts?
cheers,
Zane.
[1] https://docs.openstack.org/ironic/latest/contributor/states.html
[2]
https://github.com/metal3-io/baremetal-operator/pull/716/commits/18465130c8f42bda4aa8191b5093656dcc029d90
[3] https://docs.openstack.org/ironic/latest/admin/cleaning.html
[4] https://github.com/metal3-io/metal3-docs/pull/134/files#r492620912
[5] https://github.com/metal3-io/metal3-docs/pull/138
cleaning works in Metal³. This is something I had been confused about
for a while, and now that I figured it out I think it has some
implications for how we implement some future features, so I wanted to
take a moment to document here for everyone.
It's actually not obvious what's happening from the state machine
diagram[1] because there is some hidden state - the next state sometimes
depends on the previous state as well as the current state.
The basic idea of the ironic state machine is that when a node is not
provisioned it is in the "available" state, and when it is provisioned
it is in the "active" state. In normal use a node flips between these
two states. Automated cleaning happens when flipping from "active" to
"available". There's one additional subtlety: when the node is first
created automated cleaning also happens when moving from "manageable" to
"available". Basically automated cleaning always happens before moving
into the "available" state:
[Manageable]
|
V
Automated
Cleaning <---------+
| |
V |
[Available] | Deprovision
| |
| Provision |
V |
[Active] ---------+
How Metal³ currently works is... not like that. On deprovisioning we
wait for the automated cleaning to happen, then when the node reaches
the "available" state we move it back to "manageable". This means that
if we provision the host again we have to clean it a second time, even
though we just cleaned it.
I submit that this is not what we want, and I have proposed a patch[2]
to change it in this PR:
https://github.com/metal3-io/baremetal-operator/pull/716
The final subtlety is that there also exists "manual cleaning",[3] and
this is required to change some settings including and possibly(?) not
limited to RAID. And manual cleaning can only be triggered from the
"manageable" state. If you want to change the RAID setup between
deployments, you have to run 3 cleanings (automated, then manual, then
automated). There doesn't appear to be a way around this.
In light of this, what if we implemented a new Preparing state in Metal³
that occurs *before* the Ready state:
Inspecting -> Preparing -> Ready -> Provisioning -> Provisioned
^ | ^ |
| | | |
+--------+ +----Deprovisioning <------+
In this Preparing state we would copy over the RAID (and BIOS?) settings
and note them in the status. From the Ready state we would return to
Preparing if and only if the settings in the spec differed from the
status. Provisioning would begin only once the status matches the spec.
This differs from the current RAID proposal, which has a Preparing state
*after* the Ready state (immediately before Provisioning). If you always
provision a host immediately whenever it is available, this makes no
difference. But if you don't then it means that an available host is
always ready to be provisioned immediately without waiting for cleaning.
In this scenario, if the RAID settings don't change then it also saves
an extra manual + automated cleaning. However, that may be a bad thing:
there was some discussion that we are concerned that a user could mess
up the RAID configuration while provisioned, and that therefore we
should always reconfigure it after deprovisioning. Dmitry suggested[4]
that ironic could allow us to delete the current config during
deprovisioning. When that becomes possible, we could implement it by
clearing the RAID data in the status during the Deprovisioning state.
I think this also has the effect of eliminating any dependencies between
the RAID implementation and Dmitry's proposal[5] to add provisioning
sub-states for progress reporting. We could still potentially merge the
RAID patch as-is and change the state machine later, subject to
confirming that no API changes would be needed.
Thoughts?
cheers,
Zane.
[1] https://docs.openstack.org/ironic/latest/contributor/states.html
[2]
https://github.com/metal3-io/baremetal-operator/pull/716/commits/18465130c8f42bda4aa8191b5093656dcc029d90
[3] https://docs.openstack.org/ironic/latest/admin/cleaning.html
[4] https://github.com/metal3-io/metal3-docs/pull/134/files#r492620912
[5] https://github.com/metal3-io/metal3-docs/pull/138
Dmitry Tantsur
Nov 19, 2020, 5:11:58 AM11/19/20
to Zane Bitter, Metal3 Development List
Hi,
On Fri, Nov 13, 2020 at 10:00 PM Zane Bitter <zbi...@redhat.com> wrote:
I recently had cause to start looking a bit more deeply into how
cleaning works in Metal³. This is something I had been confused about
for a while, and now that I figured it out I think it has some
implications for how we implement some future features, so I wanted to
take a moment to document here for everyone.
It's actually not obvious what's happening from the state machine
diagram[1] because there is some hidden state - the next state sometimes
depends on the previous state as well as the current state.
FYI the expected stable state can be fetched from node.target_provision_state, it will be 'available' for automated cleaning and 'manageable' for manual.
The basic idea of the ironic state machine is that when a node is not
provisioned it is in the "available" state, and when it is provisioned
it is in the "active" state. In normal use a node flips between these
two states. Automated cleaning happens when flipping from "active" to
"available". There's one additional subtlety: when the node is first
created automated cleaning also happens when moving from "manageable" to
"available". Basically automated cleaning always happens before moving
into the "available" state:
[Manageable]
|
V
Automated
Cleaning <---------+
| |
V |
[Available] | Deprovision
| |
| Provision |
V |
[Active] ---------+
How Metal³ currently works is... not like that. On deprovisioning we
wait for the automated cleaning to happen, then when the node reaches
the "available" state we move it back to "manageable". This means that
if we provision the host again we have to clean it a second time, even
though we just cleaned it.
Ouch. Yeah, it's suboptimal.
I submit that this is not what we want, and I have proposed a patch[2]
to change it in this PR:
https://github.com/metal3-io/baremetal-operator/pull/716
The final subtlety is that there also exists "manual cleaning",[3] and
this is required to change some settings including and possibly(?) not
limited to RAID. And manual cleaning can only be triggered from the
"manageable" state. If you want to change the RAID setup between
deployments, you have to run 3 cleanings (automated, then manual, then
automated). There doesn't appear to be a way around this.
Welll... there is, but it's not exactly trivial to implement.
Some background: we have historically seen manual cleaning as a one-shot preparation (or how Red Hat used to call it "ready state") step before a node enters the available->active->available samsara. What you're talking about (changing configurations between deployments) is not exactly a use case for manual cleaning. We have invented and recently implemented "deploy steps" for the same purpose. These steps are run during deployment itself (and can even run in-between the core steps).
At this point they are requested quite differently from clean steps (there are plans to unify that), so switching to them would be a non-trivial change for metal3.
In light of this, what if we implemented a new Preparing state in Metal³
that occurs *before* the Ready state:
Inspecting -> Preparing -> Ready -> Provisioning -> Provisioned
^ | ^ |
| | | |
+--------+ +----Deprovisioning <------+
In this Preparing state we would copy over the RAID (and BIOS?) settings
and note them in the status. From the Ready state we would return to
Preparing if and only if the settings in the spec differed from the
status. Provisioning would begin only once the status matches the spec.
I like this approach as well, it's definitely less invasive than switching to deploy steps.
Dmitry
--
You received this message because you are subscribed to the Google Groups "Metal3 Development List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metal3-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/metal3-dev/23189c96-95fe-b807-b33b-46070bc3353e%40redhat.com.
--
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
Reply all
Reply to author
Forward
0 new messages