Bootstrap in chroot

[Markus Rexhepi-Lindberg]

Is it possible to bootstrap a host running in a chroot?

I want to bootstrap a host during its provisioning stage were it currently runs in a chroot environment. While bootstrapping cf-agent complains that the cf-execd daemon is not running, which seems to be the case since cfengine3.service is not allowed to start in a chroot environment.

```
# chroot /target cf-agent -B <cfehub>
notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
  notice: Trusting new key: MD5=<redacted>
R: Bootstrapping from host '<redacted>' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This autonomous node assumes the role of voluntary client
R: Updated local policy from policy server
R: Triggered an initial run of the policy
R: Restarted systemd unit cfengine3
   error: Bootstrapping failed, cf-execd is not running
  notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
```

I managed to workaround this by first trying to bootstrap, which fails, then manually start cf-execd in the chroot and finally try to bootstrap again which succeeds.

```
# chroot /traget cf-execd
# chroot /target cf-agent -B <cfehub>
R: Bootstrapping from host '<redacted>' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This autonomous node assumes the role of voluntary client
R: Updated local policy from policy server
R: Triggered an initial run of the policy
  notice: Bootstrap to '<redacted>' completed successfully!
```

This works but feels like a hack. Is there another way this could be done?

--

Markus

[Bas van der Vlies]

Markus,

I also bootstrap cfengine when a host has been provisioned. Only my options are different. Do no run policy.
```
/var/cfengine/bin/cf-agent --bootstrap="${IMAGESERVER}" --skip-bootstrap-policy-run
# I have a def.json setup with diferent json files. That is why I need this
/var/cfengine/bin/cf-agent -KI -f update.cf
```

When the server boots it wil run cfengne and everything is fine for use.

> --
> You received this message because you are subscribed to the Google Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/e6e87828-d8aa-4040-9352-ef9e6f4a1f9bn%40googlegroups.com.

[Vratislav Podzimek]

Hi Markus,

Did you check if the second run is necessary? AFAICT, the error about cf-execd
not running is a valid one, but it doesn't mean that the rest of the bootstrap
process didn't run or failed. I'd expect the host to be bootstrapped just fine
just with the daemons (cf-execd, cf-serverd, cf-monitord) not running. So I
think you can skip the last step, but you should start the other two daemons in
the chroot as well.

--
Vratislav

[Markus Rexhepi-Lindberg]

Thanks for suggestion Bas. I unfortunately get the same result when running with the suggested options.

--

Markus

[Bas van der Vlies]

It also fails if you only do:
* /var/cfengine/bin/cf-agent --bootstrap="${IMAGESERVER}"
--skip-bootstrap-policy-run

like Vratislav suggested. At our place we only use the cfengine library
not the whole framework. Slowly be more inline with it.

On 22/05/2024 13:20, Markus Rexhepi-Lindberg wrote:
> Thanks for suggestion Bas. I unfortunately get the same result when
> running with the suggested options.
>
> --
> Markus
>
> On Tuesday, May 21, 2024 at 5:20:17 PM UTC+2 Bas van der Vlies wrote:
>
> Markus,
>
> I also bootstrap cfengine when a host has been provisioned. Only my
> options are different. Do no run policy.
> ```
> /var/cfengine/bin/cf-agent --bootstrap="${IMAGESERVER}"
> --skip-bootstrap-policy-run
> # I have a def.json setup with diferent json files. That is why I
> need this

> /var/cfengine/bin/cf-agent -KI -f update.cf <http://update.cf>

> ```
>
> When the server boots it wil run cfengne and everything is fine for
> use.
>
>
> > On 20 May 2024, at 11:58, Markus Rexhepi-Lindberg
> <dazet...@gmail.com> wrote:
> >
> > Is it possible to bootstrap a host running in a chroot?
> >
> > I want to bootstrap a host during its provisioning stage were it
> currently runs in a chroot environment. While bootstrapping cf-agent
> complains that the cf-execd daemon is not running, which seems to be
> the case since cfengine3.service is not allowed to start in a chroot
> environment.
> >
> > ```
> > # chroot /target cf-agent -B <cfehub>
> > notice: Bootstrap mode: implicitly trusting server, use
> --trust-server=no if server trust is already established
> > notice: Trusting new key: MD5=<redacted>
> > R: Bootstrapping from host '<redacted>' via built-in policy

> '/var/cfengine/inputs/failsafe.cf <http://failsafe.cf>'

> > R: This autonomous node assumes the role of voluntary client
> > R: Updated local policy from policy server
> > R: Triggered an initial run of the policy
> > R: Restarted systemd unit cfengine3
> > error: Bootstrapping failed, cf-execd is not running
> > notice: Bootstrap mode: implicitly trusting server, use
> --trust-server=no if server trust is already established
> > ```
> >
> > I managed to workaround this by first trying to bootstrap, which
> fails, then manually start cf-execd in the chroot and finally try to
> bootstrap again which succeeds.
> >
> > ```
> > # chroot /traget cf-execd
> > # chroot /target cf-agent -B <cfehub>
> > R: Bootstrapping from host '<redacted>' via built-in policy

> '/var/cfengine/inputs/failsafe.cf <http://failsafe.cf>'

> > R: This autonomous node assumes the role of voluntary client
> > R: Updated local policy from policy server
> > R: Triggered an initial run of the policy
> > notice: Bootstrap to '<redacted>' completed successfully!
> > ```
> >
> > This works but feels like a hack. Is there another way this could
> be done?
> >
> > --
> > Markus
> >
> > --
> > You received this message because you are subscribed to the
> Google Groups "help-cfengine" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to help-cfengin...@googlegroups.com.
> > To view this discussion on the web visit

> https://groups.google.com/d/msgid/help-cfengine/e6e87828-d8aa-4040-9352-ef9e6f4a1f9bn%40googlegroups.com <https://groups.google.com/d/msgid/help-cfengine/e6e87828-d8aa-4040-9352-ef9e6f4a1f9bn%40googlegroups.com>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to help-cfengin...@googlegroups.com

> <mailto:help-cfengin...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/help-cfengine/f4f250c1-d132-40fa-9736-8586dd07f922n%40googlegroups.com <https://groups.google.com/d/msgid/help-cfengine/f4f250c1-d132-40fa-9736-8586dd07f922n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
--
Bas van der Vlies
| High Performance Computing & Visualization | SURF| Science Park 140 |
1098 XG Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surf.nl | www.surf.nl |

[Markus Rexhepi-Lindberg]

It seems that the bootstrap fails AFAIK. For instance the `/var/cfengine/policy_server.dat` file does not get provisioned.

I first run a bootstrap as I otherwise can't start cf-execd.

Starting cf-execd without running a bootstrap first.
```
# cf-execd
   error: There is no readable input file at '/var/cfengine/inputs/promises.cf'. (stat: No such file or directory)
   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.
   error: Error reading CFEngine policy. Exiting...
```

Running a boostrap and then starting cf-execd.
```
# cf-agent -B <redacted>

  notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
  notice: Trusting new key: MD5=<redacted>
R: Bootstrapping from host '<redacted>' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This autonomous node assumes the role of voluntary client
R: Updated local policy from policy server
R: Triggered an initial run of the policy
R: Restarted systemd unit cfengine3
   error: Bootstrapping failed, cf-execd is not running

# cf-execd
#
```

After I have done this I attempt to run the policies.
```
# cf-agent -KIC
...
error: No suitable server found for '/var/cfengine/scripts/'
...
```
I get errors suggesting that there is not suitable server to be found/used. The policies them self seem to run fine though.

If I run a bootstrap again as the last step.
```
# cf-agent -B <redacted>

  notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established

R: Bootstrapping from host '<redacted>' via built-in policy '/var/cfengine/inputs/failsafe.cf'
R: This autonomous node assumes the role of voluntary client
R: Updated local policy from policy server
R: Triggered an initial run of the policy
  notice: Bootstrap to '<redacted>' completed successfully!

# ls -l /var/cfengine/policy_server.dat
-rw------- 1 root root 26 May 22 13:35 /var/cfengine/policy_server.dat
```
It goes through and the `/var/cfengine/policy_server.dat` file gets provisioned and I do not get "No suitable server ..." errors when running `cf-agent`.

--
Markus

[Vratislav Podzimek]

On Wed, 2024-05-22 at 04:38 -0700, Markus Rexhepi-Lindberg wrote:
> It seems that the bootstrap fails AFAIK. For instance the `/var/cfengine/policy_server.dat` file does not get provisioned.

That looks like a bug we should fix. IMHO, the bootstrap run of cf-agent should
report errors, but it should do its best to actually finish the bootstrap
process.

Could you please file this at
https://northerntech.atlassian.net/jira/software/c/projects/CFE/issues ?

>
> I first run a bootstrap as I otherwise can't start cf-execd.
>
> Starting cf-execd without running a bootstrap first.
> ```
> # cf-execd
>    error: There is no readable input file at '/var/cfengine/inputs/promises.cf'. (stat: No such file or directory)
>    error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.
>    error: Error reading CFEngine policy. Exiting...
> ```

Yes, cf-execd is configured by policy so it needs to load the policy when it
starts.

>
> Running a boostrap and then starting cf-execd.
> ```
> # cf-agent -B <redacted>
>   notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
>   notice: Trusting new key: MD5=<redacted>
> R: Bootstrapping from host '<redacted>' via built-in policy '/var/cfengine/inputs/failsafe.cf'
> R: This autonomous node assumes the role of voluntary client
> R: Updated local policy from policy server
> R: Triggered an initial run of the policy
> R: Restarted systemd unit cfengine3
>    error: Bootstrapping failed, cf-execd is not running
> # cf-execd
> #
> ```
>
> After I have done this I attempt to run the policies.
> ```
> # cf-agent -KIC
> ...
> error: No suitable server found for '/var/cfengine/scripts/'
> ...
> ```
> I get errors suggesting that there is not suitable server to be found/used. The policies them self seem to run fine though.

This is because the var/cfengine/policy_server.dat file contains the information
about which server to contact for remote policy/data -- populated at the end of
the bootstrap process. Apparently only in case of a full successful bootstrap.

>
> If I run a bootstrap again as the last step.
> ```
> # cf-agent -B <redacted>
>   notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
> R: Bootstrapping from host '<redacted>' via built-in policy '/var/cfengine/inputs/failsafe.cf'
> R: This autonomous node assumes the role of voluntary client
> R: Updated local policy from policy server
> R: Triggered an initial run of the policy
>   notice: Bootstrap to '<redacted>' completed successfully!
> # ls -l /var/cfengine/policy_server.dat
> -rw------- 1 root root 26 May 22 13:35 /var/cfengine/policy_server.dat
> ```
> It goes through and the `/var/cfengine/policy_server.dat` file gets provisioned and I do not get "No suitable server ..." errors when running `cf-agent`.

Right. Like I wrote above, I believe the bootstrap process should be fixed to do
as much as it can and only report failures that happened on the way.

--
Vratislav

[Bas van der Vlies]

Markus,

I do not see that you just `skip the policy run`:

* /var/cfengine/bin/cf-agent --bootstrap="${IMAGESERVER}"
--skip-bootstrap-policy-run

Does the fail or succeed?

> '/var/cfengine/inputs/failsafe.cf <http://failsafe.cf>'

> >  R: This autonomous node assumes the role of voluntary client
> >  R: Updated local policy from policy server
> >  R: Triggered an initial run of the policy
> >  R: Restarted systemd unit cfengine3
> >     error: Bootstrapping failed, cf-execd is not running
> >    notice: Bootstrap mode: implicitly trusting server, use
> --trust-server=no if server trust is already established
> > ```
> >
> > I managed to workaround this by first trying to bootstrap, which
> fails, then manually start cf-execd in the chroot and finally try to
> bootstrap again which succeeds.
> >
> > ```
> > # chroot /traget cf-execd
> > # chroot /target cf-agent -B <cfehub>
> > R: Bootstrapping from host '<redacted>' via built-in policy

> '/var/cfengine/inputs/failsafe.cf <http://failsafe.cf>'

> > R: This autonomous node assumes the role of voluntary client
> > R: Updated local policy from policy server
> > R: Triggered an initial run of the policy
> >   notice: Bootstrap to '<redacted>' completed successfully!
> > ```
> >
> > This works but feels like a hack. Is there another way this could
> be done?
> Did you check if the second run is necessary? AFAICT, the error
> about cf-execd
> not running is a valid one, but it doesn't mean that the rest of the
> bootstrap
> process didn't run or failed. I'd expect the host to be bootstrapped
> just fine
> just with the daemons (cf-execd, cf-serverd, cf-monitord) not
> running. So I
> think you can skip the last step, but you should start the other two
> daemons in
> the chroot as well.
>
> --
> Vratislav
>

> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to help-cfengin...@googlegroups.com
> <mailto:help-cfengin...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/help-cfengine/0a378e7e-d113-42f1-8674-0efdc1e5731en%40googlegroups.com <https://groups.google.com/d/msgid/help-cfengine/0a378e7e-d113-42f1-8674-0efdc1e5731en%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Vratislav Podzimek]

Hello,
I'm happy to let you know that we added [1] a new CLI option to cf-agent,
--skip-bootstrap-service-start, which prevents the errors due to failing start
of services during bootstrap. When using this option cf-execd is not started by
the bootstrap process (though systemd may start it behind the scenes), the check
for it running is skipped and the bootstrap process successfully finishes.

[1] https://northerntech.atlassian.net/browse/ENT-11932

--
Vratislav