| When the Puppet Agent runs during boot (or at other times, during changes of various sorts), some services may still be starting, which means they show a different state than "active" for the service units. When Puppet runs, it tries to enforce { ensure=>running , enable => true} on the Service resource, which ends up restarting the already-starting service. For example, when restarting a host running a MariaDB Galera cluster, MariaDB is in the process of "catching up" with the cluster (via IST) when Puppet restarts it. That ends up breaking the incremental updates, requiring the node to take even longer to start up again by doing a full SST (which also makes another node in the cluster unavailable at that time). The desired behavior is that Puppet would see that the service is already in the process of starting, and do nothing, or possibly just log a warning. A similar issue likely exists for services which are in the process of stopping. The command `systemctl status is-active mariadb.service` returns this output in these states:
- active (exit code 0) – normal "fully started" state
- inactive (exit code 3) – fully stopped
- deactivating (exit code 3) – in the process of stopping
- activating (exit code 3) – in the process of starting
Puppet is simply leveraging an exit code of 0 or nonzero to determine the state, but it should probably be interpreting the actual state, through the contents of STDOUT from `systemctl is-active $SERVICE_NAME` for instance, so that it can properly handle these other states. |
