puppet calling systemctl instead of /etc/init.d/xxx even when told to use debian provider

[Aqueos Aqueos]

hi,

 i am on puppet 4.3 and on debian jessie with sysV init system (  not systemd). I have issues because even if i put:

Service{
        provider        =>      $operatingsystem ? {
                                        Debian  =>      'debian',
                                }
}


in site.pp puppet still try to call systemctl:

Debug: Executing: 'systemctl is-active postfix'
Debug: Executing: 'systemctl is-active ssh'
Debug: Executing: 'systemctl is-active proftpd'


 i don't know why it insist on using systemd that is not installed and not used in any recipe when we told it to use debian style init.

even if i REMOVE /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/systemd.rb , IT STILL CALL systemctl.. this drive me mad....

 Any ideas ?



regards,
Ghislain.

[Aqueos Aqueos]

in /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/debian.rb the lines:

    if ((os == 'debian' && majversion >= 8) || (os == 'ubuntu' && majversion >= 15))
      # SysVInit scripts will always return '0' for status when the service is masked,
      # even if the service is actually stopped. Use the SysVInit-Systemd compatibility
      # layer to determine the actual status. This is only necessary when the SysVInit
      # version of a service is queried. I.e, 'ntp' instead of 'ntp.service'.
      (@resource[:hasstatus] == :true) && ["systemctl", "is-active", @resource[:name]]
    else
      super
    end


this force people to install the systemd package where there is not any need for it. I don't think puppet should force install of unecessary package, if we want systemd we would not overide the default provider to debian and let it to systemd therefor those lines are more a pain thn an help for me.

best regards,
Ghislain.

[jcbollinger]

I understand your dissatisfaction, but I feel obligated to observe that Puppet isn't forcing you to do anything.  Because of peculiarities in Debian, Puppet cannot properly manage services on some versions of that distro via straight sysV initscripts.  This in no way forces you to install systemd; you also have the alternative of not managing services via Puppet in this quirky environment.

Inasmuch as this particular characteristic apparently affects only newer versions of Debian (and Debian-derived Ubuntu), I'm inclined to suppose that it arises in part because of a shift in Debian away from SysV init infrastructure, which indeed seems to have become quite the thing to do in the last few years.  If you want to complain about someone driving you toward systemd, I suggest you focus on your distro itself.


John

[Ghislain]

I understand your dissatisfaction, but I feel obligated to observe that Puppet isn't forcing you to do anything.  Because of peculiarities in Debian, Puppet cannot properly manage services on some versions of that distro via straight sysV initscripts.  This in no way forces you to install systemd; you also have the alternative of not managing services via Puppet in this quirky environment.

Inasmuch as this particular characteristic apparently affects only newer versions of Debian (and Debian-derived Ubuntu), I'm inclined to suppose that it arises in part because of a shift in Debian away from SysV init infrastructure, which indeed seems to have become quite the thing to do in the last few years.  If you want to complain about someone driving you toward systemd, I suggest you focus on your distro itself.


John

hi,

 i am sorry that you take it that way, i am not here to argue with you if you think debian is a quirky distribution, that may be the case but this is the one we use so  that's why i found this issue.

 The fact is that  debian has 2 system of init to chosse, sysV and the systemd ecosystem that do init but also so many other thing that it cannot be used in a lot of chroot like scenario.

  Right now until this particular push of new code all was working as intended for us, but this code was added in the service sysV init provider a check of systemD  in the sysvinit part. So you can be an advocate  of systemd and there is no problem on that but sytemd cannot work on a lot of situations and i do not see the point of calling it inside a sysvinit part without any way to ovveride  this choice :) If people install systemd they will use the systemd provider not the sysvinit one.

 Debian offer me the choise between systemd and sysinit and the modification to the service provider simply make things hard and increase surface of problems as even with sysvinti system we must now install the systemd packages and all the dependency hell (and even the wrapper install daemon cgroup crtl script and a bunch of thing unneeded). Your solution on maintening my own system instead of running puppet is rather an unexpected  answer.

 the problem you refers to should be the distro problem to fix the init.d script that do not behave like you want them (ie return reliable value on status), i do not think you should break ascended compatibility to ack that some script do not respond like puppet want. I know that a lot of sysvinit script return unreliable values but this was, is, and will be the case.  Why break the compatibility for this ? For all services ? Best case scenario people fix the scripts, worst case it goes away in debian9 as we will all move to freeBSD or use systemd anyway. Also i assume it should not be that hard to test for the systemd package to be present to allow this test or not and therefor make ascended compatibility work as it was before this code change :)


best regards,
Ghislain.

[jcbollinger]

On Monday, November 30, 2015 at 11:16:17 AM UTC-6, Aqueos Aqueos wrote:

I understand your dissatisfaction, but I feel obligated to observe that Puppet isn't forcing you to do anything.  Because of peculiarities in Debian, Puppet cannot properly manage services on some versions of that distro via straight sysV initscripts.  This in no way forces you to install systemd; you also have the alternative of not managing services via Puppet in this quirky environment.

Inasmuch as this particular characteristic apparently affects only newer versions of Debian (and Debian-derived Ubuntu), I'm inclined to suppose that it arises in part because of a shift in Debian away from SysV init infrastructure, which indeed seems to have become quite the thing to do in the last few years.  If you want to complain about someone driving you toward systemd, I suggest you focus on your distro itself.


John

hi,

 i am sorry that you take it that way, i am not here to argue with you if you think debian is a quirky distribution, that may be the case but this is the one we use so  that's why i found this issue.

Please allow me to clarify that I have no particular objection to nor animosity for Debian 8 "Jessie", or any earlier version of Debian.  Nevertheless, Jessie absolutely is "quirky" in the objectively measurable sense that under some circumstances, the "status" command of its SysV init scripts returns 0 when the LSB specifications dictate a different return value.  Others less generous than I might instead characterize that as a "flaw", or even a "bug".  This particular quirk presents a problem for Puppet, and therefore for you.

 

 The fact is that  debian has 2 system of init to chosse, sysV and the systemd ecosystem that do init but also so many other thing that it cannot be used in a lot of chroot like scenario.

Debian 8 moves from SysV as the default init system (and systemd available as a technology preview) to systemd as the default init system and SysV as a backwards-compatibility alternative.  IIRC, it also provides other alternatives, including at least upstart.  Although it is not exactly wrong to describe that as "[D]ebian has 2 system of init to chosse[sic]", doing so doesn't really present an accurate picture.

 

  Right now until this particular push of new code all was working as intended for us, but this code was added in the service sysV init provider a check of systemD  in the sysvinit part. So you can be an advocate  of systemd and there is no problem on that but sytemd cannot work on a lot of situations and i do not see the point of calling it inside a sysvinit part without any way to ovveride  this choice :) If people install systemd they will use the systemd provider not the sysvinit one.

I am not advocating for systemd.  I am simply observing that Debian seems to have embarked on a switch to it, inasmuch as systemd is not just an alternative init system but the default init system for Jessie.  Correlated with that switch, the behavior of the Debian SysV initscripts has changed so that the whole subsystem is no longer LSB-compliant.  As a result, the approach Puppet takes to managing services on earlier versions of Debian does not work reliably on Jessie.  Using the old system for Jessie was not a viable option.

 

 Debian offer me the choise between systemd and sysinit and the modification to the service provider simply make things hard and increase surface of problems as even with sysvinti system we must now install the systemd packages and all the dependency hell (and even the wrapper install daemon cgroup crtl script and a bunch of thing unneeded). Your solution on maintening my own system instead of running puppet is rather an unexpected  answer.

Debian seems not to offer quite the choice you think it does, as opting for sysvinit on Jessie is subtly inequivalent to using sysvinit by default on earlier Debians.  As for packages, I have to take your word for what and how many packages are required, and for how much trouble that is.  I confess, however, that I had supposed Apt to be a lot more capable and convenient than you seem to imply, and Debian's package collections -- even "unstable" -- to be more internally consistent.

In any event, if the Puppet changes necessitated by the sysvinit changes in Jessie cause Puppet to cease to be advantageous for service management for you, then why would you continue using Puppet for that purpose?  I hardly think it should be surprising to suggest preferring an alternative that is easier for you.  I do, however, object to the assertion that Puppet forces you to do anything, as if you had no choice in the matter.

With that said, you can perhaps adapt to the changes by supplying explicit 'status' commands for your Service resources, overriding the systemctl command Puppet uses on Jessie.  Inasmuch as this overrides the provider, you would have to manually ensure that it applies only to systems for which the chosen command is appropriate.  Alternatively, if you are managing only whether the service is currently running, as opposed to whether it is enabled to run automatically at system startup, then perhaps you would get the behavior you seem to want by choosing the 'init' provider instead of 'debian'.

 

 the problem you refers to should be the distro problem to fix the init.d script that do not behave like you want them (ie return reliable value on status), i do not think you should break ascended compatibility to ack that some script do not respond like puppet want.

What Puppet prefers is compliance with LSB standards.  That's not an arbitrary whim, and Debian used to provide it.  Jessie no longer does.  Puppet does not demand that either you or Debian change that; instead it implements a workaround.  Since you don't like that, either, I'm rather at a loss to determine what you think it should have done instead to solve the problem.

 

I know that a lot of sysvinit script return unreliable values but this was, is, and will be the case.  Why break the compatibility for this ? For all services ?

As I understand it, the problem is not individual init scripts, but rather a general change / new feature in sysvinit.  As a result, Puppet on Jessie could not reliably use the old method to manage any service.  Puppet did not break compatibility because Jessie already had done -- for all services, albeit not under all circumstances.

 

Best case scenario people fix the scripts, worst case it goes away in debian9 as we will all move to freeBSD or use systemd anyway.

Doing nothing was not an option.  The fix did not come arbitrarily out of thin air; the comments preceding the change show that a problem was discovered, diagnosed, and solved.

 

Also i assume it should not be that hard to test for the systemd package to be present to allow this test or not and therefor make ascended compatibility work as it was before this code change :)

Puppet's way would be to test for the systemctl program, rather than for a package.  Indeed, although all bets are off when you run in --noop mode, if in normal mode the provider does not test for the availability of systemctl before trying to use it then that would be an issue appropriate for a ticket.  HOWEVER, that does not mean that the old behavior would be a good option in the event that systemctl is not found.  The old behavior is inappropriate on Jessie regardless of whether systemd or any part of it is installed.


John

[Will Hopper]

Hey Ghislain!

 

  Right now until this particular push of new code all was working as intended for us, but this code was added in the service sysV init provider a check of systemD  in the sysvinit part. So you can be an advocate  of systemd and there is no problem on that but sytemd cannot work on a lot of situations and i do not see the point of calling it inside a sysvinit part without any way to ovveride  this choice :) If people install systemd they will use the systemd provider not the sysvinit one.

Indeed, the code block you pointed out was added recently to deal with the peculiarities of the systemd / init compatibility layer. Namely, as you mentioned, init script return values are inconsistent in Debian 8 / Ubuntu 15.04 and are not consistent with systemd's status reports. This fix works fine on standard installations which include systemd, and brings some consistency between systemd and init. However, the use case of a system which has had systemd removed or otherwise not present was not considered when originally making this change.

 

Also i assume it should not be that hard to test for the systemd package to be present to allow this test or not and therefor make ascended compatibility work as it was before this code change :)

Correct - this is actually a pretty simple fix which I've got a pull request for up at https://github.com/puppetlabs/puppet/pull/4466 (see the ticket at https://tickets.puppetlabs.com/browse/PUP-5548). I fired up a Debian 8 box and followed the steps at http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_jessie/sid_installation to purge systemd and make sysvinit the main unit, and everything looked to be working correctly from puppet's perspective.

Feel free to leave comments on the pull request or ticket. Hopefully this minor change will be everything required to get you back up and running.

[Will Hopper]

Hey John!

Thanks for the responses and analysis of the situation! You're spot on in several areas which prompted this change in the first place. Comments inline:

 

Please allow me to clarify that I have no particular objection to nor animosity for Debian 8 "Jessie", or any earlier version of Debian.  Nevertheless, Jessie absolutely is "quirky" in the objectively measurable sense that under some circumstances, the "status" command of its SysV init scripts returns 0 when the LSB specifications dictate a different return value.  Others less generous than I might instead characterize that as a "flaw", or even a "bug".  This particular quirk presents a problem for Puppet, and therefore for you.

 ...

I am not advocating for systemd.  I am simply observing that Debian seems to have embarked on a switch to it, inasmuch as systemd is not just an alternative init system but the default init system for Jessie.  Correlated with that switch, the behavior of the Debian SysV initscripts has changed so that the whole subsystem is no longer LSB-compliant.  As a result, the approach Puppet takes to managing services on earlier versions of Debian does not work reliably on Jessie.  Using the old system for Jessie was not a viable option.

Yes - this is the exact issue which prompted us to use systemd to determine service status in Debian 8 / Ubuntu 15.04 and above. As you've mentioned below, this presented a very large dilemma for Puppet, as it could no longer properly manage these services through the systemd / sysvinit compatibility layer. It should be noted, however, that this only affects default installations which still include both the systemd and sysvinit bits.

 

 

 

 Debian offer me the choise between systemd and sysinit and the modification to the service provider simply make things hard and increase surface of problems as even with sysvinti system we must now install the systemd packages and all the dependency hell (and even the wrapper install daemon cgroup crtl script and a bunch of thing unneeded). Your solution on maintening my own system instead of running puppet is rather an unexpected  answer.

Debian seems not to offer quite the choice you think it does, as opting for sysvinit on Jessie is subtly inequivalent to using sysvinit by default on earlier Debians.  As for packages, I have to take your word for what and how many packages are required, and for how much trouble that is.  I confess, however, that I had supposed Apt to be a lot more capable and convenient than you seem to imply, and Debian's package collections -- even "unstable" -- to be more internally consistent.

The key difference here, I believe, is that Ghislain (and probably others) have actively purged systemd from their Debian 8 systems, such as by following this guide: http://without-systemd.org/wiki/index.php/How_to_remove_systemd_from_a_Debian_jessie/sid_installation.

When doing this, you are effectively reinstating sysvinit as the chief init system and more or less reverting to how things worked back in Debian 7. From my brief tests around this, it seems that the old init scripts which were LSB compliant are brought back and used, meaning that Puppet can work properly.

All of this being said, I believe that the fix to use systemctl to determine service status is absolutely necessary on the default case of Debian 8 systems using the sysvinit / systemd compatibility layer (as is what you get upon a fresh installation). Not doing so breaks Puppet's ability to manage many services. However, I also think it's reasonable to check that systemctl is present on the system first, falling back to init scripts if it is not. This will allow use cases such as this one where systemd has been purged from the system.

[Ghislain]

hi,

thanks for those great insigth it is great to hear what leaded to this,
yes testing systemctl is a good option, also if i can build a logic to
change the "status" for each services based on the debian version it
would be possible but this report the logic inside the pp for all
services where i beleive it could be scaled in a easier way in the
provider code :)


best regards,
Ghislain.