Config test when restarting a service

[Jonathan Gazeley]

Hi folks,

I'm tying myself in knots trying to figure out the best metaparameters
to use here. I want to define an exec that only runs when a service is
scheduled to be restarted. It should check the config file and if the
syntax is invalid, it should abort restarting the service (i.e. won't
kill a running service).

This is what I've got so far:

service { 'dhcpd':
ensure => running,
enable => true,
hasstatus => true,
hasrestart => true,
notify => Exec['dhcpd-config-test'],
}

# This exec tests the dhcpd config and fails if it's bad
# It isn't run every time puppet runs, but only when dhcpd is to be
restarted
exec { 'dhcpd-config-test':
command => '/usr/bin/sudo /usr/sbin/dhcpd -t',
returns => 0,
refreshonly => true,
logoutput => on_failure,
}

Notify on its own is no good, since if the exec fails, the service gets
restarted anyway.

Require on its own is no good, since the exec runs every time.

Is there some way to combine the two?

Thanks,
Jonathan

[Peter Meier]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Notify on its own is no good, since if the exec fails, the service
> gets restarted anyway.
>
> Require on its own is no good, since the exec runs every time.
>
> Is there some way to combine the two?

The problem is that a notify includes a before relationship, which
means that the service resource is always being managed *before* the
config test happens. What you want is the following:


file{'/etc/dhcpd/dhcpd.conf':
notify => [Exec['dhcpd-config-test'],Service['dhcpd']],
....

}

# This exec tests the dhcpd config and fails if it's bad
# It isn't run every time puppet runs, but only when dhcpd is to be

# restarted
exec { 'dhcpd-config-test':
command => '/usr/sbin/dhcpd -t',
refreshonly => true,
before => Service['dhcpd'],

}

service { 'dhcpd':
ensure => running,
enable => true,
}

This will not restart the dhcpd if the exec fails, as they have now a
relationship and if the exec fails the service is skipped. And the
exec is only being triggered when the file changes.

However, this has the drawback that you will only realize that within
the run where puppet changes the file. On the next run, though the
previous one failed, the file won't change anymore, the exec won't be
run and the service will already be running. So no change anymore.

I don't see a solution to that without having to constantly check the
config file and/or rechange the file, so it keeps chaning on the next
run, e.g. like:

exec { 'dhcpd-config-test':
command => '/usr/sbin/dhcpd -t || (echo "#failed" >>
/etc/dhcpd/dhcpd.conf && /usr/bin/false)',
refreshonly => true,
before => Service['dhcpd'],
}

This will invalidate the target state of the configfile and so it will
be changed on the next run again, the validation fails again and you
keep having that node being reported as a failed node.
Otherwise, it will only fail once.

best

pete
-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAlWagfwACgkQbwltcAfKi39RtwCfcXp8wbJGntag04oiulW3qDHi
YiQAn0+Vk2p8QLAlf+A4dudpCeCUlYa/
=mSSL
-----END PGP SIGNATURE-----

[Jonathan Gazeley]

On 06/07/15 14:26, Peter Meier wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>> Notify on its own is no good, since if the exec fails, the service
>> gets restarted anyway.
>>
>> Require on its own is no good, since the exec runs every time.
>>
>> Is there some way to combine the two?
> The problem is that a notify includes a before relationship, which
> means that the service resource is always being managed *before* the
> config test happens. What you want is the following:

Hi Pete,

Thanks for your detailed response. At my site, the DHCP config is partly
written out by a Forge module, partly generated by a script from a
database and partly deployed from a git repo where it is edited by
humans. So it seems a bit messy to get all of these resources to notify
the config test and the service.

I also haven't decided whether I should run my database->dhcp config
script from an exec or via the system cron - in which case restarting
the dhcp daemon is done outside of puppet anyway.

The more I think about this, the more I think the best fix is to edit
the init script for the service. The stock init script for dhcpd on
CentOS 6 already includes a 'configtest' action so with a few lines of
code this could be executed before the service is started. This would
then trap any restarts done via cron, or via users.

It's not really the solution I wanted but I think it's probably the best
option all round.

Cheers,
Jonathan

[Benjamin Parmentier]

Hi,

You can do something like this :

file { '/etc/dhcp/dhcpd.conf':

  ...

  notify => Exec['dhcp-syntax-check']

}

exec { 'dhcp-syntax-check':

  ...

  subscribe => File['/etc/dhcp/dhcpd.conf'],

  notify    => Service['dhcpd']

}

service { 'dhcpd':

  ...

  subscribe => Exec['dhcp-syntax-check']

}

(subscribe metaparameters are not mandatory here)

The Service is only refreshed when the configuration file changed so you can insert an Exec between File and Service.

But it's not that simple to manage execution of an Exec. Have a look to https://docs.puppetlabs.com/references/4.2.latest/type.html#exec and check the Refresh section.

Regards,

BP

[jcbollinger]

On Monday, July 6, 2015 at 8:43:33 AM UTC-5, Jonathan Gazeley wrote:

On 06/07/15 14:26, Peter Meier wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>> Notify on its own is no good, since if the exec fails, the service
>> gets restarted anyway.
>>
>> Require on its own is no good, since the exec runs every time.
>>
>> Is there some way to combine the two?
> The problem is that a notify includes a before relationship, which
> means that the service resource is always being managed *before* the
> config test happens. What you want is the following:

Hi Pete,

Thanks for your detailed response. At my site, the DHCP config is partly
written out by a Forge module, partly generated by a script from a
database and partly deployed from a git repo where it is edited by
humans. So it seems a bit messy to get all of these resources to notify
the config test and the service.

You can always have the config-test resource 'subscribe' to the various data source resources instead of having the data sources 'notify' the exec.  The effects are identical.  Likewise, you may either have the config-test resource 'notify' the service or have the service 'subscribe' to the config-test resource.  It seems pretty clean to me to put all these relationships on the config test, as that way you can freely modify or even remove the config test without changing anything else.  The ultimate data sources can notify the service directly if you wish; that will not prevent the config test being interposed between and serving to gate the restart.  If you want the test to be performed only when one of the config sources is modified (by Puppet) then set it to be 'refreshonly'.

You cannot hook the test so tightly to the service that it is automatically run before the service is refreshed, without regard to the reason for the refresh.  You could, however, wrap Exec and Service together into a defined type, to serve as a stand-in for the bare Service resource, so that any event received by the defined type instance triggers the test before the service is restarted:

define mymodule::checked_dhcpd(
    $test_command = '/bin/true',
    # service properties ...
  ) {
  exec { 'dhcpd-config-test':
    command => $test_command,
    refreshonly => true
  }
  ~>
  service { 'dhcpd':
    # ...
  }

}

That does not protect you from resources notifying the Service directly, but it will do what you want as long as they notify only the defined-type instance (which alternatively may itself subscribe to any resource of interest).


John