Puppetserver, never-ending requests and watchdogs

[Nacho Barrientos]

Hi,

I'd like to bring up a point that was raised during the resolution of a
ticket.
The idea is to hopefully trigger a discussion and derive actions from it, if
necessary.

Bugs like the one described in PUP-7848 [0] (for which there's a fix
already,
thanks!) are quite dangerous from the operations' point of view as they
could
quickly reduce the performance of a production Puppet infrastructure.

Is there any kind of watchdog that can be configured at Puppetserver
level to
automatically destroy instances that are misbehaving like these ones
(perhaps
based on the CPU wall time, age...)? We're already using over here
max-requests-per-instance but for obvious reasons it's not useful in
this case
:)

The more agents exercising the bad code and triggering the issue, the faster
the load goes up and therefore the slower the infrastructure becomes. There
should be a way to tell Puppetserver how to protect itself. Perhaps there's
already but we could not find it [1]. In the meantime, what we're doing
is to
put some extra (and very specific) monitoring in place on our side to try to
detect this situation and alarm it but perhaps there's something that
could be
done directly at Puppetserver level to act earlier.

In case it helped we're running 2.7.2 over here.

What do you think?

Thanks!

[0] https://tickets.puppetlabs.com/browse/PUP-7848
[1]
https://docs.puppet.com/puppetserver/latest/config_file_puppetserver.html

--
bye
Nacho
http://cern.ch/nacho

[Henrik Lindberg]

I think it is a very good idea for a Puppet Server feature. I suppose it
would need to look at the compilation time and compare that against a
set timeout as it would otherwise be very difficult to figure out if a
legit load is rogue or not - a regular compilation could consume a lot
of CPU, as all depends on what is in the manifests being compiled.

I think you can log a feature request for Puppet Server with your idea
as that will more readily put this in front of those that prioritize
between features to add in upcoming releases.

Best,
- henrik

> [0] https://tickets.puppetlabs.com/browse/PUP-7848
> [1]
> https://docs.puppet.com/puppetserver/latest/config_file_puppetserver.html
>


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/