puppet unscheduling many service refreshes

[Corey Osman]

Is there any “back off” algorithm in puppet that would unscheduled a service refresh when the CPU load is too high. Is there a timeout associated with refreshing services?

My CPU load peaks as high as 18(one min)(6 Processor Count) during puppet runs . I have about 20-30 java processes that need restarting almost every time so its quite a chore for puppet to restart all these services.


My puppet runs last about 8-9 minutes for a full configuration/deployment.



Corey

[Pete Brown]

You could write a fact that checks the system load and base your
service restarts on that.
I thought there was one but there only seems to be uptime and swapfree.

Here's a ruby gem that might help get you started.
https://github.com/nethacker/usagewatch

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/D0CE60BA-B66D-4CAB-8B41-86E8CF75C31D%40logicminds.biz.
> For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

A fact probably won't help in this case since it seems to be something that's happening as the service get restarted during the Puppet run and a fact will only tell you what's happening at the start of the run.

What you may need to do is to change the timeout and restart values of your service statements to account for these anomalies. Your restart statements could call a wrapper script that restarts the service but waits in the background until enough resources are free and your timeout could be set sufficiently high that the service resource would have time to execute.

If you happen to not have anything dependent on this service restarting (which is usually the case), you could even fire the service statement off into the background with your wrapper script and allow the Puppet run to continue unabated.

This is one of those extremely sticky issues that seems to be particularly rife in Java-land.

Thanks,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJ8DPF71uaVqpS_Ap_NtH1xG8p0dohfKa-5GUiWZ%3D3zh6t3t5Q%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Corey Osman]

I am already using java service wrapper and having another wrapper around the jsw wrapper sounds super hack.  Although it would fix the issue.  I have gone as far as spreading the applications across more roles to reduce possible configuration load.  Do we know for a fact that puppet is purposely unscheduling these service refreshes due to high load.  Is this feature/bug documented anywhere?  Where would the code be that implements this "smart unschedule"?

than