How to avoid throttling of AWS t1-micro instances due to Puppet Agent cpu-load?

[Martijn]

We run a small number of t1-micro instances on Amazon Web Services, because the tasks have consistent very low CPU requirements (1-2% average). We monitor these instances via Amazon CloudWatch with alerting on things like extended 100% CPU usage.

For those who don't know, t1.micro instances have have very limited CPU power but have the ability to peak for a short time, after which the CPU will be throttled to maintain a certain average. The concept is explained in http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html

The instances are managed via Puppet Agent version 3.4.2, talking to our Puppet master, and have 267 resources applied to them. The Puppet agent is the only process on the server that manages to use enough CPU for long enough to trip the throttling threshold, affecting every process on the instance. This in turn triggers CloudWatch after a little while.

I would like to reduce the Agent's impact on the host, while keeping the host managed, perhaps by limiting the CPU usage of the process so we avoid triggering the throttle. A Puppet run taking longer would be fine in this case. I've looked at tools like cpulimit and cgroups, but am unsure how to apply those via Puppet.

How can I avoid this throttling and the resulting alerts?

Reducing the number of Puppet resources further, to reduce the load, appears difficult and would likely not be enough. Any other ideas?

Regards, Martijn

[David Schmitt]

Hi,


Would it be enough to reduce the frequency of puppet runs to, like, once
a day?

Regards, David

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/ab28b8fa-fd7d-4cc7-9900-406b200225bc%40googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Martijn]

That would certainly reduce the frequency of alerts occurring, and a nightly run would have less impact on service. How would I do that?

It's not ideal though. I'd like to have the same frequency on all nodes so that I know when the entire fleet is in compliance after a change.

Regards, Martijn

Op donderdag 6 februari 2014 13:41:44 UTC+1 schreef David Schmitt:

[José Luis Ledesma]

You can use mcollective to force a run whenever you want, but I don't know how it may affect resource consuming.

Regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f3304c21-a1e9-4b3e-97ee-b21ca04bfca1%40googlegroups.com.

[jcbollinger]

On Thursday, February 6, 2014 6:58:39 AM UTC-6, Martijn wrote:

That would certainly reduce the frequency of alerts occurring, and a nightly run would have less impact on service. How would I do that?

There are two main alternatives:

1. Use the "runinterval" configuration parameter (http://docs.puppetlabs.com/references/latest/configuration.html#runinterval)
2. Run Puppet via a scheduler (e.g. cron) instead of as its own daemon
   

If you want to control the actual time of day of your runs then option 2 is better.

 

It's not ideal though. I'd like to have the same frequency on all nodes so that I know when the entire fleet is in compliance after a change.

You can manage the nodes' puppet.conf via Puppet to ensure that they are all configured with the same runinterval.  Or you can manage the Puppet cron job via Puppet once you bootstrap Puppet on your nodes.

Another approach along this line might be to apply Puppet schedules to some of your resources that are expensive to sync (http://docs.puppetlabs.com/references/latest/metaparameter.html#schedule ; http://docs.puppetlabs.com/references/latest/type.html#schedule).

That begs the question, though, of which resources it is that are so costly to sync (or if indeed there are any specially costly rsources).  Running the agent once with --evaltrace might help you determine that.  If there are indeed issues with specific resources then we might be able to help you find better approaches to managing those.


John