puppet cron jobs overwhelming host

[Tim Dunphy]

Hey guys,

 Ok, so I've run into a situation where I setup some cron jobs via puppet that do chown's and chmod's to make sure the web directory is owned by the web server and that the contents have the right permissions.

The idea was to have the developer push their code via subversion and then have their changes automatically take place on the web every few minutes. 

I have another cron setup to do an svn up in the web root at regular intervals to do that. That's never given me any issue.

But for some reason these cron jobs are spiking the load:

Here they are:

cron { "apache-chown":

          command => "/bin/chown -R apache:ftpgroup /var/www",

          user    => 'root',

          hour    => 0,

          minute  => '*'

      }

  cron { "chmod-files":

          command => "/bin/find /var/www -type f -exec chmod -v 664 {} \;",

          user    => 'root',

          hour    => 0,

          minute  => '*'

      }

   cron { "chmod-directories":

          command => "/bin/find /var/www -type d -exec chmod -v 775 {} \;",

          user    => 'root',

          hour    => 0,

          minute  => '*'

      }

That produces the following crons in my crontab:

# Puppet Name: chmod-files

* 0 * * * /bin/find /var/www -type f -exec chmod -v 664 {} \;

# Puppet Name: apache-chown

* 0 * * * /bin/chown -R apache:ftpgroup /var/www

# Puppet Name: chmod-directories

* 0 * * * /bin/find /var/www -type d -exec chmod -v 775 {} \;

And what I've noticed is that the host can be fine for several days. And then become really slow to respond. And start throwing nagios alerts for load. And when you log into the box you can see load as high as 70-80% !!! 

Sometimes the host becomes so unresponsive due to the load that you'll have to reboot it just you can log into the host!

And investigating with top and ps you find that it's THESE cron jobs that are causing the issue. 

What'll happen is that you will see like 30 or 40 or more chmod and chown jobs all running at the same time and spiking the load. 

And when you kill the jobs load returns to normal. 

If I read my cron correctly that should run only once every midnight. 

So two questions: am I reading this cron correctly ? That it is supposed to run once a day at midnight?

And does anybody have any ideas at all why all these chown and chmod commands would keep stacking up and spiking the load??

Thanks,

Tim

--

GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

[Peter Kristolaitis]

On 05/01/2015 12:31 PM, Tim Dunphy wrote:
>
> That produces the following crons in my crontab:
>
> # Puppet Name: chmod-files
> * 0 * * * /bin/find /var/www -type f -exec chmod -v 664 {} \;
> # Puppet Name: apache-chown
> * 0 * * * /bin/chown -R apache:ftpgroup /var/www
> # Puppet Name: chmod-directories
> * 0 * * * /bin/find /var/www -type d -exec chmod -v 775 {} \;
>
>
>

> If I read my cron correctly that should run only once every midnight.
>
> So two questions: am I reading this cron correctly ? That it is
> supposed to run once a day at midnight?
>
>

This isn't really a Puppet problem, but regardless:

Those cron entries are for "every minute during the zeroth hour". So at
00:00 it will run, then at 00:01, then 00:02, and so on all the way to
00:59, then will stop until 00:00 the next day. Therefore, if the
chmod/chown processes take more than 1 minute to run, they will stack up
during that period of time.

The entry you want is "0 0 * * *" -- that will run exactly once, at
00:00 each day. In your manifest, you can express this with "hour => 0,
minute => 0,".

[Tim Dunphy]

This isn't really a Puppet problem, but regardless:
Those cron entries are for "every minute during the zeroth hour". So at 00:00 it will run, then at 00:01, then 00:02, and so

 on all the way to 00:59, then will stop until 00:00 the next day. Therefore, if the chmod/chown processes take more than 1 minute to run, they will stack up during that period of time.
The entry you want is "0 0 * * *" -- that will run exactly once, at 00:00 each day.  In your manifest, you can express this with "hour => 0, minute => 0,".

Ok got it! I've corrected it and this is what I have now:

 cron { "apache-chown":

          command => "/bin/chown -R apache:ftpgroup /var/www",

          user    => 'root',

          hour    => 0,

          minute  => '0'

      }

  cron { "chmod-files":

          command => "/bin/find /var/www -type f -exec chmod -v 664 {} \;",

          user    => 'root',

          hour    => 0,

          minute  => '0'

      }

   cron { "chmod-directories":

          command => "/bin/find /var/www -type d -exec chmod -v 775 {} \;",

          user    => 'root',

          hour    => 0,

          minute  => '0'

      } 

Thanks for your input!

Tim

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5543AC42.6020407%40alter3d.ca.
For more options, visit https://groups.google.com/d/optout.

[Peter Bukowinski]

On May 1, 2015, at 12:52 PM, Tim Dunphy <bluet...@gmail.com> wrote:

This isn't really a Puppet problem, but regardless:
Those cron entries are for "every minute during the zeroth hour". So at 00:00 it will run, then at 00:01, then 00:02, and so

 on all the way to 00:59, then will stop until 00:00 the next day. Therefore, if the chmod/chown processes take more than 1 minute to run, they will stack up during that period of time.
The entry you want is "0 0 * * *" -- that will run exactly once, at 00:00 each day.  In your manifest, you can express this with "hour => 0, minute => 0,".

Ok got it! I've corrected it and this is what I have now:

 cron { "apache-chown":

          command => "/bin/chown -R apache:ftpgroup /var/www",

          user    => 'root',

          hour    => 0,

          minute  => '0'

      }

  cron { "chmod-files":

          command => "/bin/find /var/www -type f -exec chmod -v 664 {} \;",

          user    => 'root',

          hour    => 0,

          minute  => '0'

      }

   cron { "chmod-directories":

          command => "/bin/find /var/www -type d -exec chmod -v 775 {} \;",

          user    => 'root',

          hour    => 0,

          minute  => '0'

      } 

Thanks for your input!

Tim

In addition to Peter's excellent catch of the scheduling issue, depending on how many files you have in /var/www and how much other contention there is for disk I/O, your find commands could simply be getting bogged down traversing the directory tree. They are pretty inefficient, as written, so you should optimize them to minimize their impact.

Instead of this command, which will run one chmod command for each matching file,

    /bin/find /var/www -type f -exec chmod -v 664 {} \;

consider using the following command, which will exclude files that already have the correct permissions and will batch the found files into a single chmod command (note the plus sign instead of a semicolon):

    /bin/find /var/www -type f ! -perm 0644 -exec chmod -v 644 {} +

The directory chmod command would look pretty much the same:

    /bin/find /var/www -type d ! -perm 0775 -exec chmod -v 775 {} +

--

Peter Bukowinski

[Tim Dunphy]

In addition to Peter's excellent catch of the scheduling issue, depending on how many files you have in /var/www and how much other contention there is for disk I/O, your find commands could simply be getting bogged down traversing the directory tree. They are pretty inefficient, as written, so you should optimize them to minimize their impact.

Instead of this command, which will run one chmod command for each matching file,

    /bin/find /var/www -type f -exec chmod -v 664 {} \;

consider using the following command, which will exclude files that already have the correct permissions and will batch the found files into a single chmod command (note the plus sign instead of a semicolon):

    /bin/find /var/www -type f ! -perm 0644 -exec chmod -v 644 {} +

The directory chmod command would look pretty much the same:

    /bin/find /var/www -type d ! -perm 0775 -exec chmod -v 775 {} +

That's some pretty wonderful advice! I'll admit that my use of the find command may not have been that sophisticated. But I think you're example will help me with a better approach that I hope to learn from!

Thanks!

Tim

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CB45AA5E-A8C1-4422-BA4E-2B8D8FEBB7F4%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Denmat]

Also you could consider wrapping these in a flock to prevent actions piling up behind each other - like described here:

http://www.elevatedcode.com/2013/05/07/flock-for-cron-jobs.html

Many other examples on the net.

Den

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAOZy0emPCpHn1pdU_-38W5AMO-%2B%2BXVdOmsUpHMwdw%3DEUF74Ggg%40mail.gmail.com.

[Tim Dunphy]

Also you could consider wrapping these in a flock to prevent actions piling=

up behind each other - like described here:
http://www.elevatedcode.com/2013/05/07/flock-for-cron-jobs.html

Many other examples on the net.

Ok, Dan. That's really good info! Thanks and I'll be sure to check it out!!

Tim

Sent from my iPhone

On May 1, 2015, at 7:36 PM, Denmat <tu2b...@gmail.com> wrote:

Also you could consider wrapping these in a flock to prevent actions piling=

[Neil]

Hello

Looks like your intent is to run that daily?
I'd use cron.daily for that. That mean you script runs even if the server is not on at midnight.

what web applications do you run?

You do NOT generally want apache owning the web content. Apache should mostly have read access and only write to files it needs. That's storage areas in php apps etc. And those should be outside the webroot.

Looks like your users upload with ftp I'd do away with these checks and have you ftp server set the group and permissions so apache can read the files.

Neil

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/26679809-7CD9-4007-AE99-587EDC4E1132%40gmail.com.