high cpu usage and slow puppet performance..
barrowkwan
longer time to finish a catalog run
file { "/some/path":
owner => "user1",
group => "group1",
ensure => directory,
recurse => true
}
especially when /some/path contain lot of files and sub-directories...
anyone have the same problem?
Lindsay Holmwood
Puppet has to do a stat and md5sum on every file under that path,
which depending on your system could take forever and push the iowait
up.
I've had this problem with a Rails app that stored session on the
filesystem that was deployed through Puppet's file type(!).
Recursion on the file type should be used in moderation. :-)
Cheers,
Lindsay
>
> >
Robin Lee Powell
>
> On 12/03/2009, at 18:41, barrowkwan <barro...@gmail.com> wrote:
>
> >
> > looks like the following could cause high cpu usage and take
> > puppet longer time to finish a catalog run
> >
> >
> > file { "/some/path":
> > owner => "user1",
> > group => "group1",
> > ensure => directory,
> > recurse => true
> > }
> >
> > especially when /some/path contain lot of files and
> > sub-directories...
> >
> > anyone have the same problem?
>
>
> Yes, and it's totally reasonable for it to.
>
> Puppet has to do a stat and md5sum on every file under that path,
It *what*? Just for an owner and group check? Why?
Can that be turned off? We got a lot of performance out of cfengine
at work by telling it to use mtime instead of checksumming?
-Robin
--
They say: "The first AIs will be built by the military as weapons."
And I'm thinking: "Does it even occur to you to try for something
other than the default outcome?" -- http://shorl.com/tydruhedufogre
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/
Lindsay Holmwood
>
> On Thu, Mar 12, 2009 at 06:51:59PM +1100, Lindsay Holmwood wrote:
>>
>> Puppet has to do a stat and md5sum on every file under that path,
>
> It *what*? Just for an owner and group check? Why?
>
> Can that be turned off? We got a lot of performance out of cfengine
> at work by telling it to use mtime instead of checksumming?
I should have mentioned that this was under an older version of Puppet
(almost 2 years ago). I can't comment on how the current code works.
Lindsay
--
http://holmwood.id.au/~lindsay/ (me)
Bruce Richardson
>
> anyone have the same problem?
Which filesystem are you using? Some of them perform badly for tasks
that iterate over a large directory heirarchy. JFS is particularly bad
at this.
--
Bruce
Remember you're a Womble.
Brice Figureau
Do you have checksums enabled by default?
See my comments (look for masterzen comments) in this bug:
http://projects.reductivelabs.com/issues/1973
Basically, if you have a default checksum in your manifests, then what
puppet does is computing all the checksums for all the files (and
storing them in RAM too). This can take a lots of time if you only want
to manage ownership.
The solution is to rewrite to:
file { "/some/path":
owner => "user1",
group => "group1",
ensure => directory,
recurse => true,
checksum => undef
}
Also you can read my blog post which talks about this issue:
http://www.masterzen.fr/2009/02/21/help-puppetd-is-eating-my-server/
--
Brice Figureau
My Blog: http://www.masterzen.fr/
barrowkwan
I am a bit confused. sometime it said the patch is on 0.25 but
sometimes it said 0.24.8, so which version/release will have this
fix? this problem really hurt our system... :(
thanks!
On Mar 12, 3:25 am, Brice Figureau <brice-pup...@daysofwonder.com>
wrote:
Brice Figureau
> very nice explanation in your blog. Thanks!
>
> I am a bit confused. sometime it said the patch is on 0.25 but
> sometimes it said 0.24.8, so which version/release will have this
> fix? this problem really hurt our system... :(
Are you talking about the fix for the bug #1469?
If yes, it will be in 0.25 only (it would be very difficult to backport
it as it was possible to implement it for 0.25 because Luke did a huge
refactor of the file type).
Note, that this bug improves the situation only if your file{} resource
has the source attribute (downloading files from the puppetmaster).
In your case, I think using checksum => undef should be enough to
tremendously speed-up this operation.
[snipped original discussion]
--
Brice Figureau
Days of Wonder
http://www.daysofwonder.com
Joel Krauska
Peter Meier
> Are you talking about the fix for the bug #1469?
> If yes, it will be in 0.25 only (it would be very difficult to backport
> it as it was possible to implement it for 0.25 because Luke did a huge
> refactor of the file type).
>
> Note, that this bug improves the situation only if your file{} resource
> has the source attribute (downloading files from the puppetmaster).
>
> In your case, I think using checksum => undef should be enough to
> tremendously speed-up this operation.
I'm currently a bit in the same or at least similar situation as we try
to ensure that certain folders and their content of webapplications
always have the right permission. so our idea have been to use the file
resource on these folders, with recursive set to true, etc.
Speed doesn't matter that much, what matters more is the memory usage.
So this is our test-scenario:
--- foo.pp ---
$documentroot='/var/www/vhosts/path/to/joomla/installation/www'
file{ [ "$documentroot/administrator/backups",
"$documentroot/administrator/components",
"$documentroot/administrator/language",
"$documentroot/administrator/modules",
"$documentroot/administrator/templates",
"$documentroot/components",
"$documentroot/images",
"$documentroot/language",
"$documentroot/media",
"$documentroot/modules",
"$documentroot/plugins",
"$documentroot/templates",
"$documentroot/cache",
"$documentroot/administrator/cache" ]:
owner => 'user',
group => 'group',
mode => 0660,
recurse => true,
backup => false,
checksum => undef,
}
---
a run of just that manifest (so puppet foo.pp) can take up to:
VIRT RES
273m 193m
of Memory Usage.
which is for:
# du -sch administrator/backups administrator/components
administrator/language administrator/modules administrator/templates
components images language media modules plugins templates cache
administrator/cache
16K administrator/backups
5.8M administrator/components
4.2M administrator/language
664K administrator/modules
1.9M administrator/templates
3.2M components
892K images
2.6M language
616K media
1.8M modules
7.8M plugins
9.0M templates
224K cache
16K administrator/cache
39M total
# find administrator/backups administrator/components
administrator/language administrator/modules administrator/templates
components images language media modules plugins templates cache
administrator/cache | wc -l
4336
# ruby -version
ruby 1.8.6 (2007-09-24 patchlevel 111) [x86_64-linux]
in my opinion still "a bit" too much.
if I don't set recurse (and this is the way we go currently)
everything's fine. So it certainly has to do with recursing a file tree.
I'm running nearly latest head, at least #1965 [1] is applied.
So I'm waiting for 0.25! :) Would the standalone client already be
usable? So I could give feedback if the same manifest runs smoother.
thanks for all the work and cheers
pete
Brice Figureau
Yes, that's a lot. Unfortunately for such manifests puppet has to create
a child file object per file it found while recursing the hierarchy.
If your aim is just to change/maintain ownership/perms, maybe an exec of
a recusive chown or combination of find/chown would be better, something
akin to:
exec {
"chown-joomla-files":
command => "find -type f ! -user 'user' | xargs chown user"
}
...
You see the idea...
Yes, the exec will execute on each catalog run, but that shouldn't be an
issue.
> if I don't set recurse (and this is the way we go currently)
> everything's fine. So it certainly has to do with recursing a file tree.
> I'm running nearly latest head, at least #1965 [1] is applied.
#1965 won't help on the client side. I didn't reread the patch, but if I
remember correctly it helps for storing the path used in the AST tree.
I think I'll have a closer look to this issue soon.