Puppet 100% CPU usage
DieterVDW
I've been experimenting with Puppet for a few days now, and I really
like it.
But! I'm having real CPU usage problems.
Puppet is still happily eating away 100% CPU for almost an hour at a
time, with no apparent things happening.
(puppetd -tv --trace --debug, but nothing appearing in the console
during this.)
I believe the following resource is to blame:
file { "/some/data/dir":
owner => "$username",
group => "$username",
recurse => "true",
ensure => "directory",
checksum => undef
}
I just want this resource to make sure that all files in the directory
are owned by user and group $username.
/some/data/dir contains 300M in 6000+ files.
This resource executes swiftly, but after the last file has been
chown'd, the puppet hogs the CPU with 100% usage, lasting 40+ minutes.
After that the puppet process finishes normally.
So executing puppet takes about 45 minutes, of which 40 minutes the
CPU usage is 100% and nothing seems to happen.
Some information about my setup:
puppetd en puppetmasterd are 0.25.4
Both running on Xen Dom-U instances
uname -a:
Linux hostname 2.6.18.8 #2 SMP Wed May 27 15:54:07 CEST 2009 x86_64
GNU/Linux
Ubuntu intrepid 8.10
dpkg --list | grep ruby:
ii ruby 4.2 An interpreter of object-oriented scripting
ii ruby1.8 1.8.7.72-1 Interpreter of object-oriented scripting lan
Not really any logging to show, since nothing is logged...
Does anybody have any idea why this happens?
Is puppet just unable to handle large amounts of files and data?
Hoping to be able to use puppet in the near future,
Dieter Van de Walle
PS: I only just discovered this mailinglist, and yesterday I've made a
bugpost for this issue:
http://projects.reductivelabs.com/issues/3365
Not really sure if that was appropriate, hope you don't mind.
Patrick
On Mar 12, 2010, at 1:18 AM, DieterVDW wrote:
> I just want this resource to make sure that all files in the directory
> are owned by user and group $username.
> /some/data/dir contains 300M in 6000+ files.
>
Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
How to create a Ubuntu repository:
http://nerdica.com/?p=43
How to create packages:
http://www.debian-administration.org/articles/336
Good luck. I hope you get it working. With luck it's a bug that will get fixed.
Peter Meier
Peter Meier
DieterVDW
> Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
The problem is, I -am- using apt!
Those files are downloaded and installed using apt, I just want puppet
to make sure they are owned by a certain user and group.
That's the only thing puppet needs to do.
If I do the same in bash:
find directory/ | while read file ; do chown user:group "$file" ; done
real 0m28.119s
user 0m4.064s
sys 0m12.725s
I can live with overhead from slow ruby, etc etc ...
But apparently Puppet is 60x slower than bash for doing the same
action!
During my investigation for this problem I've seen a lot of people
saying things like:
"Puppet doesn't handle a folder with lots of files well"
60x times slower isn't really "not handling well". It's crap.
I'm a bit baffled by the defeatism I see in the Puppet community
concerning Puppet CPU usage.
I can't really believe people would call what I am experiencing "not
handling well", so I suppose I'm having another, worse, issue
concerning my setup?
Also the usual suspect for puppet not handling large file collections
well seems to be the checksumming.
But with that turned of, what's keeping puppet busy?
Puppet eats up 30 minutes of CPU time, I see two options:
- Puppet is doing something necessary for it's functioning during that
time, in which case somebody should know what it is doing. Anybody?
- Or the code is gravely flawed and needs to be fixed.
Is this a stroke of bad luck, or should I conclude that Puppet isn't
production ready?
I really can't have this on production machines...
DieterVDW
just discovered:
Changes:
Total: 4271
Resources:
Applied: 4271
Out of sync: 2138
Scheduled: 4435
Total: 115
Time:
Config retrieval: 1.36
Exec: 0.77
File: 19.23
Filebucket: 0.00
Host: 0.00
Package: 31.99
Schedule: 0.00
Service: 1.42
User: 0.01
Total: 54.78
warning: Value of 'preferred_serialization_format' (pson) is invalid
for report, using default (marshal)
notice: Finished catalog run in 1877.06 seconds
It seems to me the cause of the delays is not recorded in the time
overview?
D
Trevor Vaughan
19 in the File type.
The Package type probably won't eat up much CPU at all as it's just
fetching and installing packages (unless they are some huge packages).
You seem to be trying to recursively manage *something* using the File type.
What happens is that Puppet creates an actual file object for each
file and directory in the recursed path. Then, it takes the graph and
applies the changes to each file that has been seen. I'm not certain,
but I think that it builds this tree each time it runs to ensure that
nothing has changed under the path.
This processing has to happen on the client since the server has no
previous knowledge of the client's file structure.
That said, for deep directory/many file file statements, just spawn an
exec until the internal Puppet logic can be optimized.
Trevor
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>
--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
DieterVDW
Because I can't match the "Total: 54.78" line to the "notice: Finished
catalog run in 1877.06 seconds" .
Also, the strange thing is that the actual changing of owner and group
happens pretty fast.
I see these messages passing during +- 20s for all files:
notice: /File[/some/file]/owner: owner changed 'wronguser' to
'rightuser'
notice: /File[/some/file]/group: group changed 'wronggroup' to
'rightgroup'
It's AFTER this has happened that puppetd stalls for 30+ minutes.
So it must be some kind of wrapup action... ?
D
> > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699
DieterVDW
If I execute puppetd -tv and wait until all files have been chown'd,
and then hit CTRL-C and run puppetd again,
the second time, puppetd finishes in about a minute.
The total process of running puppetd twice takes about 4 minutes .
So by manipulating the execution of puppetd using a CTRL-C keystroke I
can reach the intended end state in 4 minutes.
If I leave puppetd to do it by itself, it takes 30+ minutes of 100%
CPU usage...
Martin Wheldon
Is this directory the only resource you are managing or are there others.
If not then try disabling all other resources and see if you still get
the same issue?
Best Regards
Martin Wheldon
Trevor Vaughan
That's what I get for answering e-mails pre-caffeine.
Did you try spitting out a --debug to a file and then post-analyzing that?
Trevor
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>
--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
Brice Figureau
This is a known (at least to me) regression in 0.25: Puppet still uses
md5 checksums on every managed files, even when it is not needed.
> Puppet eats up 30 minutes of CPU time, I see two options:
> - Puppet is doing something necessary for it's functioning during that
> time, in which case somebody should know what it is doing. Anybody?
It is checksumming every file.
> - Or the code is gravely flawed and needs to be fixed.
This is a bug, but I won't say it is gravely flawed, because in the end
you have the correct result.
> Is this a stroke of bad luck, or should I conclude that Puppet isn't
> production ready?
Puppet is production ready, as many here are running it on critical
production systems or on big infrastructure.
It's just that like every piece of software it can contain small defects
that needs to be reported and fixed.
> I really can't have this on production machines...
I don't see why. Would you throw the complete software and revert
managing everything by hand because you encountered a small performance
issue for which you can find correct work-arounds, like an exec of a
recursive chmod?
--
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!
Thomas Mueller
> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
>> Puppet doesn't handle a folder with lots of files well. It handles
>> large files even worse. The standard advice is "Try putting the files
>> in a package and distributing them using apt." Another common answer
>> is to try combining exec and rsync. I ended up using apt. Here are
>> the tutorials I used:
>
> The problem is, I -am- using apt!
> Those files are downloaded and installed using apt, I just want puppet
> to make sure they are owned by a certain user and group. That's the only
> thing puppet needs to do.
if the files are installed by apt and do not have the permissions you
need, you may need to have a look at dpkg-statoverride (man dpkg-
statoverride).
- Thomas
DieterVDW
I must admit I'd really like to like puppet, so I'm a bit agitated by
this issue, and won't rest till I know the cause :) .
Sorry for the information overload, hope you don't mind.
First of all, I've followed Martin's suggestion and simplified my
config.
This is my configuration in it's entirety:
nodes.pp:
node 'puptest.domain.com' {
include onlyone
}
onlyone.pp:
class onlyone {
$pack_list = ["large-package"]
package { $pack_list:
ensure => installed
}
$username = "someuser"
user { "$username":
comment => "Some user",
home => "/home/$username",
managehome => "true",
shell => '/bin/bash'
}
file { "/data/installed/by/large-package":
owner => "$username",
group => "$username",
recurse => "true",
require => [ User["$username"], Package["large-package"] ],
ensure => "directory",
checksum => undef
}
}
No effect though, the issue still occurs after all files have been
chown'd.
I've also tried attaching gdb to the puppetd process and breaking
during high CPU usage.
If I do a backtrace then, most of the backtrace contains ?? -> can't
derive.
If I only leave the methods gdb can determine, this is the stacktrace:
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
st_foreach () from /usr/lib/libruby1.8.so.1.8
ruby_xmalloc () from /usr/lib/libruby1.8.so.1.8
rb_yield_values () from /usr/lib/libruby1.8.so.1.8
rb_ary_each () from /usr/lib/libruby1.8.so.1.8
rb_funcall2 () from /usr/lib/libruby1.8.so.1.8
rb_iterate () from /usr/lib/libruby1.8.so.1.8
rb_block_call () from /usr/lib/libruby1.8.so.1.8
No idea if this is of any use ...
I've also tried using --debug, but no debug statements are printed
either during the entire time puppetd is hogging the CPU.
Will have a second look though, maybe something interesting is printed
earlier...
On Mar 12, 1:58 pm, Martin Wheldon <mwhel...@googlemail.com> wrote:
> Hi,
>
> Is this directory the only resource you are managing or are there others.
> If not then try disabling all other resources and see if you still get
> the same issue?
>
> Best Regards
>
> Martin Wheldon
>
DieterVDW
wrote:
> It is checksumming every file.
Aha. This is a known issue?
Is there a bugreport for this?
Possibly a patch?
Is it normal no debug information is shown during this process?
Brice Figureau
> On Mar 12, 2:21 pm, Brice Figureau <brice-pup...@daysofwonder.com>
> wrote:
> > It is checksumming every file.
>
> Aha. This is a known issue?
At least to me :-)
> Is there a bugreport for this?
I think so:
http://projects.reductivelabs.com/issues/3170
and
http://projects.reductivelabs.com/issues/2929
and
http://projects.reductivelabs.com/issues/2630
> Possibly a patch?
None that I am aware.
> Is it normal no debug information is shown during this process?
Yes it is, because it happens in the "file serving" code as part of
getting information about a given file. This part of the process doesn't
know getting the checksum is unnecessary (in fact this is the same code
that is used to serve file metadata over the wire, except in this case
it is locally executed).
This isssue seems to be frequently raised, so I think I'll work on it,
if I have some spare time soon.
Trevor Vaughan
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>
>
--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
Peter Meier
DieterVDW
wrote:
> Yes it is, because it happens in the "file serving" code as part of
> getting information about a given file. This part of the process doesn't
> know getting the checksum is unnecessary (in fact this is the same code
> that is used to serve file metadata over the wire, except in this case
> it is locally executed).
And this happens also if the files aren't fileserved?
Bug 3170 seems to suggest checksum => undef should work though?
Anyway, it would indeed be great if this were solved.
I wish I understood Ruby, would love to peek under the hood of
puppet...
Thanks for taking the time to look at this, and making our lives
easier :) .
Brice Figureau
> On Mar 12, 2:48 pm, Brice Figureau <brice-pup...@daysofwonder.com>
> wrote:
> > Yes it is, because it happens in the "file serving" code as part of
> > getting information about a given file. This part of the process doesn't
> > know getting the checksum is unnecessary (in fact this is the same code
> > that is used to serve file metadata over the wire, except in this case
> > it is locally executed).
>
> And this happens also if the files aren't fileserved?
Yes. The checksum are needed when remotely fileserving (and sometimes
locally), thus they are also computed when managing local (not sourced)
files, because the same code is run.
> Bug 3170 seems to suggest checksum => undef should work though?
It used to work in 0.24.x, but we lost this in 0.25 when the whole file
serving system was refactored (and greatly improved despite what you
found).
> Anyway, it would indeed be great if this were solved.
I concur. This is annoying. I have a 5 minutes hack to disable
checksumming, but it far from being production ready. I don't think I'll
have time this week-end to work on this, but if I can find 5 minutes,
I'll try to send a preliminary patch to puppet-dev.
> I wish I understood Ruby, would love to peek under the hood of
> puppet...
I learnt ruby by fixing bugs in Puppet. You can do the same :-)
> Thanks for taking the time to look at this, and making our lives
> easier :) .
Russ Allbery
Under most circumstances, though, all you need to do is ensure that the
files have the correct ownership in the *.deb when you create it and then
dpkg will do the correct thing when installing the package.
--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>
Patrick
On Mar 12, 2010, at 3:30 AM, DieterVDW wrote:
> The problem is, I -am- using apt!
> Those files are downloaded and installed using apt, I just want puppet
> to make sure they are owned by a certain user and group.
> That's the only thing puppet needs to do.
>
I didn't read the bug report before my last reply. I know this isn't what you want, but how about adding chown to the Makefile used to generate the deb. Also, you could just run chown using Exec.
Alan Barrett
> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
> Those files are downloaded and installed using apt, I just want puppet
> to make sure they are owned by a certain user and group.
> That's the only thing puppet needs to do.
As a workaround, instead of a recursive file resource like this:
file { "/some/dir":
ensure => directory,
recurse => inf,
owner => "someuser",
group => "somegroup",
require => Package["the package that created the files"],
}
try an exec resorce like this:
exec { "fix permissions in /some/dir":
command => "chown -R someuser:somegroup /some/dir",
require => Package["the package that created the files"],
}
The exec will be much faster, but it will run every time (adding
a message in the log), even if the files already have the correct
ownership. To get rid of the unwanted log message at the expense of
slower execution, add
onlyif => "some command to check whether there's a problem",
The onlyif command could use something involving "find" to print the
names of bad files, and "test" to see whether find's output was empty.
--apb (Alan Barrett)
