I've been experimenting with Puppet for a few days now, and I really like it. But! I'm having real CPU usage problems.
Puppet is still happily eating away 100% CPU for almost an hour at a time, with no apparent things happening. (puppetd -tv --trace --debug, but nothing appearing in the console during this.)
I believe the following resource is to blame: file { "/some/data/dir": owner => "$username", group => "$username", recurse => "true", ensure => "directory", checksum => undef
}
I just want this resource to make sure that all files in the directory are owned by user and group $username. /some/data/dir contains 300M in 6000+ files. This resource executes swiftly, but after the last file has been chown'd, the puppet hogs the CPU with 100% usage, lasting 40+ minutes. After that the puppet process finishes normally. So executing puppet takes about 45 minutes, of which 40 minutes the CPU usage is 100% and nothing seems to happen.
Some information about my setup: puppetd en puppetmasterd are 0.25.4
Both running on Xen Dom-U instances uname -a: Linux hostname 2.6.18.8 #2 SMP Wed May 27 15:54:07 CEST 2009 x86_64 GNU/Linux Ubuntu intrepid 8.10
dpkg --list | grep ruby: ii ruby 4.2 An interpreter of object-oriented scripting ii ruby1.8 1.8.7.72-1 Interpreter of object-oriented scripting lan
Not really any logging to show, since nothing is logged...
Does anybody have any idea why this happens? Is puppet just unable to handle large amounts of files and data?
Hoping to be able to use puppet in the near future,
Dieter Van de Walle
PS: I only just discovered this mailinglist, and yesterday I've made a bugpost for this issue: http://projects.reductivelabs.com/issues/3365 Not really sure if that was appropriate, hope you don't mind.
> I just want this resource to make sure that all files in the directory > are owned by user and group $username. > /some/data/dir contains 300M in 6000+ files.
Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
> Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
The problem is, I -am- using apt! Those files are downloaded and installed using apt, I just want puppet to make sure they are owned by a certain user and group. That's the only thing puppet needs to do.
If I do the same in bash: find directory/ | while read file ; do chown user:group "$file" ; done
real 0m28.119s user 0m4.064s sys 0m12.725s
I can live with overhead from slow ruby, etc etc ... But apparently Puppet is 60x slower than bash for doing the same action!
During my investigation for this problem I've seen a lot of people saying things like: "Puppet doesn't handle a folder with lots of files well" 60x times slower isn't really "not handling well". It's crap. I'm a bit baffled by the defeatism I see in the Puppet community concerning Puppet CPU usage.
I can't really believe people would call what I am experiencing "not handling well", so I suppose I'm having another, worse, issue concerning my setup?
Also the usual suspect for puppet not handling large file collections well seems to be the checksumming. But with that turned of, what's keeping puppet busy?
Puppet eats up 30 minutes of CPU time, I see two options: - Puppet is doing something necessary for it's functioning during that time, in which case somebody should know what it is doing. Anybody? - Or the code is gravely flawed and needs to be fixed.
Is this a stroke of bad luck, or should I conclude that Puppet isn't production ready? I really can't have this on production machines...
> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
> > Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
> The problem is, I -am- using apt! > Those files are downloaded and installed using apt, I just want puppet > to make sure they are owned by a certain user and group. > That's the only thing puppet needs to do.
> If I do the same in bash: > find directory/ | while read file ; do chown user:group "$file" ; done
> real 0m28.119s > user 0m4.064s > sys 0m12.725s
> I can live with overhead from slow ruby, etc etc ... > But apparently Puppet is 60x slower than bash for doing the same > action!
> During my investigation for this problem I've seen a lot of people > saying things like: > "Puppet doesn't handle a folder with lots of files well" > 60x times slower isn't really "not handling well". It's crap. > I'm a bit baffled by the defeatism I see in the Puppet community > concerning Puppet CPU usage.
> I can't really believe people would call what I am experiencing "not > handling well", so I suppose I'm having another, worse, issue > concerning my setup?
> Also the usual suspect for puppet not handling large file collections > well seems to be the checksumming. > But with that turned of, what's keeping puppet busy?
> Puppet eats up 30 minutes of CPU time, I see two options: > - Puppet is doing something necessary for it's functioning during that > time, in which case somebody should know what it is doing. Anybody? > - Or the code is gravely flawed and needs to be fixed.
> Is this a stroke of bad luck, or should I conclude that Puppet isn't > production ready? > I really can't have this on production machines...
So, it looks like 30 minutes is being taken up in the Package type and 19 in the File type.
The Package type probably won't eat up much CPU at all as it's just fetching and installing packages (unless they are some huge packages).
You seem to be trying to recursively manage *something* using the File type.
What happens is that Puppet creates an actual file object for each file and directory in the recursed path. Then, it takes the graph and applies the changes to each file that has been seen. I'm not certain, but I think that it builds this tree each time it runs to ensure that nothing has changed under the path.
This processing has to happen on the client since the server has no previous knowledge of the client's file structure.
That said, for deep directory/many file file statements, just spawn an exec until the internal Puppet logic can be optimized.
On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: > Some more information from the (undocumented) --summarize option I > just discovered:
> Changes: > Total: 4271 > Resources: > Applied: 4271 > Out of sync: 2138 > Scheduled: 4435 > Total: 115 > Time: > Config retrieval: 1.36 > Exec: 0.77 > File: 19.23 > Filebucket: 0.00 > Host: 0.00 > Package: 31.99 > Schedule: 0.00 > Service: 1.42 > User: 0.01 > Total: 54.78 > warning: Value of 'preferred_serialization_format' (pson) is invalid > for report, using default (marshal) > notice: Finished catalog run in 1877.06 seconds
> It seems to me the cause of the delays is not recorded in the time > overview?
> D
> On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
>> > Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
>> The problem is, I -am- using apt! >> Those files are downloaded and installed using apt, I just want puppet >> to make sure they are owned by a certain user and group. >> That's the only thing puppet needs to do.
>> If I do the same in bash: >> find directory/ | while read file ; do chown user:group "$file" ; done
>> real 0m28.119s >> user 0m4.064s >> sys 0m12.725s
>> I can live with overhead from slow ruby, etc etc ... >> But apparently Puppet is 60x slower than bash for doing the same >> action!
>> During my investigation for this problem I've seen a lot of people >> saying things like: >> "Puppet doesn't handle a folder with lots of files well" >> 60x times slower isn't really "not handling well". It's crap. >> I'm a bit baffled by the defeatism I see in the Puppet community >> concerning Puppet CPU usage.
>> I can't really believe people would call what I am experiencing "not >> handling well", so I suppose I'm having another, worse, issue >> concerning my setup?
>> Also the usual suspect for puppet not handling large file collections >> well seems to be the checksumming. >> But with that turned of, what's keeping puppet busy?
>> Puppet eats up 30 minutes of CPU time, I see two options: >> - Puppet is doing something necessary for it's functioning during that >> time, in which case somebody should know what it is doing. Anybody? >> - Or the code is gravely flawed and needs to be fixed.
>> Is this a stroke of bad luck, or should I conclude that Puppet isn't >> production ready? >> I really can't have this on production machines...
> -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Aren't the time values in seconds? Because I can't match the "Total: 54.78" line to the "notice: Finished catalog run in 1877.06 seconds" .
Also, the strange thing is that the actual changing of owner and group happens pretty fast. I see these messages passing during +- 20s for all files: notice: /File[/some/file]/owner: owner changed 'wronguser' to 'rightuser' notice: /File[/some/file]/group: group changed 'wronggroup' to 'rightgroup'
It's AFTER this has happened that puppetd stalls for 30+ minutes. So it must be some kind of wrapup action... ?
D
On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote:
> So, it looks like 30 minutes is being taken up in the Package type and > 19 in the File type.
> The Package type probably won't eat up much CPU at all as it's just > fetching and installing packages (unless they are some huge packages).
> You seem to be trying to recursively manage *something* using the File type.
> What happens is that Puppet creates an actual file object for each > file and directory in the recursed path. Then, it takes the graph and > applies the changes to each file that has been seen. I'm not certain, > but I think that it builds this tree each time it runs to ensure that > nothing has changed under the path.
> This processing has to happen on the client since the server has no > previous knowledge of the client's file structure.
> That said, for deep directory/many file file statements, just spawn an > exec until the internal Puppet logic can be optimized.
> Trevor
> On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: > > Some more information from the (undocumented) --summarize option I > > just discovered:
> > It seems to me the cause of the delays is not recorded in the time > > overview?
> > D
> > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
> >> > Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
> >> The problem is, I -am- using apt! > >> Those files are downloaded and installed using apt, I just want puppet > >> to make sure they are owned by a certain user and group. > >> That's the only thing puppet needs to do.
> >> If I do the same in bash: > >> find directory/ | while read file ; do chown user:group "$file" ; done
> >> real 0m28.119s > >> user 0m4.064s > >> sys 0m12.725s
> >> I can live with overhead from slow ruby, etc etc ... > >> But apparently Puppet is 60x slower than bash for doing the same > >> action!
> >> During my investigation for this problem I've seen a lot of people > >> saying things like: > >> "Puppet doesn't handle a folder with lots of files well" > >> 60x times slower isn't really "not handling well". It's crap. > >> I'm a bit baffled by the defeatism I see in the Puppet community > >> concerning Puppet CPU usage.
> >> I can't really believe people would call what I am experiencing "not > >> handling well", so I suppose I'm having another, worse, issue > >> concerning my setup?
> >> Also the usual suspect for puppet not handling large file collections > >> well seems to be the checksumming. > >> But with that turned of, what's keeping puppet busy?
> >> Puppet eats up 30 minutes of CPU time, I see two options: > >> - Puppet is doing something necessary for it's functioning during that > >> time, in which case somebody should know what it is doing. Anybody? > >> - Or the code is gravely flawed and needs to be fixed.
> >> Is this a stroke of bad luck, or should I conclude that Puppet isn't > >> production ready? > >> I really can't have this on production machines...
> > -- > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.
Also, whatever Puppetd is doing then, it seems to be unnecessary.
If I execute puppetd -tv and wait until all files have been chown'd, and then hit CTRL-C and run puppetd again, the second time, puppetd finishes in about a minute. The total process of running puppetd twice takes about 4 minutes .
So by manipulating the execution of puppetd using a CTRL-C keystroke I can reach the intended end state in 4 minutes. If I leave puppetd to do it by itself, it takes 30+ minutes of 100% CPU usage...
On Mar 12, 1:43 pm, DieterVDW <dieter...@gmail.com> wrote:
> Aren't the time values in seconds? > Because I can't match the "Total: 54.78" line to the "notice: Finished > catalog run in 1877.06 seconds" .
> Also, the strange thing is that the actual changing of owner and group > happens pretty fast. > I see these messages passing during +- 20s for all files: > notice: /File[/some/file]/owner: owner changed 'wronguser' to > 'rightuser' > notice: /File[/some/file]/group: group changed 'wronggroup' to > 'rightgroup'
> It's AFTER this has happened that puppetd stalls for 30+ minutes. > So it must be some kind of wrapup action... ?
> D
> On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote:
> > So, it looks like 30 minutes is being taken up in the Package type and > > 19 in the File type.
> > The Package type probably won't eat up much CPU at all as it's just > > fetching and installing packages (unless they are some huge packages).
> > You seem to be trying to recursively manage *something* using the File type.
> > What happens is that Puppet creates an actual file object for each > > file and directory in the recursed path. Then, it takes the graph and > > applies the changes to each file that has been seen. I'm not certain, > > but I think that it builds this tree each time it runs to ensure that > > nothing has changed under the path.
> > This processing has to happen on the client since the server has no > > previous knowledge of the client's file structure.
> > That said, for deep directory/many file file statements, just spawn an > > exec until the internal Puppet logic can be optimized.
> > Trevor
> > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: > > > Some more information from the (undocumented) --summarize option I > > > just discovered:
> > > It seems to me the cause of the delays is not recorded in the time > > > overview?
> > > D
> > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: > > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
> > >> > Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
> > >> The problem is, I -am- using apt! > > >> Those files are downloaded and installed using apt, I just want puppet > > >> to make sure they are owned by a certain user and group. > > >> That's the only thing puppet needs to do.
> > >> If I do the same in bash: > > >> find directory/ | while read file ; do chown user:group "$file" ; done
> > >> real 0m28.119s > > >> user 0m4.064s > > >> sys 0m12.725s
> > >> I can live with overhead from slow ruby, etc etc ... > > >> But apparently Puppet is 60x slower than bash for doing the same > > >> action!
> > >> During my investigation for this problem I've seen a lot of people > > >> saying things like: > > >> "Puppet doesn't handle a folder with lots of files well" > > >> 60x times slower isn't really "not handling well". It's crap. > > >> I'm a bit baffled by the defeatism I see in the Puppet community > > >> concerning Puppet CPU usage.
> > >> I can't really believe people would call what I am experiencing "not > > >> handling well", so I suppose I'm having another, worse, issue > > >> concerning my setup?
> > >> Also the usual suspect for puppet not handling large file collections > > >> well seems to be the checksumming. > > >> But with that turned of, what's keeping puppet busy?
> > >> Puppet eats up 30 minutes of CPU time, I see two options: > > >> - Puppet is doing something necessary for it's functioning during that > > >> time, in which case somebody should know what it is doing. Anybody? > > >> - Or the code is gravely flawed and needs to be fixed.
> > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't > > >> production ready? > > >> I really can't have this on production machines...
> > > -- > > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > > > To post to this group, send email to puppet-users@googlegroups.com. > > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.
Is this directory the only resource you are managing or are there others. If not then try disabling all other resources and see if you still get the same issue?
On Fri, Mar 12, 2010 at 12:55 PM, DieterVDW <dieter...@gmail.com> wrote: > Also, whatever Puppetd is doing then, it seems to be unnecessary.
> If I execute puppetd -tv and wait until all files have been chown'd, > and then hit CTRL-C and run puppetd again, > the second time, puppetd finishes in about a minute. > The total process of running puppetd twice takes about 4 minutes .
> So by manipulating the execution of puppetd using a CTRL-C keystroke I > can reach the intended end state in 4 minutes. > If I leave puppetd to do it by itself, it takes 30+ minutes of 100% > CPU usage...
> On Mar 12, 1:43 pm, DieterVDW <dieter...@gmail.com> wrote: >> Aren't the time values in seconds? >> Because I can't match the "Total: 54.78" line to the "notice: Finished >> catalog run in 1877.06 seconds" .
>> Also, the strange thing is that the actual changing of owner and group >> happens pretty fast. >> I see these messages passing during +- 20s for all files: >> notice: /File[/some/file]/owner: owner changed 'wronguser' to >> 'rightuser' >> notice: /File[/some/file]/group: group changed 'wronggroup' to >> 'rightgroup'
>> It's AFTER this has happened that puppetd stalls for 30+ minutes. >> So it must be some kind of wrapup action... ?
>> D
>> On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote:
>> > So, it looks like 30 minutes is being taken up in the Package type and >> > 19 in the File type.
>> > The Package type probably won't eat up much CPU at all as it's just >> > fetching and installing packages (unless they are some huge packages).
>> > You seem to be trying to recursively manage *something* using the File type.
>> > What happens is that Puppet creates an actual file object for each >> > file and directory in the recursed path. Then, it takes the graph and >> > applies the changes to each file that has been seen. I'm not certain, >> > but I think that it builds this tree each time it runs to ensure that >> > nothing has changed under the path.
>> > This processing has to happen on the client since the server has no >> > previous knowledge of the client's file structure.
>> > That said, for deep directory/many file file statements, just spawn an >> > exec until the internal Puppet logic can be optimized.
>> > Trevor
>> > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: >> > > Some more information from the (undocumented) --summarize option I >> > > just discovered:
>> > > It seems to me the cause of the delays is not recorded in the time >> > > overview?
>> > > D
>> > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: >> > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
>> > >> > Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
>> > >> The problem is, I -am- using apt! >> > >> Those files are downloaded and installed using apt, I just want puppet >> > >> to make sure they are owned by a certain user and group. >> > >> That's the only thing puppet needs to do.
>> > >> If I do the same in bash: >> > >> find directory/ | while read file ; do chown user:group "$file" ; done
>> > >> real 0m28.119s >> > >> user 0m4.064s >> > >> sys 0m12.725s
>> > >> I can live with overhead from slow ruby, etc etc ... >> > >> But apparently Puppet is 60x slower than bash for doing the same >> > >> action!
>> > >> During my investigation for this problem I've seen a lot of people >> > >> saying things like: >> > >> "Puppet doesn't handle a folder with lots of files well" >> > >> 60x times slower isn't really "not handling well". It's crap. >> > >> I'm a bit baffled by the defeatism I see in the Puppet community >> > >> concerning Puppet CPU usage.
>> > >> I can't really believe people would call what I am experiencing "not >> > >> handling well", so I suppose I'm having another, worse, issue >> > >> concerning my setup?
>> > >> Also the usual suspect for puppet not handling large file collections >> > >> well seems to be the checksumming. >> > >> But with that turned of, what's keeping puppet busy?
>> > >> Puppet eats up 30 minutes of CPU time, I see two options: >> > >> - Puppet is doing something necessary for it's functioning during that >> > >> time, in which case somebody should know what it is doing. Anybody? >> > >> - Or the code is gravely flawed and needs to be fixed.
>> > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't >> > >> production ready? >> > >> I really can't have this on production machines...
>> > > -- >> > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> > > To post to this group, send email to puppet-users@googlegroups.com. >> > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >> > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.
>> > -- This account not approved for unencrypted proprietary information --
> -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Fri, Mar 12, 2010 at 7:55 AM, DieterVDW <dieter...@gmail.com> wrote: > Also, whatever Puppetd is doing then, it seems to be unnecessary.
> If I execute puppetd -tv and wait until all files have been chown'd, > and then hit CTRL-C and run puppetd again, > the second time, puppetd finishes in about a minute. > The total process of running puppetd twice takes about 4 minutes .
> So by manipulating the execution of puppetd using a CTRL-C keystroke I > can reach the intended end state in 4 minutes. > If I leave puppetd to do it by itself, it takes 30+ minutes of 100% > CPU usage...
> On Mar 12, 1:43 pm, DieterVDW <dieter...@gmail.com> wrote: >> Aren't the time values in seconds? >> Because I can't match the "Total: 54.78" line to the "notice: Finished >> catalog run in 1877.06 seconds" .
>> Also, the strange thing is that the actual changing of owner and group >> happens pretty fast. >> I see these messages passing during +- 20s for all files: >> notice: /File[/some/file]/owner: owner changed 'wronguser' to >> 'rightuser' >> notice: /File[/some/file]/group: group changed 'wronggroup' to >> 'rightgroup'
>> It's AFTER this has happened that puppetd stalls for 30+ minutes. >> So it must be some kind of wrapup action... ?
>> D
>> On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote:
>> > So, it looks like 30 minutes is being taken up in the Package type and >> > 19 in the File type.
>> > The Package type probably won't eat up much CPU at all as it's just >> > fetching and installing packages (unless they are some huge packages).
>> > You seem to be trying to recursively manage *something* using the File type.
>> > What happens is that Puppet creates an actual file object for each >> > file and directory in the recursed path. Then, it takes the graph and >> > applies the changes to each file that has been seen. I'm not certain, >> > but I think that it builds this tree each time it runs to ensure that >> > nothing has changed under the path.
>> > This processing has to happen on the client since the server has no >> > previous knowledge of the client's file structure.
>> > That said, for deep directory/many file file statements, just spawn an >> > exec until the internal Puppet logic can be optimized.
>> > Trevor
>> > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: >> > > Some more information from the (undocumented) --summarize option I >> > > just discovered:
>> > > It seems to me the cause of the delays is not recorded in the time >> > > overview?
>> > > D
>> > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: >> > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
>> > >> > Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
>> > >> The problem is, I -am- using apt! >> > >> Those files are downloaded and installed using apt, I just want puppet >> > >> to make sure they are owned by a certain user and group. >> > >> That's the only thing puppet needs to do.
>> > >> If I do the same in bash: >> > >> find directory/ | while read file ; do chown user:group "$file" ; done
>> > >> real 0m28.119s >> > >> user 0m4.064s >> > >> sys 0m12.725s
>> > >> I can live with overhead from slow ruby, etc etc ... >> > >> But apparently Puppet is 60x slower than bash for doing the same >> > >> action!
>> > >> During my investigation for this problem I've seen a lot of people >> > >> saying things like: >> > >> "Puppet doesn't handle a folder with lots of files well" >> > >> 60x times slower isn't really "not handling well". It's crap. >> > >> I'm a bit baffled by the defeatism I see in the Puppet community >> > >> concerning Puppet CPU usage.
>> > >> I can't really believe people would call what I am experiencing "not >> > >> handling well", so I suppose I'm having another, worse, issue >> > >> concerning my setup?
>> > >> Also the usual suspect for puppet not handling large file collections >> > >> well seems to be the checksumming. >> > >> But with that turned of, what's keeping puppet busy?
>> > >> Puppet eats up 30 minutes of CPU time, I see two options: >> > >> - Puppet is doing something necessary for it's functioning during that >> > >> time, in which case somebody should know what it is doing. Anybody? >> > >> - Or the code is gravely flawed and needs to be fixed.
>> > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't >> > >> production ready? >> > >> I really can't have this on production machines...
>> > > -- >> > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. >> > > To post to this group, send email to puppet-users@googlegroups.com. >> > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. >> > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.
>> > -- This account not approved for unencrypted proprietary information --
> -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Fri, 2010-03-12 at 03:30 -0800, DieterVDW wrote: > On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: > > Puppet doesn't handle a folder with lots of files well. It handles > large files even worse. The standard advice is "Try putting the files > in a package and distributing them using apt." Another common answer > is to try combining exec and rsync. I ended up using apt. Here are > the tutorials I used:
> The problem is, I -am- using apt! > Those files are downloaded and installed using apt, I just want puppet > to make sure they are owned by a certain user and group. > That's the only thing puppet needs to do.
> If I do the same in bash: > find directory/ | while read file ; do chown user:group "$file" ; done
> real 0m28.119s > user 0m4.064s > sys 0m12.725s
> I can live with overhead from slow ruby, etc etc ... > But apparently Puppet is 60x slower than bash for doing the same > action!
> During my investigation for this problem I've seen a lot of people > saying things like: > "Puppet doesn't handle a folder with lots of files well" > 60x times slower isn't really "not handling well". It's crap. > I'm a bit baffled by the defeatism I see in the Puppet community > concerning Puppet CPU usage.
> I can't really believe people would call what I am experiencing "not > handling well", so I suppose I'm having another, worse, issue > concerning my setup?
> Also the usual suspect for puppet not handling large file collections > well seems to be the checksumming. > But with that turned of, what's keeping puppet busy?
This is a known (at least to me) regression in 0.25: Puppet still uses md5 checksums on every managed files, even when it is not needed.
> Puppet eats up 30 minutes of CPU time, I see two options: > - Puppet is doing something necessary for it's functioning during that > time, in which case somebody should know what it is doing. Anybody?
It is checksumming every file.
> - Or the code is gravely flawed and needs to be fixed.
This is a bug, but I won't say it is gravely flawed, because in the end you have the correct result.
> Is this a stroke of bad luck, or should I conclude that Puppet isn't > production ready?
Puppet is production ready, as many here are running it on critical production systems or on big infrastructure. It's just that like every piece of software it can contain small defects that needs to be reported and fixed.
> I really can't have this on production machines...
I don't see why. Would you throw the complete software and revert managing everything by hand because you encountered a small performance issue for which you can find correct work-arounds, like an exec of a recursive chmod? -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
Am Fri, 12 Mar 2010 03:30:59 -0800 schrieb DieterVDW:
> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: >> Puppet doesn't handle a folder with lots of files well. It handles >> large files even worse. The standard advice is "Try putting the files >> in a package and distributing them using apt." Another common answer >> is to try combining exec and rsync. I ended up using apt. Here are >> the tutorials I used:
> The problem is, I -am- using apt! > Those files are downloaded and installed using apt, I just want puppet > to make sure they are owned by a certain user and group. That's the only > thing puppet needs to do.
if the files are installed by apt and do not have the permissions you need, you may need to have a look at dpkg-statoverride (man dpkg- statoverride).
I must admit I'd really like to like puppet, so I'm a bit agitated by this issue, and won't rest till I know the cause :) . Sorry for the information overload, hope you don't mind.
First of all, I've followed Martin's suggestion and simplified my config. This is my configuration in it's entirety: nodes.pp: node 'puptest.domain.com' { include onlyone
}
onlyone.pp: class onlyone { $pack_list = ["large-package"]
package { $pack_list: ensure => installed }
$username = "someuser" user { "$username": comment => "Some user", home => "/home/$username", managehome => "true", shell => '/bin/bash' }
No effect though, the issue still occurs after all files have been chown'd.
I've also tried attaching gdb to the puppetd process and breaking during high CPU usage. If I do a backtrace then, most of the backtrace contains ?? -> can't derive. If I only leave the methods gdb can determine, this is the stacktrace: st_foreach () from /usr/lib/libruby1.8.so.1.8 st_foreach () from /usr/lib/libruby1.8.so.1.8 st_foreach () from /usr/lib/libruby1.8.so.1.8 st_foreach () from /usr/lib/libruby1.8.so.1.8 st_foreach () from /usr/lib/libruby1.8.so.1.8 st_foreach () from /usr/lib/libruby1.8.so.1.8 st_foreach () from /usr/lib/libruby1.8.so.1.8 ruby_xmalloc () from /usr/lib/libruby1.8.so.1.8 rb_yield_values () from /usr/lib/libruby1.8.so.1.8 rb_ary_each () from /usr/lib/libruby1.8.so.1.8 rb_funcall2 () from /usr/lib/libruby1.8.so.1.8 rb_iterate () from /usr/lib/libruby1.8.so.1.8 rb_block_call () from /usr/lib/libruby1.8.so.1.8
No idea if this is of any use ...
I've also tried using --debug, but no debug statements are printed either during the entire time puppetd is hogging the CPU. Will have a second look though, maybe something interesting is printed earlier...
On Mar 12, 1:58 pm, Martin Wheldon <mwhel...@googlemail.com> wrote:
> Is this directory the only resource you are managing or are there others. > If not then try disabling all other resources and see if you still get > the same issue?
> Best Regards
> Martin Wheldon
> On Fri, Mar 12, 2010 at 12:55 PM, DieterVDW <dieter...@gmail.com> wrote: > > Also, whatever Puppetd is doing then, it seems to be unnecessary.
> > If I execute puppetd -tv and wait until all files have been chown'd, > > and then hit CTRL-C and run puppetd again, > > the second time, puppetd finishes in about a minute. > > The total process of running puppetd twice takes about 4 minutes .
> > So by manipulating the execution of puppetd using a CTRL-C keystroke I > > can reach the intended end state in 4 minutes. > > If I leave puppetd to do it by itself, it takes 30+ minutes of 100% > > CPU usage...
> > On Mar 12, 1:43 pm, DieterVDW <dieter...@gmail.com> wrote: > >> Aren't the time values in seconds? > >> Because I can't match the "Total: 54.78" line to the "notice: Finished > >> catalog run in 1877.06 seconds" .
> >> Also, the strange thing is that the actual changing of owner and group > >> happens pretty fast. > >> I see these messages passing during +- 20s for all files: > >> notice: /File[/some/file]/owner: owner changed 'wronguser' to > >> 'rightuser' > >> notice: /File[/some/file]/group: group changed 'wronggroup' to > >> 'rightgroup'
> >> It's AFTER this has happened that puppetd stalls for 30+ minutes. > >> So it must be some kind of wrapup action... ?
> >> D
> >> On Mar 12, 1:26 pm, Trevor Vaughan <tvaug...@onyxpoint.com> wrote:
> >> > So, it looks like 30 minutes is being taken up in the Package type and > >> > 19 in the File type.
> >> > The Package type probably won't eat up much CPU at all as it's just > >> > fetching and installing packages (unless they are some huge packages).
> >> > You seem to be trying to recursively manage *something* using the File type.
> >> > What happens is that Puppet creates an actual file object for each > >> > file and directory in the recursed path. Then, it takes the graph and > >> > applies the changes to each file that has been seen. I'm not certain, > >> > but I think that it builds this tree each time it runs to ensure that > >> > nothing has changed under the path.
> >> > This processing has to happen on the client since the server has no > >> > previous knowledge of the client's file structure.
> >> > That said, for deep directory/many file file statements, just spawn an > >> > exec until the internal Puppet logic can be optimized.
> >> > Trevor
> >> > On Fri, Mar 12, 2010 at 6:32 AM, DieterVDW <dieter...@gmail.com> wrote: > >> > > Some more information from the (undocumented) --summarize option I > >> > > just discovered:
> >> > > It seems to me the cause of the delays is not recorded in the time > >> > > overview?
> >> > > D
> >> > > On Mar 12, 12:30 pm, DieterVDW <dieter...@gmail.com> wrote: > >> > >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote:
> >> > >> > Puppet doesn't handle a folder with lots of files well. It handles large files even worse. The standard advice is "Try putting the files in a package and distributing them using apt." Another common answer is to try combining exec and rsync. I ended up using apt. Here are the tutorials I used:
> >> > >> The problem is, I -am- using apt! > >> > >> Those files are downloaded and installed using apt, I just want puppet > >> > >> to make sure they are owned by a certain user and group. > >> > >> That's the only thing puppet needs to do.
> >> > >> If I do the same in bash: > >> > >> find directory/ | while read file ; do chown user:group "$file" ; done
> >> > >> real 0m28.119s > >> > >> user 0m4.064s > >> > >> sys 0m12.725s
> >> > >> I can live with overhead from slow ruby, etc etc ... > >> > >> But apparently Puppet is 60x slower than bash for doing the same > >> > >> action!
> >> > >> During my investigation for this problem I've seen a lot of people > >> > >> saying things like: > >> > >> "Puppet doesn't handle a folder with lots of files well" > >> > >> 60x times slower isn't really "not handling well". It's crap. > >> > >> I'm a bit baffled by the defeatism I see in the Puppet community > >> > >> concerning Puppet CPU usage.
> >> > >> I can't really believe people would call what I am experiencing "not > >> > >> handling well", so I suppose I'm having another, worse, issue > >> > >> concerning my setup?
> >> > >> Also the usual suspect for puppet not handling large file collections > >> > >> well seems to be the checksumming. > >> > >> But with that turned of, what's keeping puppet busy?
> >> > >> Puppet eats up 30 minutes of CPU time, I see two options: > >> > >> - Puppet is doing something necessary for it's functioning during that > >> > >> time, in which case somebody should know what it is doing. Anybody? > >> > >> - Or the code is gravely flawed and needs to be fixed.
> >> > >> Is this a stroke of bad luck, or should I conclude that Puppet isn't > >> > >> production ready? > >> > >> I really can't have this on production machines...
> >> > > -- > >> > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > >> > > To post to this group, send email to puppet-users@googlegroups.com. > >> > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > >> > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.
> >> > -- This account not approved for unencrypted proprietary information --
> > -- > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > > For more options, visit this group athttp://groups.google.com/group/puppet-users?hl=en.
On Fri, 2010-03-12 at 05:42 -0800, DieterVDW wrote: > On Mar 12, 2:21 pm, Brice Figureau <brice-pup...@daysofwonder.com> > wrote: > > It is checksumming every file.
> Is it normal no debug information is shown during this process?
Yes it is, because it happens in the "file serving" code as part of getting information about a given file. This part of the process doesn't know getting the checksum is unnecessary (in fact this is the same code that is used to serve file metadata over the wire, except in this case it is locally executed).
This isssue seems to be frequently raised, so I think I'll work on it, if I have some spare time soon. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
<brice-pup...@daysofwonder.com> wrote: > On Fri, 2010-03-12 at 05:42 -0800, DieterVDW wrote: >> On Mar 12, 2:21 pm, Brice Figureau <brice-pup...@daysofwonder.com> >> wrote: >> > It is checksumming every file.
>> Is it normal no debug information is shown during this process?
> Yes it is, because it happens in the "file serving" code as part of > getting information about a given file. This part of the process doesn't > know getting the checksum is unnecessary (in fact this is the same code > that is used to serve file metadata over the wire, except in this case > it is locally executed).
> This isssue seems to be frequently raised, so I think I'll work on it, > if I have some spare time soon. > -- > Brice Figureau > Follow the latest Puppet Community evolutions on www.planetpuppet.org!
> -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Mar 12, 2:48 pm, Brice Figureau <brice-pup...@daysofwonder.com> wrote:
> Yes it is, because it happens in the "file serving" code as part of > getting information about a given file. This part of the process doesn't > know getting the checksum is unnecessary (in fact this is the same code > that is used to serve file metadata over the wire, except in this case > it is locally executed).
And this happens also if the files aren't fileserved? Bug 3170 seems to suggest checksum => undef should work though?
Anyway, it would indeed be great if this were solved. I wish I understood Ruby, would love to peek under the hood of puppet...
Thanks for taking the time to look at this, and making our lives easier :) .
On Fri, 2010-03-12 at 06:18 -0800, DieterVDW wrote: > On Mar 12, 2:48 pm, Brice Figureau <brice-pup...@daysofwonder.com> > wrote: > > Yes it is, because it happens in the "file serving" code as part of > > getting information about a given file. This part of the process doesn't > > know getting the checksum is unnecessary (in fact this is the same code > > that is used to serve file metadata over the wire, except in this case > > it is locally executed).
> And this happens also if the files aren't fileserved?
Yes. The checksum are needed when remotely fileserving (and sometimes locally), thus they are also computed when managing local (not sourced) files, because the same code is run.
> Bug 3170 seems to suggest checksum => undef should work though?
It used to work in 0.24.x, but we lost this in 0.25 when the whole file serving system was refactored (and greatly improved despite what you found).
> Anyway, it would indeed be great if this were solved.
I concur. This is annoying. I have a 5 minutes hack to disable checksumming, but it far from being production ready. I don't think I'll have time this week-end to work on this, but if I can find 5 minutes, I'll try to send a preliminary patch to puppet-dev.
> I wish I understood Ruby, would love to peek under the hood of > puppet...
I learnt ruby by fixing bugs in Puppet. You can do the same :-)
> Thanks for taking the time to look at this, and making our lives > easier :) .
-- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org!
Thomas Mueller <tho...@chaschperli.ch> writes: > Am Fri, 12 Mar 2010 03:30:59 -0800 schrieb DieterVDW: >> On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: >>> Puppet doesn't handle a folder with lots of files well. It handles >>> large files even worse. The standard advice is "Try putting the files >>> in a package and distributing them using apt." Another common answer >>> is to try combining exec and rsync. I ended up using apt. Here are >>> the tutorials I used: >> The problem is, I -am- using apt! >> Those files are downloaded and installed using apt, I just want puppet >> to make sure they are owned by a certain user and group. That's the only >> thing puppet needs to do. > if the files are installed by apt and do not have the permissions you > need, you may need to have a look at dpkg-statoverride (man dpkg- > statoverride).
Under most circumstances, though, all you need to do is ensure that the files have the correct ownership in the *.deb when you create it and then dpkg will do the correct thing when installing the package.
> The problem is, I -am- using apt! > Those files are downloaded and installed using apt, I just want puppet > to make sure they are owned by a certain user and group. > That's the only thing puppet needs to do.
I didn't read the bug report before my last reply. I know this isn't what you want, but how about adding chown to the Makefile used to generate the deb. Also, you could just run chown using Exec.
On Fri, 12 Mar 2010, DieterVDW wrote: > On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: > Those files are downloaded and installed using apt, I just want puppet > to make sure they are owned by a certain user and group. > That's the only thing puppet needs to do.
As a workaround, instead of a recursive file resource like this:
file { "/some/dir": ensure => directory, recurse => inf, owner => "someuser", group => "somegroup", require => Package["the package that created the files"], }
try an exec resorce like this:
exec { "fix permissions in /some/dir": command => "chown -R someuser:somegroup /some/dir", require => Package["the package that created the files"], }
The exec will be much faster, but it will run every time (adding a message in the log), even if the files already have the correct ownership. To get rid of the unwanted log message at the expense of slower execution, add
onlyif => "some command to check whether there's a problem",
The onlyif command could use something involving "find" to print the names of bad files, and "test" to see whether find's output was empty.
On Sat, Mar 13, 2010 at 5:32 AM, Alan Barrett <a...@cequrux.com> wrote: > On Fri, 12 Mar 2010, DieterVDW wrote: > > On Mar 12, 11:21 am, Patrick <kc7...@gmail.com> wrote: > > Those files are downloaded and installed using apt, I just want puppet > > to make sure they are owned by a certain user and group. > > That's the only thing puppet needs to do.
> As a workaround, instead of a recursive file resource like this:
> file { "/some/dir": > ensure => directory, > recurse => inf, > owner => "someuser", > group => "somegroup", > require => Package["the package that created the files"], > }
> try an exec resorce like this:
> exec { "fix permissions in /some/dir": > command => "chown -R someuser:somegroup /some/dir", > require => Package["the package that created the files"], > }
> The exec will be much faster, but it will run every time (adding > a message in the log), even if the files already have the correct > ownership. To get rid of the unwanted log message at the expense of > slower execution, add
> onlyif => "some command to check whether there's a problem",
> The onlyif command could use something involving "find" to print the > names of bad files, and "test" to see whether find's output was empty.
> --apb (Alan Barrett)
> -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@google groups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.
