augeas slooow
Jennings, Jared L CTR USAF AFMC 46 SK/CCI
entire set of manifests, what-have-you) and 11 custom lenses, each for
only one or two files, and it takes like 2 1/2 minutes to apply the
whole thing. Looking at the --debug output, there are a whole lot of
pauses right after "opening Augeas" sorts of lines. (That happens 79
times.) I think that's because Augeas is populating its whole tree, each
time.
I know I can avoid that behavior by specifying the lens and incl
parameters in my augeas resources, but I like the clarity I get by not
doing that, and want to preserve it.
Has anyone ever tried to make it so that only one Augeas connection is
opened per Puppet run?
How can I profile my Puppet catalog, to see what resources take the
longest to set straight? (I've only got puppet apply for now - no master
yet)
saurabh verma
>
> How can I profile my Puppet catalog, to see what resources take the
> longest to set straight? (I've only got puppet apply for now - no master
> yet)
>
+1 for profiling feature
Jennings, Jared L CTR USAF AFMC 46 SK/CCI
> -----Original Message-----
> From: puppet...@googlegroups.com [mailto:puppet-
> us...@googlegroups.com] On Behalf Of Nigel Kersten
> Sent: Tuesday, March 22, 2011 4:40 PM
> To: puppet...@googlegroups.com
> Cc: saurabh verma
> Subject: Re: [Puppet Users] augeas slooow
>
> On Tue, Mar 22, 2011 at 1:24 PM, saurabh verma
<nitw.s...@gmail.com>
> More than --evaltrace ?
>
>
>
>
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> > To post to this group, send email to puppet...@googlegroups.com.
> > To unsubscribe from this group, send email to puppet-
> users+un...@googlegroups.com.
> > For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
> >
> >
>
>
>
> --
> Nigel Kersten
> Product, Puppet Labs
> @nigelkersten
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-
> users+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
Nigel Kersten
More than --evaltrace ?
>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
Dominic Cleal
> I've got somewhere around 130 Augeas resources in my policy (catalog,
> entire set of manifests, what-have-you) and 11 custom lenses, each for
> only one or two files, and it takes like 2 1/2 minutes to apply the
> whole thing. Looking at the --debug output, there are a whole lot of
> pauses right after "opening Augeas" sorts of lines. (That happens 79
> times.) I think that's because Augeas is populating its whole tree, each
> time.
>
> I know I can avoid that behavior by specifying the lens and incl
> parameters in my augeas resources, but I like the clarity I get by not
> doing that, and want to preserve it.
>
> Has anyone ever tried to make it so that only one Augeas connection is
> opened per Puppet run?
I saw your suggestion here and filed it a little while back:
https://projects.puppetlabs.com/issues/6920
More recently at Puppet Camp I realised there's a slightly better way of
optimising the loading, which I've tried to implement in this issue:
https://projects.puppetlabs.com/issues/7285
If you get the chance to test it, I'd appreciate any feedback via the
issue report. I suspect the method's a bit fragile, so testing's needed.
There's also a change that Mike Knox has been working on, which will
halve the time taken by Puppet when an Augeas resource has to be
applied. It's under the guise of showing file diffs for Augeas actions,
but happily speeds it up too:
https://projects.puppetlabs.com/issues/2728
Hopefully this focus on Puppet+Augeas performance will help out your
catalog runs in the future!
Cheers,
--
Dominic Cleal
Red Hat Consulting
m: +44 (0)7818 512168
Aaron Lippold
From a previous email of let...@repeat.com.
" ...
To avoid that, I added 'lens' and 'incl' parameters to the puppet type a while ago. If you set these, Augeas will only read a specific file, avoiding most of the unnecessary overhead.
So, you should say something like
augeas {
add_a_host:
changes => "...",
lens => "Hosts.lns",
incl => "/etc/hosts"
}
in your manifests, and everything will be lightning fast, even without connection caching."
Jennings, Jared L CTR USAF AFMC 46 SK/CCI
Aaron (quoting letter@):
> To avoid that, I added 'lens' and 'incl' parameters to the puppet type
> a while ago. If you set these, Augeas will only read a specific file,
> avoiding most of the unnecessary overhead.
> ... everything will be lightning fast, even without
> connection caching."
Me:
> I know I can avoid [Augeas populating its whole tree] by specifying
> the lens and incl parameters in my augeas resources, but I like the
> clarity I get by not doing that, and want to preserve it.
The reason I brought the issue up is that I wanted to know why Augeas
connections are not cached. I think they should be, but someone on this
list may have tried to make that change, failed, and gained wisdom.
> lens => "Hosts.lns",
It's nice to find that I may not have to give the full path of the lens.
Patrick
On May 5, 2011, at 6:28 AM, Jennings, Jared L CTR USAF AFMC 46 SK/CCI wrote:
> Hi, Aaron.
>
> Aaron (quoting letter@):
>> To avoid that, I added 'lens' and 'incl' parameters to the puppet type
>> a while ago. If you set these, Augeas will only read a specific file,
>> avoiding most of the unnecessary overhead.
>> ... everything will be lightning fast, even without
>> connection caching."
>
> Me:
>> I know I can avoid [Augeas populating its whole tree] by specifying
>> the lens and incl parameters in my augeas resources, but I like the
>> clarity I get by not doing that, and want to preserve it.
>
> The reason I brought the issue up is that I wanted to know why Augeas
> connections are not cached. I think they should be, but someone on this
> list may have tried to make that change, failed, and gained wisdom.
I can tell you one reason why. One of the most expensive parts of starting Augeas is scanning the whole filesystem (or at least all auto-included files, of which there are a lot) and making that into a tree. It would need to rescan every file before processing each resource. I think modern versions do this by checking the mtime of every file, which is expensive, though better than parsing all from scratch.
Still, I would make a random guess that caching the connections might actually be slower than using noautoinclude. Personally, as soon as all our machines are 2.6.x, I'm planning to try wrapping augeas in a define that will make "context" the the file path be the same value. (Technically I need to add "/files" at the beginning too.)
Dominic Cleal
>
> On May 5, 2011, at 6:28 AM, Jennings, Jared L CTR USAF AFMC 46 SK/CCI wrote:
>
>> Hi, Aaron.
>>
>> Aaron (quoting letter@):
>>> To avoid that, I added 'lens' and 'incl' parameters to the puppet type
>>> a while ago. If you set these, Augeas will only read a specific file,
>>> avoiding most of the unnecessary overhead.
>>> ... everything will be lightning fast, even without
>>> connection caching."
>>
>> Me:
>>> I know I can avoid [Augeas populating its whole tree] by specifying
>>> the lens and incl parameters in my augeas resources, but I like the
>>> clarity I get by not doing that, and want to preserve it.
>>
>> The reason I brought the issue up is that I wanted to know why Augeas
>> connections are not cached. I think they should be, but someone on this
>> list may have tried to make that change, failed, and gained wisdom.
I've yet to try, but it's a tricky request as it's mean having ways to
ensure the tree is in a sane state each time.
> I can tell you one reason why. One of the most expensive parts of starting Augeas is scanning the whole filesystem (or at least all auto-included files, of which there are a lot) and making that into a tree. It would need to rescan every file before processing each resource. I think modern versions do this by checking the mtime of every file, which is expensive, though better than parsing all from scratch.
>
> Still, I would make a random guess that caching the connections might actually be slower than using noautoinclude.
Indeed, I came to the same conclusion. The way I've proposed doing this
(as implemented in the patch for #7285) is to use noautoload, then strip
out those files that don't match the context - if it's supplied. You
still need the autoinclude as you need to know both the lens and the
file, you can't just specify the file.