testing 0.25.x (pre 0.25.5) memory leaks with augeas
Peter Meier
Hash: SHA1
Hi
so for #2929 I installed HEAD of 0.25.x and applied Brice patch on top
of it. Brice already wrote what we figured out while playing with it.
However, something else suddenly popped up to my eyes: the new puppet
version uses a lot more memory by applying my normal manifests than
before: actually up to 5 times more! So, well, a hell lot more.
So I compared the two versions: 0.25.x and 0.25.4 (from epel) and run on
an unused host for each version once 'puppetd -t'. By monitoring memory
usage of the puppet process each 2 seconds I could measure that 0.25.x
took up to 1028408 of VmSize while 0.25.4 used up to 222248.
The issue seems to be related #3379, hence due to #2327 because memory
starts right to grow when puppet arrives with the leaking version at:
debug: Augeas[grub_first_default](provider=augeas): Opening augeas with
root /, lens path , flags 0
debug: Augeas[grub_first_default](provider=augeas): Augeas version 0.7.0
is installed
debug: Augeas[grub_first_default](provider=augeas): Will attempt to save
and only run if files changed
debug: Augeas[grub_first_default](provider=augeas): sending command
'set' with params ["/files/etc/grub.conf/default", "0"]
<----- here it hangs and grows.
any ideas what that might cause? It looks really that maybe even #3379
is closely related to that issue. So either my augeas receipe is bad,
but puppet should never leak memory due to that or we have a bug which
leaks memory.
I could also observe that memory grows enormous by applying just that
manifest with the puppet standalone bin.
I didn't yet add that to the bug report, because I later connected the
issue with the memory leak and yeah maybe somebody has a better
idea/explanation than "It's leaking!!!!1111"
I have pushed my the branch from which I built the package up to
http://github.com/duritong/puppet/tree/0.25.x
I could provide the rpm as well, if somebody is interested.
If anybody has any further ideas, please let me know.
cheers pete
[1] http://projects.reductivelabs.com/issues/2327
[2] http://projects.reductivelabs.com/issues/3379
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkugPkMACgkQbwltcAfKi3/QWwCgkplXugnD2UXKqQ6WIgh2MGpU
qWgAoIG0r6IqHTD4YlOoieoGVEKNd7Vw
=90r8
-----END PGP SIGNATURE-----
Markus Roberts
That is very interesting.
I can't see any way at all the changes in #2327 could cause a memory leak.
Are you saying that it leaks with those changes, and doesn't without them? That would be very odd but, if true, very interesting.
-- Markus
Peter Meier
me neither.
> Are you saying that it leaks with those changes, and doesn't without them?
> That would be very odd but, if true, very interesting.
Not directly. I'm saying that puppet leaks with all the patches
applied since 0.25.4 and that it leaks either in catalog or in
standalone mode while applying the augeas recipe. It starts to leak
before the point, where the augeas type fails due to the filed bug.
And it leaks for quite a while.
I could observe that it even leaks in standalone mode up to 800M of
memory just for that litlle augeas recipe.
And it does not leak with 0.25.4.
This is why I'm suspecting these augeas changes to be the cause of the
problem. But I'm only suspecting them.
cheers pete
Brice Figureau
Does it leak if you remove all the augeas parts from your manifest?
--
Brice Figureau
Follow the latest Puppet Community evolutions on www.planetpuppet.org!
Peter Meier
>> memory just for that litlle augeas recipe.
>>
>> And it does not leak with 0.25.4.
>>
>> This is why I'm suspecting these augeas changes to be the cause of the
>> problem. But I'm only suspecting them.
>
> Does it leak if you remove all the augeas parts from your manifest?
If I run a manifest with the standalone cmd containing some package
and file resources I cannot observe it leaking. However, I could
observe it leaking up to 800M by applying _only_ the augeas resource
mentioned in the bug with the standalone cmd. Therefore I suspect the
augeas changes.
cheers pete
Peter Meier
so I applied the patch bryan provided for #3379 [1] and it fixes the
problem with the augeas resource and it _also_ fixes the encountered
memory problem.
as I mentioned:
> debug: Augeas[grub_first_default](provider=augeas): Opening augeas
> with root /, lens path , flags 0 debug:
> Augeas[grub_first_default](provider=augeas): Augeas version 0.7.0 is
> installed debug: Augeas[grub_first_default](provider=augeas): Will
> attempt to save and only run if files changed debug:
> Augeas[grub_first_default](provider=augeas): sending command 'set'
> with params ["/files/etc/grub.conf/default", "0"] <----- here it
> hangs and grows.
if you compare that to the bugreport it hung and grew right before
closing the conenction to augeas:
> debug: Augeas[grub_first_default](provider=augeas): sending command
> 'set' with params ["/files/etc/grub.conf/default", "0"]
> debug: Augeas[grub_first_default](provider=augeas): Closed the
> augeas connection
> /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:51:in `fail'
> [...]
I didn't yet look any deeper in the code, but something allocated a
lot of memory in communication with augeas. :/
Anybody any ideas on that? Or do we want to assume that either there
have been a fault on my side or that magic can happen?
cheers pete
bk
<SNIP>
> Anybody any ideas on that? Or do we want to assume that either there
> have been a fault on my side or that magic can happen?
>
Just asking the stupid question, the memory is not because of loading
the library the first time is it? Barring that, i am willing to go
with Magic!
-- bk
James Turnbull
Hash: SHA1
And did you see:
http://projects.reductivelabs.com/issues/3387
:)
Regards
James
- --
Author of:
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet (http://tinyurl.com/pupbook)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQEVAwUBS6InayFa/lDkFHAyAQKBkAgAjWs9n8qbUVJv+F/+pivUTJnWeqwRJmT1
plDJAcr25CV1kf9E3nGZwRgSY2VJAKB40m6uCGAEalu9+9koVm4NGtuBJ0LCrGpm
hekvSNA+OuawmNqDV1IEkCuhgspca0gcMQtywJd6MDYjNodEq5cAnVWb6VbYzUm+
fTQtWE78IcFKyXYLwbgyP/dLGiRP4Fcj6pzPiY4y2md+7vKxeelgETfjnGvbHPNB
zEHcnkyc3GI1ZJ+6bIn+jX8DBugZ2vXZ5Nc/p9pL+xmgVRtrg5As0xTi2x8hBfIt
zPlhqi+0HtpSGZtAKYbf55e+rQu0dki5bBl4JpY25ndhqqSmMrKBbA==
=ymxt
-----END PGP SIGNATURE-----
Peter Meier
>> Anybody any ideas on that? Or do we want to assume that either there
>> have been a fault on my side or that magic can happen?
>>
>
> Just asking the stupid question, the memory is not because of loading
> the library the first time is it? Barring that, i am willing to go
> with Magic!
800MB for the single resource I put in the bug report? I hope not.
And as I mentioned it started to leak _after_ augeas had done its
changes and puppet should close the connection to augeas.
cheers pete
Bryan Kearney
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 18/03/10 11:38 PM, bk wrote:
>> On Mar 18, 6:24 am, Peter Meier<peter.me...@immerda.ch> wrote:
>> <SNIP>
>>> Anybody any ideas on that? Or do we want to assume that either there
>>> have been a fault on my side or that magic can happen?
>>>
>>
>> Just asking the stupid question, the memory is not because of loading
>> the library the first time is it? Barring that, i am willing to go
>> with Magic!
>>
>
> And did you see:
>
> http://projects.reductivelabs.com/issues/3387
>
> :)
>
> Regards
>
> James
>
Yeah.. saw that one. Thanks!
-- bk
Bryan Kearney
>> And did you see:
>>
>> http://projects.reductivelabs.com/issues/3387
>>
>> :)
>>
>> Regards
>>
>> James
>>
>
will submit a pull request via the normal means.
-- bk