Checksum Behavior Change

[Paul Lathrop]

Hiya,

Since upgrading our puppetmaster to 0.25.1, we've noticed a behavior
change which is causing us some problems, and I'm not sure if it is
expected behavior. We have the following resource definition (inside
of a define):

file {
"/var/lib/mysql/extdbfiles/$name":
owner => mysql,
group => mysql,
mode => 2660,
recurse => true,
ensure => directory,
backup => false,
require => File["/var/lib/mysql"];
}

What we're noticing is that our puppet runs are now taking a very long
time and consuming most of the I/O and CPU resources on the machine.
Running puppetd -t under strace reveals that puppet reads the contents
of all the files under this directory (even though we aren't managing
file contents). My immediate thought was that adding checksum => mtime
to the resource definition would solve the problem, but doing so does
not seem to change the behavior.

It smells like a bug to me; intuitively if we aren't managing the file
contents in any way, puppet should not bother calculating checksums of
the contents. Even if that isn't true, specifying checksum => mtime
seems to be the right approach to suppressing calculation of checksums
of the file contents.

For the time being we're working around this by using an exec resource
and chmod, but this is non-ideal. Is this a bug? If so can someone
help me understand this subsection of the puppet internals so I can
contribute a patch?

--Paul

[Markus Roberts]

Paul --

I share your intuition--it smells like a bug to me too. 

Specifically, it smells like #2630 (http://projects.reductivelabs.com/issues/2630) though it may be unrelated.

We deferred that as just an efficiency issue (in that the results are correct), but it still needs to be addressed.

-- Markus

[Paul Lathrop]

Well, the issue may be related, but it isn't the same - I don't think
these files should be getting checksummed *at all* (even once would be
too much). I'll definitely put my vote in for #2630 though.

--Paul

[Luke Kanies]

Hmm. This definitely constitutes a new bug and I would consider it
high priority, and it should be an easy fix (sorry to disagree with
Markus).

Not sure where exactly you should go to look in the internals - if you
look in the 'content' type in File, you should be able to add some
debugging there to see where it's getting created. My guess is that
recursion process is overzealouslessly adding all parameters to check,
rather than just those that the parent resource checks, so you could
also check the 'recurse' method in the file type.

--
Wear the old coat and buy the new book. -- Austin Phelps
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[Markus Roberts]

My reasoning for saying they may be related (I'm not saying that it's correct) was that both could be characterized by "all files being checksumed one time more than necessary--twice, when it should have been once, and once, when it should have been never.

Also, I wasn't saying it was low priority just that (if they are in fact the same) we had not considered this case in deprioritizing it.

Paul, if you want me to help track this down, just let me know.

-- Markus

[Brice Figureau]

Hi Paul,

On 14/11/09 00:29, Paul Lathrop wrote:
>
> Hiya,
>
> Since upgrading our puppetmaster to 0.25.1, we've noticed a behavior
> change which is causing us some problems, and I'm not sure if it is
> expected behavior. We have the following resource definition (inside
> of a define):
>
> file {
> "/var/lib/mysql/extdbfiles/$name":
> owner => mysql,
> group => mysql,
> mode => 2660,
> recurse => true,
> ensure => directory,
> backup => false,
> require => File["/var/lib/mysql"];
> }
>
> What we're noticing is that our puppet runs are now taking a very long
> time and consuming most of the I/O and CPU resources on the machine.
> Running puppetd -t under strace reveals that puppet reads the contents
> of all the files under this directory (even though we aren't managing
> file contents). My immediate thought was that adding checksum => mtime
> to the resource definition would solve the problem, but doing so does
> not seem to change the behavior.

In 0.24, I was using checksum => undef to be sure we aren't checksumming
everything.
That was one of the reason I added #1469:
http://projects.reductivelabs.com/issues/1469

And also check #1973:
http://projects.reductivelabs.com/issues/1973

It doesn't look like a new issue to me.

> It smells like a bug to me; intuitively if we aren't managing the file
> contents in any way, puppet should not bother calculating checksums of
> the contents. Even if that isn't true, specifying checksum => mtime
> seems to be the right approach to suppressing calculation of checksums
> of the file contents.

Correct. Maybe you have a global default ala:
File { checksum => md5 }

> For the time being we're working around this by using an exec resource
> and chmod, but this is non-ideal. Is this a bug? If so can someone
> help me understand this subsection of the puppet internals so I can
> contribute a patch?

Recursion is managed by spawning new sub-file resources inheriting
parent properties/parameter. Maybe we're generating those with an
invalid checksum property?

I think you should add some debug print in lib/puppet/type/file.rb in
the newchild method.
You can also first reproduce a small testcase with the "puppet", once
done it will be easier to debug/understand.

Hope that helps,
--
Brice Figureau
My Blog: http://www.masterzen.fr/

[Luke Kanies]

On Nov 13, 2009, at 6:57 PM, Markus Roberts wrote:

> My reasoning for saying they may be related (I'm not saying that
> it's correct) was that both could be characterized by "all files
> being checksumed one time more than necessary--twice, when it should
> have been once, and once, when it should have been never.

I see. The flaw with that argument is that there shouldn't even be an
instance of the 'content' parameter in files that aren't checking
their content or checksum, while for those files that are, the content
param should be there. So it seems to be pretty different.

> Also, I wasn't saying it was low priority just that (if they are in
> fact the same) we had not considered this case in deprioritizing it.
>
> Paul, if you want me to help track this down, just let me know.

--
The leader of Jamestown was "John Smith" (not his real name), under
whose direction the colony engaged in a number of activities,
primarily related to starving. -- Dave Barry, "Dave Barry Slept Here"