Re: performance of source vs template vs file

[jcbollinger]

On Tuesday, October 9, 2012 9:45:42 AM UTC-5, Mihai Stan wrote:

Hello everyone,

I'm currently looking into using puppet to manage my homedir .dot files, including .vim/ folder with a bunch of plugins totaling a few tens of files (almost 100), in order to have a pleasant and productive working experience on our organization's 500+ machines.

While this looks like a cool thing to do, I can't help but wonder how will this affect puppet's performance both on the masters' and agents' sides and how good puppet is at distributing lots of infrequently updated files.

My common sense tells me that from a performance point of view, I could use the following methods, first in the list being the most recommended:

file {
content => file('file'),
}

file {

  source => "puppet:///modules/...",

}

file {

  content => template('template.erb'),

}

Personally, I'd go with using file() function for distributing static files and source => "puppet:///" to manage directories recursively, which would simplify the setup part a lot.

Does anyone have a more in-depth insight into the performance of all these methods ? Or, for that matter, why I shouldn't be doing the above as it blows puppet performance in ways I can't comprehend ?

I would expect a bigger difference between using 'content' and using 'source' than between specifying the content via file() and specifying it via template().  When you use 'content', the body of the target file is included in the catalog, whereas when you use 'source', the agent will request file content from the Puppet file server if it needs it.

For ~100 small files (<= 1 KB on average, say), my gut tells me that 'content' will win at least when there are more than a couple of changes, and that it will not lose too badly when there are few or none.

Certainly you should not use template() for completely static files, but when you do need it, you need it.

Managing directories recursively is a potential problem, but it really comes down more to the total number and combined size of the files you are managing.  Each file and directory adds memory and processing cost on both ends, and, separately, Puppet expends computing effort proportional to the total content size to compute hashes of file contents.

My expectation would be that for dotfiles, you will see acceptable performance any way around.  Nevertheless, as with any performance question, the only way to get a reliable answer is to test it.


John

[Eric Sorenson]

I think all that is right John--one subtlety that led me to use content=>template everywhere instead of mixing it with content=>file for things which didn't need erb interpolation is that the argument to template() uses module-like syntax and file() needs a full filesystem path, so it doesn't work very well with environments that have different root paths.

(this is https://projects.puppetlabs.com/issues/5158)

-=Eric