Help -- puppet master cache not in fact caching

[Victor Danilchenko]

Hi all,

We have a puppetmaster installation (CentOS, Puppet 3.4.3)  in production, and as the scope and complexity of our installation grows, the manifest processing has been getting slower and slower. I looked into it today, and realized that caching wasn't working. So I put some config options into /etc/puppet/puppet.conf to enable caching (I stuck them into both [main] and [master] blocks, just to be safe, I wasn't sure which one was the right place to put them):

use_cached_catalog = true

environment_timeout = 1d

However, it's still not helping. Puppet manifests take 20s to 200s to compile, the system is slow as molasses. The compiled catalog is being stored:

$ ls -l yaml/node/MYNODE.yaml

-rw-rwxr-x 1 puppet puppet 9813 Oct  7 16:20 yaml/node/MYNODE.yaml

However, note that it's tiny. The problem seems to be that the classes array is empty; i.e. puppetmaster seems to be caching facts and parameters, but not any actual manifest content. When I run a compile by hand, it comes out much larger, and has all the stuff in it:

$ sudo puppet master --compile MYNODE > manualCatalog.yaml

$ # ls -lh manualCatalog.yaml 

-rw-r-xr-x 1 root root 1.3M Oct  7 16:19 manualCatalog.yaml

I tried replacing the empty compiled catalog (small one) with the manuaIly compiled catalog (big one), but it gets ignored and replaced with a small one upon the next run.

I have a feeling that I am missing something extremely simple and obvious, but I can't figure out what it is, and the puppet master logs are of no help -- all they keep saying is that the catalog was compiled:

Tue Oct 07 16:21:01 -0400 2014 Puppet (notice): Compiled catalog for MYNODE in environment development in 18.25 seconds

Any help, suggestions, pointers, anyone?

Thanks in advance!

[Felix Frank]

Hi,

On 10/07/2014 11:52 PM, Victor Danilchenko wrote:
> Hi all,
>
> We have a puppetmaster installation (CentOS, Puppet 3.4.3) in
> production, and as the scope and complexity of our installation grows,
> the manifest processing has been getting slower and slower. I looked
> into it today, and realized that caching wasn't working. So I put some
> config options into /etc/puppet/puppet.conf to enable caching (I stuck
> them into both [main] and [master] blocks, just to be safe, I wasn't
> sure which one was the right place to put them):
>
> use_cached_catalog = true

I believe this is an agent setting, although the documentation is
somewhat ambiguous. But since the agent definitely caches catalogs, it
would make more sense for it to use this option.

> environment_timeout = 1d

This is more appropriate, but the setting is new in 3.6.0. Your master
version does not use it yet.

> However, it's still not helping. Puppet manifests take 20s to 200s to
> compile, the system is slow as molasses. The compiled catalog is being
> stored:
>
> $ ls -l yaml/node/MYNODE.yaml

> -rw-rwxr-x 1 puppet puppet *9813 *Oct 7 16:20 yaml/node/MYNODE.yaml

>
>
> However, note that it's tiny. The problem seems to be that the classes

> array is empty; i.e. *puppetmaster seems to be caching facts and
> parameters, but not any actual manifest content*. When I run a compile

This is a misconception: Puppet has the node data cached, not the
catalog. Actually...

> by hand, it comes out much larger, and has all the stuff in it:
>
> $ sudo puppet master --compile MYNODE > manualCatalog.yaml
> $ # ls -lh manualCatalog.yaml

> -rw-r-xr-x 1 root root *1.3M* Oct 7 16:19 manualCatalog.yaml

...this would not even be possible if you had no stored node data for
the compiler to consume.

> I tried replacing the empty compiled catalog (small one) with the
> manuaIly compiled catalog (big one), but it gets ignored and replaced
> with a small one upon the next run.
>
> I have a feeling that I am missing something extremely simple and
> obvious, but I can't figure out what it is, and the puppet master logs
> are of no help -- all they keep saying is that the catalog was compiled:
>
> Tue Oct 07 16:21:01 -0400 2014 Puppet (notice): Compiled catalog for
> MYNODE in environment development in 18.25 seconds
>
>
> Any help, suggestions, pointers, anyone?

You just opened a new thread conerning migration to a newer version and
directory environments. This is the right call. Caching should be
feasible then.

Cheers,
Felix