Puppet 2.6.8 agent slow execution

[pachanga]

Hi!

I'm incrementally developing puppet rules on the master server and
applying them to the testing machine. For some reason puppet agent
applies changes from the master very slowly, about 3 minutes every
time. At this speed it's very tiresome to test the new rules...

I'm running puppet agent as follows:

#puppet agent --no-daemonize --verbose --onetime --summarize --debug --
trace

Here is what it shows in the end:

Changes:
Events:
Resources:
Total: 46
Time:
Filebucket: 0.00
Exec: 0.00
Schedule: 0.00
Ssh authorized key: 0.00
Group: 0.00
Package: 0.00
User: 0.01
Service: 0.07
Config retrieval: 12.25
Last run: 1306935632
File: 84.25
Total: 96.58

During the run most of the time is spent in the following phases:

....
info: Applying configuration version '1306935546'
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: Service[sshd](provider=redhat): Executing '/sbin/service sshd
status'
debug: Puppet::Type::Service::ProviderRedhat: Executing '/sbin/
chkconfig sshd'
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw
yaml; using pson
debug: Finishing transaction 23816162721700

Any ideas why this can be happening? Thanks.

[Patrick]

This probably means you are doing one of these (all of which are slow):

1) Using "File" for big file(s)

2) Using "File" with "recurse=true" to deploy lots of little files.

3) Using "File" with recurse=true" to deploy into a directory with lots of little files, even though you are only deploying very few.  ( purge=fasle )

4) Using "File" over a connection with high ping.

Of course, combining more than one is even worse.

Mitigations for 1 or 2:

1) Put your files in a package and let your package manager of choice handle it for you.

2) Use rsync with a cron job or rsync with an exec

Mitigation for 3:

1) Try "recurse=remote".

Mitigation for 4:

1) If the combined size of the files are small, you can try using "content" instead of "source" for the files.

[Pavel Shevaev]

> Mitigation for 4:
> 1) If the combined size of the files are small, you can try using "content"
> instead of "source" for the files.

Wow, that did the trick.... Thanks a lot! Deployment time is now 10-12
seconds. I'm a puppet newbie and don't quite understand why it worked
but it's great :)

--
Best regards, Pavel

[Jacob Helwig]

So, the big difference there is that by using 'content', the file
contents are shipped down to the agent as part of the catalog.

With 'source', the agent will have to make a new HTTP request per file
to get the content when it attempts to manage the resource.

You end up saving a potentially large number of HTTP round-trips, which
adds up.

--
Jacob Helwig

[Pavel Shevaev]

> You end up saving a potentially large number of HTTP round-trips, which
> adds up.

Thanks for explanations

--
Best regards, Pavel

[Patrick]

So, to expand on this, the reason why the files need to be "small" is because all the files's contents get sent every run, instead of the normal puppet module that just sends the checksum and asks for the file if the checksum doesn't match. (As a trivial example, the file's checksum won't match if the file is empty or doesn't exist.)