Puppet agent run runs slowly

1,324 views
Skip to first unread message

Constantin Wolber

unread,
Aug 25, 2015, 7:49:28 AM8/25/15
to Puppet Users
Hi,

i searched quite a bit with google but cannot really find an idea of what to change.

If i do a puppet agent run on a few of my managed servers i get the following results:

Notice: Finished catalog run in 486.54 seconds



Changes:

Events:

Resources:

            Total: 2304

Time:

       Filebucket: 0.00

          Apt key: 0.00

           Anchor: 0.00

         Schedule: 0.00

            Group: 0.00

             Exec: 0.00

             User: 0.00

   Ssh authorized key: 0.00

          Package: 0.39

         Last run: 1440488641

          Service: 2.71

   Config retrieval: 2.91

             File: 22.60

            Total: 28.62

Version:

           Config: 1440488048

           Puppet: 3.8.1



The summarized view looks good but it seems puppet spends a lot of time in different other places that don't count for the summarize.


1. Debug: Loaded state in 119.18 seconds

2. Debug: Loaded state in 156.15 seconds

3. Debug: Stored state in 252.81 seconds


What is causing those excessive times for Loading and Storing the state. I did not really find a lot of hints on that topic. 


Due to some ideas i already changed most of my recurse options for file type to recurse => "remote"


I also found the hint to use checksum => "none" but not sure about the effect of that change. 


Any hints are welcome


Regards


Constantin

kaustubh chaudhari

unread,
Aug 25, 2015, 4:14:21 PM8/25/15
to Puppet Users

Hi,

You can run profiler to check who and what is taking time in detail.

https://puppetlabs.com/blog/tune-puppet-performance-profiler

-Kaustubh

Constantin Wolber

unread,
Aug 26, 2015, 2:57:08 PM8/26/15
to puppet...@googlegroups.com
Hi,

i tried the performance profiler, but it did not really help me figuring things out. 

2015-08-26 14:48:21 +0000 Puppet (debug): Storing state

2015-08-26 14:55:23 +0000 Puppet (debug): Stored state in 422.03 seconds


What I'm interested in is how i can improve performance of that step. 


--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/BzfsN9axWss/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/05bacd31-8a6c-4ab5-bdd3-a323f12ff249%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Constantin Wolber

unread,
Aug 26, 2015, 3:13:26 PM8/26/15
to puppet...@googlegroups.com
Hi,

after some more searching and investigating i came to the solution that the state.yaml file is the problem. The affected system contained a state.yaml file which was 147M big. So i deleted the file and after the next run it was down to 347K and no performance issues occurred.

Any idea what could have caused the file to get that big?

Regards

Constantin

Christopher Wood

unread,
Aug 26, 2015, 5:26:31 PM8/26/15
to puppet...@googlegroups.com
On Wed, Aug 26, 2015 at 05:13:19PM +0200, Constantin Wolber wrote:
> Hi,
> after some more searching and investigating i came to the solution that
> the state.yaml file is the problem. The affected system contained a
> state.yaml file which was 147M big. So i deleted the file and after the
> next run it was down to 347K and no performance issues occurred.
> Any idea what could have caused the file to get that big?
> Regards
> Constantin

I recall a thread way back where somebody was purging a growing directory with puppet, I wonder if this is the same issue where the list of resources to be deleted are appended to state.yaml ad infinitum. Maybe save state.yaml backups and diff them?

Similar:

https://ask.puppetlabs.com/question/6544/state-file-constantly-growing/

https://groups.google.com/forum/#!topic/puppet-users/Laj0WFBmdsU

> 2015-08-26 16:56 GMT+02:00 Constantin Wolber
> <[1]constant...@gmail.com>:
>
> Hi,
> i tried the performance profiler, but it did not really help me figuring
> things out. 
>
> 2015-08-26 14:48:21 +0000 Puppet (debug): Storing state
>
> 2015-08-26 14:55:23 +0000 Puppet (debug): Stored state in 422.03 seconds
>
> What I'm interested in is how i can improve performance of that step. 
>
> 2015-08-25 18:14 GMT+02:00 kaustubh chaudhari <[2]kaus...@gmail.com>:
>
> Hi,
>
> You can run profiler to check who and what is taking time in detail.
>
> [3]https://puppetlabs.com/blog/tune-puppet-performance-profiler
> [4]https://groups.google.com/d/topic/puppet-users/BzfsN9axWss/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [5]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [6]https://groups.google.com/d/msgid/puppet-users/05bacd31-8a6c-4ab5-bdd3-a323f12ff249%40googlegroups.com.
> For more options, visit [7]https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [8]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [9]https://groups.google.com/d/msgid/puppet-users/CAFYu9xxbT3egioN0crCeBBuXX%2BiFeDhDSiznWXJy1MTX-wmOmg%40mail.gmail.com.
> For more options, visit [10]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:constant...@gmail.com
> 2. mailto:kaus...@gmail.com
> 3. https://puppetlabs.com/blog/tune-puppet-performance-profiler
> 4. https://groups.google.com/d/topic/puppet-users/BzfsN9axWss/unsubscribe
> 5. mailto:puppet-users...@googlegroups.com
> 6. https://groups.google.com/d/msgid/puppet-users/05bacd31-8a6c-4ab5-bdd3-a323f12ff249%40googlegroups.com?utm_medium=email&utm_source=footer
> 7. https://groups.google.com/d/optout
> 8. mailto:puppet-users...@googlegroups.com
> 9. https://groups.google.com/d/msgid/puppet-users/CAFYu9xxbT3egioN0crCeBBuXX%2BiFeDhDSiznWXJy1MTX-wmOmg%40mail.gmail.com?utm_medium=email&utm_source=footer
> 10. https://groups.google.com/d/optout

Christian Flamm

unread,
Aug 27, 2015, 9:07:37 AM8/27/15
to Puppet Users, christop...@pobox.com
I most sincerely love you!
Thank you, thank you, thank you!!!
Reply all
Reply to author
Forward
0 new messages