1.8.7 => 1.9.3 migration, compilation times raised twice!?

[asq]

i've made a migration on our puppetmasters yesterday, hoping it will run faster and/or make less load on machines with newer ruby. but actual result is just opposite, our compilation times went up a lot, ie:

Nov 27 14:30:12 punch puppet-master[15293]: Compiled catalog for lb1 in environment production in 183.42 seconds
Nov 27 15:38:42 punch puppet-master[24399]: Compiled catalog for lb1 in environment production in 158.49 seconds
Nov 27 15:51:19 punch puppet-master[11963]: Compiled catalog for lb1 in environment production in 175.00 seconds
Nov 27 16:04:15 punch puppet-master[15264]: Compiled catalog for lb1 in environment production in 120.36 seconds
Nov 27 19:25:52 punch puppet-master[552]: Compiled catalog for lb1 in environment production in 281.54 seconds
Nov 27 20:20:19 punch puppet-master[2273]: Compiled catalog for lb1 in environment production in 239.05 seconds
Nov 27 22:26:03 punch puppet-master[4919]: Compiled catalog for lb1 in environment production in 275.38 seconds
Nov 28 00:55:24 punch puppet-master[12293]: Compiled catalog for lb1 in environment production in 267.91 seconds

guess when migration has occured. this machine (lb1) has ~2000 resources.

i'm a bit worried about scalability... we can go back to 1.8.7 now, but it won't be supported in 7 months (ie. no security updates).

[Michael Stahnke]

 What version of Puppet are you on? Where did you get your 1.9.3? 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/NV-H40aVfrcJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[asq]

W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael Stanhke napisał:

 What version of Puppet are you on? Where did you get your 1.9.3?

3.0.1 and 1.9.3p327 from rvm.

[Jeff McCune]

Ouch, do you happen to have a large number of ruby Gem's installed?

-Jeff

[asq]

yes, a bunch of gems for Foreman...

 <5>root@punch(8)~$ gem stale | wc -l
91

but it's the same pack that we've used for 1.8.7.

[Jeff McCune]

On Thu, Nov 29, 2012 at 8:56 AM, asq <asq...@gmail.com> wrote:

W dniu czwartek, 29 listopada 2012 17:42:48 UTC+1 użytkownik Jeff McCune napisał:

On Thu, Nov 29, 2012 at 12:24 AM, asq <asq...@gmail.com> wrote:
> W dniu środa, 28 listopada 2012 22:57:40 UTC+1 użytkownik Michael Stanhke
> napisał:
>>
>>  What version of Puppet are you on? Where did you get your 1.9.3?
>
> 3.0.1 and 1.9.3p327 from rvm.

Ouch, do you happen to have a large number of ruby Gem's installed?

-Jeff

yes, a bunch of gems for Foreman...

It might not be the cause, but one related change we made in Puppet 3 is that we scan Ruby gem's and will load extensions contained within them.  This is #7788 [1]  This change in behavior could be causing the Puppet autoloader to perform a _lot_ of additional stat() system calls.  Could you run strace, truss, or dtrace and compare the number of system calls between Puppet 2.7 and 3.0 in your environment?

[1] http://projects.puppetlabs.com/issues/7788

-Jeff 

[asq]

W dniu czwartek, 29 listopada 2012 18:23:23 UTC+1 użytkownik Jeff McCune napisał:

It might not be the cause, but one related change we made in Puppet 3 is that we scan Ruby gem's and will load extensions contained within them.  This is #7788 [1]  This change in behavior could be causing the Puppet autoloader to perform a _lot_ of additional stat() system calls.  Could you run strace, truss, or dtrace and compare the number of system calls between Puppet 2.7 and 3.0 in your environment?

not sure if it's about 2.7 => 3.0 migration, as we did it month ago with no aparent degradation (but also no such load decrease as on some slides from PC).
now we just switched to newer ruby and passenger (1.8.7 => 1.9.3p327 and 3.0.2 => 3.0.17).

i'll try to do some tests tomorrow but i need to get both instances up at the same server.

another thing that is involved (and suspected) is hiera, as compilation time deltas differ on different groups of nodes, and it might be related to how much of modules with hiera lookups are included on mostly affected nodes (ie. 120s => 240s on our load balancers).
another thing is puppetdb, but nodes that collect resources, which should impose highest load on puppetdb are not the ones which are much affected (ie. 320s => 360s increase on our nagios server).

is it possible to do evaltrace on puppetmaster?