Module triage: 2014-02-11

[Ashley Penney]

Hi all,

We had our first module triage and I wanted to send out some quick notes.  This was slightly different, being our first, from the platform triages and we discussed some non-PR stuff:

We talked a little about testing, specifically beaker, documentation, module skeletons, and all those kinds of good things.  We talked over some apache 2.4 stuff as well as some generic conversation around datacat/concat within the apache module, and problems we're having with travis and ruby 1.8.7.

Ruby 1.8.7 and testing

Each time you enter a next context within rspec-puppet a lot has to happen under the hood, and a new catalog is generated.  Due to some slowness within Puppet it takes about 2.5 seconds per context to generate a catalog with ruby 1.8.7.  This happens frequently enough in tests that it's a cause of some significant slowness for us.  Andy Parker helped profile the code and ensure there's nothing obviously wrong with the codepath we're taking from testing.  The only solution to faster tests is:

1/ Reduce the number of describe/context's within the tests.

2/ Improve certain parts of puppet, especially the qualified_scope bits.

In the meantime we're going to drop 1.8 from our travis tests (as they simply time out) and look at the number of contexts and see what we can do there to keep the same catalog for more tests.

PRs:

602:  (apenney) Follow up to get a reproducible test case as people feel the current functionality works, and we're wondering if it's a specific version of apache that is causing this.

613:  (apenney) See above.

607:  (apenney) This has some problems for newer versions of ruby, and we'd like to discuss pushing this PR into the passenger module instead and focus on removing the current apache::passenger code in favor of more generic, reusable code.

604: (apenney) Merge post README changes.

603: (apenney) Same, docs waiting.

546: (apenney) Closed this, based on facter 1.7 having this support in, we didn't want to merge 1.6 changes.

Look forward to seeing everyone next Tuesday!

--

Ashley Penney

ashley...@puppetlabs.com

Module Engineer

Join us at PuppetConf 2014, September 23-24 in San Francisco

[Andy Parker]

On Tue, Feb 11, 2014 at 12:03 PM, Ashley Penney <ashley...@puppetlabs.com> wrote:

Hi all,

We had our first module triage and I wanted to send out some quick notes.  This was slightly different, being our first, from the platform triages and we discussed some non-PR stuff:

We talked a little about testing, specifically beaker, documentation, module skeletons, and all those kinds of good things.  We talked over some apache 2.4 stuff as well as some generic conversation around datacat/concat within the apache module, and problems we're having with travis and ruby 1.8.7.

Ruby 1.8.7 and testing

Each time you enter a next context within rspec-puppet a lot has to happen under the hood, and a new catalog is generated.  Due to some slowness within Puppet it takes about 2.5 seconds per context to generate a catalog with ruby 1.8.7.  This happens frequently enough in tests that it's a cause of some significant slowness for us.  Andy Parker helped profile the code and ensure there's nothing obviously wrong with the codepath we're taking from testing.  The only solution to faster tests is:

1/ Reduce the number of describe/context's within the tests.

2/ Improve certain parts of puppet, especially the qualified_scope bits.

If anyone is interested into digging into what we were looking at you can take a look at the trace file at https://dl.dropboxusercontent.com/u/27950908/callgrind.apache-mod-on-a-redhat-osfamily-for-non-special-modules.1392144545.trace. You'll need kcachegrind to view it. You'll notice that the largest amount of time is in the lexer, which looks to have been working on 34 different files. So that would be one place to look for improvements. Henrik Lindberg has been putting a lot of effort into the future parser, however, and so we haven't been concentrating too much on the older one. The other place that is showing a lot of calls and time is the file.rb and scope.rb files. These 2 seem to be related with scope.rb causing a lot of lookups for hostclasses (the internal name for the pp class construct) and looking for the existence of a lot of different files. The majority of this seemes to be triggered by qualified variable lookups.

 

In the meantime we're going to drop 1.8 from our travis tests (as they simply time out) and look at the number of contexts and see what we can do there to keep the same catalog for more tests.

PRs:

602:  (apenney) Follow up to get a reproducible test case as people feel the current functionality works, and we're wondering if it's a specific version of apache that is causing this.

613:  (apenney) See above.

607:  (apenney) This has some problems for newer versions of ruby, and we'd like to discuss pushing this PR into the passenger module instead and focus on removing the current apache::passenger code in favor of more generic, reusable code.

604: (apenney) Merge post README changes.

603: (apenney) Same, docs waiting.

546: (apenney) Closed this, based on facter 1.7 having this support in, we didn't want to merge 1.6 changes.

Look forward to seeing everyone next Tuesday!

--

Ashley Penney

ashley...@puppetlabs.com

Module Engineer

Join us at PuppetConf 2014, September 23-24 in San Francisco

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAC9eg%2Bmr2ijJbxHPZ2uE_BKPfyMCFYco6Y5fOWZas1ZHWErpxg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 23-24 in San Francisco - http://bit.ly/pupconf14