Compiling catalogs as CD pipeline

[Alex Harvey]

Hi all,

As part of an upgrade to Puppet 4, my team is considering to switch from an in-house tool that compiles puppet manifests into catalog against the installed version of Puppet

https://github.com/oldNoakes/puppet-validator

Looking around it seems that others do things like puppet-lint, puppet parser validate, puppet-rspec and beaker in their pipelines.

Do others also compile catalogs from their CI/CD systems?  If so, how do others do it?

Thanks,

Alex

[Clayton O'Neill]

We do before and after catalog compiles for every change pre-merge, and push the results into Gerrit (the code review tool we use).  We have jenkins jobs that pull back the puppet facts from every puppet master a few times a day ,and we do the catalog compiles against every node type served by every puppet master.  We've got some home grown scripts for doing this that parallelizes the catalog compiles across multiple jenkins slaves and across multiple cpus on each slave so it only takes a few minutes to run the test.  We've found the before and after diff to be hugely useful.  We're using R.I.Pienaar's catalog-diff tool, since rodjek's is a bit more opinionated in ways that don't work as well for us.

We've proposed a talk for PuppetConf to talk about this and some other aspects of our CI/CD infra that might be common in other organizations, but we've not seen many people talk about. 

[Alex Harvey]

Yes, I think that's a great idea.  The code we use also compiles catalogs for every node, but doesn't go as far as to tell us the differences before and after for every node.  If you have skeptical managers worried that Puppet could break every node in your system, your idea would give you a very high level of confidence that such things could never happen.

If I may ask, how do you actually compile the catalog though.  Are you using puppet master --compile or calling the internals directly?  Is your code publicly available?

[Clayton O'Neill]

On Monday, May 25, 2015 at 11:09:43 AM UTC-4, Alex Harvey wrote:

Yes, I think that's a great idea.  The code we use also compiles catalogs for every node, but doesn't go as far as to tell us the differences before and after for every node.  If you have skeptical managers worried that Puppet could break every node in your system, your idea would give you a very high level of confidence that such things could never happen.

If I may ask, how do you actually compile the catalog though.  Are you using puppet master --compile or calling the internals directly?  Is your code publicly available?

We're using the "puppet master --compile" approach.  It's not available right now but it's a pretty simple python wrapper around puppet compile using the multiprocessing library.  Right now it makes a lot of assumptions about the layout of facts on disk. 

[Tim Hughes]

I have been using something far more rudimentary for testing upgrades to puppet masters. But the basic idea is the same. Just uploaded my code as a starting point for someone else https://git.timhughes.org/puppet/puppet_compare_manifests

Haven't done it in a while but I installed puppet and R.I.Pienaar's catalog-diff tool using rbenv then used the scripts in that git repo.

[Alex Harvey]

In case anyone is looking at this in the archives:

I ended up concluding that the tool we were using is redundant and superseded by rspec-puppet.

I now have rspec-puppet host tests that look something like:

require 'spec_helper'

['myhost1, 'myhost2'].each do |fqdn|

  hostname, node_environment, n, node_datacentre =

    /(.*)\.(.*)([12])\..*\.(.*)\..*\.mydomain.com/.match(fqdn).captures

  node_stream = node_environment + n

  describe fqdn do

    let(:facts) {{

      :hostname   => hostname,

      :otherfacts => otherfacts,

    }}

    it {

      should compile.with_all_deps

    }

  end

end

I also set up parallel_tests from https://github.com/grosser/parallel_tests  and this got my tests running in less than 5 minutes, whereas they were taking over 20 minutes before I did that.

There isn't a lot of good documentation on how to do all this, and I hope to write a blog post fairly soon.

[David Schmitt]

Excellent, that would be most helpful!



Cheers, David

[Alex Harvey]

I finally got around to writing the promised blog post:

http://razorconsulting.com.au/parallelising-rspec-puppet.html

I might add right here that I'd be willing to add this functionality into the Puppet Labs supported modules if I could get someone to give me feedback on the approach and if people think it's a valuable contribution.