Module Testing and Facter 3

[Kylo Ginsberg]

In the next few weeks we’ll be releasing facter 3, a native (compiled C++11) implementation of facter. Note that the only packages containing facter 3 will be the puppet-agent packages (e.g. there will be no facter 3 gems).

This change may have some workflow implications for module developers, and we’d love to get some feedback on that, ideas for improvements, etc.

Starting with spec tests: no change *needed* but a small caveat. Specifically, a module can continue to test against gems using its Gemfile, including using the latest puppet 4 and facter 2 gems. The caveat is that those spec tests will then *not* be running against facter 3. Facter 3 supports the facter 2 ruby api for custom facts, so that should “just work” (and we don’t know of any issues where it doesn’t) but at the same there is some risk there.

However, fear not! You can spec test against facter 3 (or more generally against puppet-agent) like so:

• install the desired puppet-agent build

• comment out the puppet and facter lines from your Gemfile

• profit!

On a related note: some such puppet-agent based workflow for specs may be the long-term direction we want to take module spec testing, so that we’re testing against the curated combination of puppet/facter/ruby/etc that a puppet-agent build represents, as opposed to the larger combinatorial matrix (e.g. X puppet versions vs Y facter versions vs Z ruby versions) that many spec tests use today.

Moving on to module acceptance tests: here I’d more proactively encourage module testing to move to using puppet-agent packages for acceptance tests. This will be a benefit both for the facter 3 issue at hand, but more generally (as mentioned above) to ensure that acceptance tests are run against the puppet/facter/ruby/etc combinations that PL itself tests against.

Hope that all makes some sense. Please chime in with any comments.

Thanks,

Kylo

--

Kylo Ginsberg | ky...@puppetlabs.com | irc: kylo | twitter: @kylog

PuppetConf 2015 is coming to Portland, Oregon! Join us October 5-9.

Register now to take advantage of the Early Adopter discount —save $349!

[Gareth Rushgrove]

I'd be happy to see a PR against puppet-module-skeleton
(https://github.com/garethr/puppet-module-skeleton) for how you
envisage testing against puppet-agent to work on (on Travis in this
case). That might be a good way of getting a few people from the
module community to comment too.

Ditto a PR regarding how the standard acceptance tests in
puppet-module-skeleton would use the puppet-agent package/facter 3.

One of the issues I see at present is that no puppet-agent package
exists for OSX I believe, so lots of people who currently run spec
tests locally won't be able to use the puppet-agent builds at present?

Gareth

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/CALsUZFHbyrx28s0eCcrbP_NaeDBK5UGp_DJPAP5GAbUfriuHDA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.



--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Joshua Hoblitt]

I don't typically have system packages for either puppet or facter in my
dev environment as I've had bazaar issues in the past where the system
facter has leaked into the bundler env and cause odd test failures.

Has anyone looked into embedding cfacter in a gem?

Another implication, which is CI system specific and not a deal breaker,
is that having to install packages would require disabling travis
container based builds. I expect that would impact CI for the vast
majority of forge modules.

-Josh

--

[Kylo Ginsberg]

Thanks for the feedback, Gareth and Josh. We need to figure out the steps here, so this is super useful.

A couple minor comments below, and I'm consolidating the replies.

On Sun, May 17, 2015 at 8:37 AM, Gareth Rushgrove <gar...@morethanseven.net> wrote:

I'd be happy to see a PR against puppet-module-skeleton
(https://github.com/garethr/puppet-module-skeleton) for how you
envisage testing against puppet-agent to work on (on Travis in this
case). That might be a good way of getting a few people from the
module community to comment too.

Ditto a PR regarding how the standard acceptance tests in
puppet-module-skeleton would use the puppet-agent package/facter 3.

PRs for both are a great idea. I'll try to work something up in the next couple weeks. I'll probably start with the latter because the need is more pressing there.

 

One of the issues I see at present is that no puppet-agent package
exists for OSX I believe, so lots of people who currently run spec
tests locally won't be able to use the puppet-agent builds at present?

Yeah, excellent point. I confirmed that we *will* have OSX puppet-agent builds (for mavericks and yosemite) -- we just haven't quite gotten there yet. Stay tuned!

 

On Sun, May 17, 2015 at 1:03 PM, Joshua Hoblitt <jo...@hoblitt.com> wrote:

I don't typically have system packages for either puppet or facter in my
dev environment as I've had bazaar issues in the past where the system
facter has leaked into the bundler env and cause odd test failures.

Has anyone looked into embedding cfacter in a gem?

Somewhat, yes. Since it's not actually ruby code any longer, my concern is that this approach could get contorted pretty quickly. I.e. we'd end up with a variety of architecture-specific gems, and we'd have a build chain parallel to the puppet-agent build chain to generate these gems. It's also throwaway work once we start converting puppet code itself to be native (unless we did the same trick there but that extends the contortion).

Another implication, which is CI system specific and not a deal breaker,
is that having to install packages would require disabling travis
container based builds. I expect that would impact CI for the vast
majority of forge modules.

Yep, that occurred to me too. Ideally one could install all needed bits as non-root. I agree that it's not a deal-breaker, but it's definitely a nice-to-have.

Thanks for the great feedback.

Kylo