Puppet module template mentioned at contributor summit

[Gareth Rushgrove]

This came up in discussion a couple of times at the Puppet contributor
summit at Config Management Camp in Gent over the last couple of days
so I thought I'd write up.

A while ago I put together a pretty complete/opinionated skeleton for
puppet modules. Especially if you're not too familiar with ruby or the
ruby ecosystem, or just getting started with testing it should be a
useful starting point.

https://github.com/garethr/puppet-module-skeleton

I've added a bunch more features (including a Guardfile, resource
coverage and support for Beaker integration tests) and got round to
writing up a blog post about what and why:

http://www.morethanseven.net/2014/02/05/a-template-for-puppet-modules/

Hopefully it's useful to a few people. Any features or issues let me know.

Gareth

--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Atom Powers]

Thank you Gareth.
Your module skeleton has helped my module development tremendously.
Even though I created my own, differently opinionated, fork most of
the skeleton is the same and your effort is greatly appreciated.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAFi_6y%2BiRQPPKk8yTLBMiHCNOsLdNFYeaPO8oTCCcuaASj6SaQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Perfection is just a word I use occasionally with mustard.
--Atom Powers--

[Nikola Petrov]

On Wed, Feb 05, 2014 at 06:38:24PM +0100, Gareth Rushgrove wrote:
> This came up in discussion a couple of times at the Puppet contributor
> summit at Config Management Camp in Gent over the last couple of days
> so I thought I'd write up.
>
> A while ago I put together a pretty complete/opinionated skeleton for
> puppet modules. Especially if you're not too familiar with ruby or the
> ruby ecosystem, or just getting started with testing it should be a
> useful starting point.
>
> https://github.com/garethr/puppet-module-skeleton

Damn! I have to confess that I have used your module and I have a
Guardfile that also checks the templates. Maybe it's time I send a pull
request (shame) :). Just know that there are people out there that are
using this!

>
> I've added a bunch more features (including a Guardfile, resource
> coverage and support for Beaker integration tests) and got round to
> writing up a blog post about what and why:
>
> http://www.morethanseven.net/2014/02/05/a-template-for-puppet-modules/
>
> Hopefully it's useful to a few people. Any features or issues let me know.
>
> Gareth
>
> --
> Gareth Rushgrove
> @garethr
>
> devopsweekly.com
> morethanseven.net
> garethrushgrove.com
>

[Craig Dunn]

This is cool, though I realise that it's a (self confessed) opinionated module design, the only thing that really stands out for me is that it follows a rather old, and limited, 'params.pp' pattern.   There is no place for Hiera in this model without hard coding hiera lookup functions in the classes.  Personally I think a 'defaults.pp' pattern is more sensible in todays Puppet.

Eg:

class base (

   $parameter = $base::defaults::$parameter

) inherits base::defaults {

  ...
}

class base::defaults {

   $parameter = $logic ? {

      'foo' => 'bar'
   }
}

Your classes can then look up values as $base::parameter.  This allows the module to default (rather than dictate) attributes based on whatever logic you want to implement but allows the implementer to override the values either at the resource declaration or using Hiera data mapping for base::parameter.

Regards

Craig

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAFi_6y%2BiRQPPKk8yTLBMiHCNOsLdNFYeaPO8oTCCcuaASj6SaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Enviatics | Automation and configuration management

http://www.enviatics.com | @Enviatics

Puppet Training http://www.enviatics.com/training/

[Alessandro Franceschi]

Craig,

Not sure to have understood the difference between a defaults.pp pattern and a params.pp pattern, given that I suppose that if there were parameters in the main module class of Gareth's example they would inherit values in params.pp exactly as the defaults example you've written.

Can be elaborate or link examples of this defaults.pp pattern?

To the list of public modules skeletons let me add this one, that follows stdmod naming conventions:

https://github.com/stdmod/puppet-skeleton-standard

and this alternative with Rip's data in module approach:

https://github.com/stdmod/puppet-skeleton-standard/tree/hiera

Al

[Craig Dunn]

The main difference between Gareth's current params.pp and the 'defaults.pp' model I was suggesting is that in the Gareth's pattern the params class is inherited by the base class, and all the component subclasses reference the variables explicitly in the scope of base::params.... eg:   

service { $<%= metadata.name %>::params::service_name:

Since the params class is not parameterized there is no way to easily override this data using hiera/data mapping.  

By making the base class parameterized, with it's defaults being set in 'defaults.pp' (defaults is a more sensible name than params for this example) and having your component subclass reference $::baseclass::var you can override data on class declaration or in hiera using data mapping.

It's not that different, but allows for more flexibility and tighter hiera integration 

Craig

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/feaadcb3-cc99-45c3-825d-57ba26dc4dc0%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Alessandro Franceschi]

Ok, I think we are talking about the same thing, then:

https://github.com/stdmod/puppet-skeleton-standard/blob/develop/manifests/init.pp.erb

should follow what you described as defaults.pp patterns , just it uses a class named params and not default.

[Gareth Rushgrove]

On 28 February 2014 10:20, Craig Dunn <cr...@craigdunn.org> wrote:
>
> The main difference between Gareth's current params.pp and the 'defaults.pp'
> model I was suggesting is that in the Gareth's pattern the params class is
> inherited by the base class, and all the component subclasses reference the
> variables explicitly in the scope of base::params.... eg:
>
> service { $<%= metadata.name %>::params::service_name:
>

Actually that's weird. I don't actually do that in any of my actual
modules based on this skeleton. I probably merged a pull request
without looking properly.

I'll fix.

Anything in params should never be referenced directly outside init in my view.

Gareth

> https://groups.google.com/d/msgid/puppet-users/CACxdKhF3qheX37vQCraHnzeVP9ObmkHamruBydHq%2BneUMkq%2BeQ%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/groups/opt_out.



--

[Gareth Rushgrove]

On 1 March 2014 09:38, Gareth Rushgrove <gar...@morethanseven.net> wrote:
> On 28 February 2014 10:20, Craig Dunn <cr...@craigdunn.org> wrote:
>>
>> The main difference between Gareth's current params.pp and the 'defaults.pp'
>> model I was suggesting is that in the Gareth's pattern the params class is
>> inherited by the base class, and all the component subclasses reference the
>> variables explicitly in the scope of base::params.... eg:
>>
>> service { $<%= metadata.name %>::params::service_name:
>>
>
> Actually that's weird. I don't actually do that in any of my actual
> modules based on this skeleton. I probably merged a pull request
> without looking properly.
>
> I'll fix.
>
> Anything in params should never be referenced directly outside init in my view.
>

Changed to:

service { $<%= metadata.name %>::service_name:

With the params on init making overriding in hiera easy. Cheers for
the heads up.

Gareth

[Robin Bowes]

We're currently arguing^w discussing the use of the params class in conjunction with hiera lookups and we've arrived at what I think is a pretty good pattern. Let me explain by way of example, using the venerable ntp application

class ntp{

  include ::ntp::params

  Class['::ntp::params']->

  class{'::ntp::install':}->

  class{'::ntp::config':}~>

  class{'::ntp::service':}->

  class{'::ntp::firewall':}->

  Class['::Ntp']

}

class ntp::params(

  $servers,

  $package_ensure = 'present',

  $service_ensure = 'running',

  $service_enable = true,

) {

  # os-specific stuff could go in here

  $packages = ['ntp']

  $service_name = 'ntpd'

  $config_file = '/etc/ntp/ntpd.conf'

}

class ntp::install{

  include ::ntp::params

  package{$::ntp::params::packages:

    ensure => $::ntp::params::package_ensure,

  }

}

class ntp::config{

  include ::ntp::params

  # explicitly declare any vars used in the template

  $servers = $::ntp::params::servers

  file{$::ntp::params::config_file":

    content => template(...)

  }

}

class ntp::service{

  include ::ntp::params

  service{$::ntp::params::service_name":

    ensure => $::ntp::params::service_ensure,

    enable => $::ntp::params::service_enable",

  }

}

class ntp::firewall{

  include ::ntp::params

  # firewall definition in here

}

The basic approach is that all user-changeable values are passed in via the params class. This class should be instantiated before the ntp class.

Typically, this class would be used in a profile class something like this:

class profile_ntp(

  $servers,

) {

  class{'::ntp::params':

    servers => $servers,

  }->

  class{'::ntp'}

}

Alternatively, $servers could come from an explicit hiera lookup:

class profile_ntp{

  class{'::ntp::params':

    servers => hiera('ntp::servers')

  }->

  class{'::ntp'}

}

The hiera key would of course be profile_ntp::servers

I can't decide whether I prefer the explicit lookup or the automatic approach.

I'd be interested to hear any feedback on this approach.

R.

[Christopher Wood]

We could have been talking in my cube. My points when I'm discussing this with coworkers generally go like so...

If you use this:

class { name: }

You will only be able to declare that name once. If you declare classes like this:

include ::name
include ::name::otherclass

Then you will be able to include things anywhere you want without resource conflicts.

You don't need to include a params class since you can inherit from it. Keep in mind what the docs say about inheritance:

http://docs.puppetlabs.com/puppet/latest/reference/lang_classes.html#inheritance

So for your stuff, some example subsets, pretending they're in different files:

class ntp::params {
$servers = ['reasonable', 'defaults']
$service_name = 'ntp'
$service_ensure = 'running'
$package_name = 'ntp'
$package_ensure = 'present'
}

class ntp::install (
$package_name = $ntp::params::package_name,
$package_ensure = $ntp::params::package_ensure,
) inherits ntp::params {
package { $package_name:
ensure => $package_ensure,
}
}

# (other classes here)

class ntp {
include ntp::install
include ntp::config
include ntp::service

Class['ntp::install'] -> Class['ntp::config']
Class['ntp::install'] ~> Class['ntp::service']
Class['ntp::config'] ~> Class['ntp::service']
}

Then you would control the install/config/service parameters via the hiera automatic parameter lookup and avoid needing to explicitly hiera() or hiera_hash(). (Hiera will do an auto-lookup anyway, though if you specify a hiera() as the value of the class parameter, would it do the lookup twice? Not sure, never tried it.)

If you don't know which kind of hiera lookup you may want, this module provides a useful example of how to configure which one with hiera data:

http://forge.puppetlabs.com/mthibaut/users

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> [1]https://groups.google.com/d/msgid/puppet-users/8d0b5ce0-fb3c-4091-bd91-e4791a993616%40googlegroups.com.
> For more options, visit [2]https://groups.google.com/groups/opt_out.
>
> References
>
> Visible links
> 1. https://groups.google.com/d/msgid/puppet-users/8d0b5ce0-fb3c-4091-bd91-e4791a993616%40googlegroups.com
> 2. https://groups.google.com/groups/opt_out

[Robin Bowes]

Hi Christopher,

On 3 Mar 2014, at 17:55, Christopher Wood <christop...@pobox.com> wrote:

We could have been talking in my cube. My points when I'm discussing this with coworkers generally go like so...

If you use this:

class { name: }

You will only be able to declare that name once. If you declare classes like this:

include ::name
include ::name::otherclass

Then you will be able to include things anywhere you want without resource conflicts.

Yes, I realise that. Hence the suggestion to use a profile_ntp class that may be included multiple times without resource conflicts.

You don't need to include a params class since you can inherit from it. Keep in mind what the docs say about inheritance:

http://docs.puppetlabs.com/puppet/latest/reference/lang_classes.html#inheritance

So for your stuff, some example subsets, pretending they're in different files:

class ntp::params {
 $servers = ['reasonable', 'defaults']
 $service_name = 'ntp'
 $service_ensure = 'running'
 $package_name = 'ntp'
 $package_ensure = 'present'
}

class ntp::install (
 $package_name = $ntp::params::package_name,
 $package_ensure = $ntp::params::package_ensure,
) inherits ntp::params {
 package { $package_name:
   ensure => $package_ensure,
 }
}

# (other classes here)

class ntp {
 include ntp::install
 include ntp::config
 include ntp::service

 Class['ntp::install'] -> Class['ntp::config']
 Class['ntp::install'] ~> Class['ntp::service']
 Class['ntp::config'] ~> Class['ntp::service']
}

This is essentially how we’ve been using the params class until very recently.

One of the concerns raised by the use of inheritance was the duplication involved in declaring everything in two places, ie. in the params class and the class that inherits the params class (the ntp::install class in your example).

Additionally, it creates two places in which the parameters can be automatically set. eg. ntp::install::package_name and ntp::params::package_name - this has bitten us a couple of times.

Note also that it is not necessary to pass all params to the inheriting class. Take this example:

class foo::params{

  $bar = ‘params class’

}

class foo inherits ::foo::params {

  notify{$bar:}

}

$bar will contain ‘params class’ as it is inherited from foo::params. Potentially confusing.

This pattern is also influence by one of Gary Larizza’s blog posts [1] in which he recommends:

"

Do all Hiera lookups in the profile

“

Having worked with roles/profiles/modules for some time, and having ended up with a confusing mess of hiera data, some automatically included, some not, that statement was a lightbulb-moment for me. My aim now is to write my modules to pass in all required information via parameters and to do the lookups at the profile level, either explicitly with hiera() lookups or automatically with APL.

As always, there are many ways to skin this particular cat, and YMMV. :)

R.

[1]  http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/

[Christopher Wood]

(inline)

On Mon, Mar 03, 2014 at 06:24:47PM +0000, Robin Bowes wrote:
> Hi Christopher,

> On 3 Mar 2014, at 17:55, Christopher Wood <[1]christop...@pobox.com>

> wrote:
>
> We could have been talking in my cube. My points when I'm discussing
> this with coworkers generally go like so...
>
> If you use this:
>
> class { name: }
>
> You will only be able to declare that name once. If you declare classes
> like this:
>
> include ::name
> include ::name::otherclass
>
> Then you will be able to include things anywhere you want without
> resource conflicts.
>
> Yes, I realise that. Hence the suggestion to use a profile_ntp class that
> may be included multiple times without resource conflicts.
>
> You don't need to include a params class since you can inherit from it.
> Keep in mind what the docs say about inheritance:
>

> [2]http://docs.puppetlabs.com/puppet/latest/reference/lang_classes.html#inheritance

How did it bite you? Somebody around here tried using module::params::variable in hiera and found it didn't work for them. They had to use the module::class::variable even though it had a params default.

> Note also that it is not necessary to pass all params to the inheriting
> class. Take this example:
> class foo::params{
>   $bar = ‘params class’
> }
> class foo inherits ::foo::params {
>   notify{$bar:}
> }
> $bar will contain ‘params class’ as it is inherited from foo::params.
> Potentially confusing.

The confusion seems to depend on if you value whether or not the content of the string has any meaning beyond whether it's sourced correctly from puppet/hiera. I am likely explaining this badly, but at this level the fact that the string matches puppet/hiera is sufficient. The content only matters at the level using the value for something, and arbitrary strings of metadata is a good way to run into trouble. Better to enforce that people use something already there for this purpose ($title) which is a cultural problem not a technological one.

To wit, with different results for the commenting/uncommenting (bad use of class not include for the hiera-less example):


class params {
$bar = "not params class"
notice($title)
}

class other (
$bar = $params::bar,
) inherits ::params {
notice($bar)
notice($title)
}

class { 'other':
#bar => 'useless string rhubarbing',
}


This provides my expected result, since I'm using the $title variable rather than a string I've set myself.

As to that blog post, my mileage definitely varied. The pattern choice seems to depend more on what any group of sysadmins prefers than any specific point of utility.

> This pattern is also influence by one of Gary Larizza’s blog posts [1] in
> which he recommends:
> "
>
> Do all Hiera lookups in the profile
>
> “
> Having worked with roles/profiles/modules for some time, and having ended
> up with a confusing mess of hiera data, some automatically included, some
> not, that statement was a lightbulb-moment for me. My aim now is to write
> my modules to pass in all required information via parameters and to do
> the lookups at the profile level, either explicitly with hiera() lookups
> or automatically with APL.
> As always, there are many ways to skin this particular cat, and YMMV. :)
> R.

> [1]  [3]http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> [4]https://groups.google.com/d/msgid/puppet-users/C3FDC7C0-B150-4D1B-B075-D97F7D7B66D0%40yo61.com.
> For more options, visit [5]https://groups.google.com/groups/opt_out.
>
> References
>
> Visible links
> 1. mailto:christop...@pobox.com
> 2. http://docs.puppetlabs.com/puppet/latest/reference/lang_classes.html#inheritance
> 3. http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/
> 4. https://groups.google.com/d/msgid/puppet-users/C3FDC7C0-B150-4D1B-B075-D97F7D7B66D0%40yo61.com
> 5. https://groups.google.com/groups/opt_out

[Robin Bowes]

(also inline)

On 3 Mar 2014, at 19:25, Christopher Wood <christop...@pobox.com> wrote:

> (inline)
>
> On Mon, Mar 03, 2014 at 06:24:47PM +0000, Robin Bowes wrote:
>>
>> Additionally, it creates two places in which the parameters can be
>> automatically set. eg. ntp::install::package_name and
>> ntp::params::package_name - this has bitten us a couple of times.
>
> How did it bite you? Somebody around here tried using module::params::variable in hiera and found it didn't work for them. They had to use the module::class::variable even though it had a params default.

It worked just fine, but it meant that there were two places where a value could be set, $foo::params::bar and $foo::bar. Given a non-trivial hiera config (ie. more than a couple of levels, spread across multiple modules), it is easy to end up using both and not understand when changing one of them does not work (because it is over-ridden by the other one of which you are not aware).

>
>> Note also that it is not necessary to pass all params to the inheriting
>> class. Take this example:
>> class foo::params{
>> $bar = ‘params class’
>> }
>> class foo inherits ::foo::params {
>> notify{$bar:}
>> }
>> $bar will contain ‘params class’ as it is inherited from foo::params.
>> Potentially confusing.
>
> The confusion seems to depend on if you value whether or not the content of the string has any meaning beyond whether it's sourced correctly from puppet/hiera. I am likely explaining this badly, but at this level the fact that the string matches puppet/hiera is sufficient. The content only matters at the level using the value for something, and arbitrary strings of metadata is a good way to run into trouble. Better to enforce that people use something already there for this purpose ($title) which is a cultural problem not a technological one.

No, I mean that variables from the foo::params class “pollute” the main foo class, even without being explicitly imported or declared. I find this opaque and non-intuitive. I find this much clearer:

class foo::params{
$bar = ‘some value’
}

class foo{
include ::foo::params
$bar = $::foo::params::bar

}

> To wit, with different results for the commenting/uncommenting (bad use of class not include for the hiera-less example):
>
> class params {
> $bar = "not params class"
> notice($title)
> }
>
> class other (
> $bar = $params::bar,
> ) inherits ::params {
> notice($bar)
> notice($title)
> }
>
> class { 'other':
> #bar => 'useless string rhubarbing',
> }
>
> This provides my expected result, since I'm using the $title variable rather than a string I've set myself.

I’m not sure what this example is trying to show?

> As to that blog post, my mileage definitely varied. The pattern choice seems to depend more on what any group of sysadmins prefers than any specific point of utility.

Oh, absolutely - there are lots of ways to do things and the final choice depends on many factors, and it’s not always the most technically pure (in a CS way) that wins out.

I still think it makes a lot of sense to not do hiera lookups from “base” modules and instead group them all in the profile modules so all data lookups are done in one place. It’s certainly a lot clearer.

R.