Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

Hi All,

I'm working on a module that builds KVM/libvirt hosts and populates them with predefined VMs.

So far I have the module to where it can create any number of virtual nets, storage pools and volumes using virsh, but it isn't pretty.

I've read on various threads here that create_resources is not a good function to use. This was stated quite emphatically by R.I. Pienaar and others.

Already I've run into the situation where it's hard to control order in which two separate create_resources functions are run and I've seen some kludges to fix it, but I'm looking for a better way and hoping that it doesn't involve the use of custom types because that's currently more than I want to deal with.

Here's what I have and hopefully someone can help me do this in a cleaner way. Sorry that it's not in Github as I'm not ready to put it there yet.

This is an example of the hiera data that I'm working with:

kvm::servertype: 'kvm'

kvm::virtnet_name: 'br0'

kvm::virtnet_forwardmode: 'bridge'

kvm::virtbridge_name: 'br0'

kvm::virtnet_macaddress: '52:54:00:1F:95:6C'

kvm::virtpool_hash:

  default:

    virtpool_size: '429496729601'

  iso-images:

    virtpool_target: '/var/lib/libvirt/iso-images'

    virtpool_format: 'iso'

kvm::virtvol_hash:

  dtlrazorts1.img:

    volcapacity: '80G'

    volformat: 'qcow2'

  dtlrepots1.img:

    volcapacity: '80G'

    volformat: 'qcow2'

  dtlwebvirtmants1.img:

    volcapacity: '60G'

    volformat: 'qcow2'

My init.pp

class kvm (

  $servertype          = $kvm::params::servertype,

  $virtnet_name        = 'undef',

  $virtnet_forwardmode = 'undef',

  $virtbridge_name     = 'undef',

  $virtnet_macaddress  = 'undef',

  $virtpool_hash       = 'undef',

  $virtvol_hash        = 'undef',

) inherits kvm::params {

  include kvm::fw

  if ($servertype == 'kvm') {

    

    file { '/var/opt/lib/pe-puppet/temp':

      ensure => directory,

      owner  => 'pe-puppet',

      group  => 'pe-puppet',      

    }

    

    package { ['libvirt',

               'python-virtinst',

               'qemu-kvm',

               'qemu-kvm-tools',

               'bridge-utils',

               'virt-manager',

               'libguestfs-tools',]:

      ensure => present,

    }

    service { 'libvirtd':

      ensure     => running,

      enable     => true,

      hasstatus  => true,

      hasrestart => true,

      require    => Package['libvirt'],

    }

    

    if $virtpool_hash {

    create_resources('kvm::virtpool', $virtpool_hash)

    }

    include kvm::virtnet

    if $virtvol_hash {

        create_resources('kvm::virtvol', $virtvol_hash)

    }

  }

  if $servertype == 'kvmwebmgr' {

    include kvm::kvmwebmgr

  }

}

The virtpool.pp file:

define kvm::virtpool (

  $virtpool        = $name,

  $virtpool_size   = undef,

  $virtpool_target = '/var/lib/libvirt/images',

  $virtpool_type   = 'dir',

  $virtpool_format = 'raw',

) {

  

  file { "${virtpool_target}":

    ensure => directory,

  } ->  

  exec { "virsh pool-define-as ${name} --target ${virtpool_target} --type ${virtpool_type} --source-format ${virtpool_format} && virsh pool-autostart ${name} && virsh pool-start ${name}":

    path        => '/usr/bin',

    unless      => "virsh pool-list | /bin/grep ${name}",

#    refreshonly => true,

  }

    

}

The virtnet.pp file:

class kvm::virtnet inherits kvm {

  

  file { "/var/opt/lib/pe-puppet/temp/${hostname}_virtnet_${virtnet_name}.xml":

    ensure  => file,

    content => template('kvm/virtnet.xml.erb'),

    owner   => 'pe-puppet',

    group   => 'pe-puppet',

  } ->  

  exec { 'virsh net-destroy default && virsh net-undefine default':

    path   => '/usr/bin',

    onlyif => 'virsh net-list | /bin/grep default',

  } ->  

  exec { "virsh net-define /var/opt/lib/pe-puppet/temp/${hostname}_virtnet_${virtnet_name}.xml && virsh net-autostart ${virtnet_name} && virsh net-start ${virtnet_name}":

    path   => '/usr/bin',

    unless => "virsh net-list | /bin/grep ${virtnet_name}",

  }

    

}

The virtvol.pp file:

define kvm::virtvol (

  $virtvol     = $name,

  $pool        = 'default',

  $volcapacity = '60G',

  $volformat   = 'qcow2',

) {

  

  exec { "virsh vol-create-as ${pool} ${name} ${volcapacity} --format ${volformat}":

    path   => '/usr/bin',

    unless => "virsh vol-list ${name} | /bin/grep ${name}",

    onlyif => "virsh pool-list ${pool} | /bin/grep ${pool}",    

  } 

  

}

The virtnet.xml.erb:

<network>

  <name><%= @virtnet_name %></name>

  <forward mode='<%= @virtnet_forwardmode %>' />

  <bridge name='<%= @virtbridge_name %>'/>

  <mac address='<%= @virtnet_macaddress %>'/>

</network>

-----------------------------------------------------------------------------

end files

-----------------------------------------------------------------------------

It would be super helpful if anyone could point me to a puppet module on github that presents me with a better pattern to use with the hiera data.

Any other help or criticisms are also welcome.

Thanks,

Alex

[Alex Scoble]

By the way, if it's not clear, my ultimate goal is to automate everything so that bare metal servers pxe boot to a razor server, get their CentOS install ISO and kickstart from razor, get installed, boot up, get their network interfaces and logical volumes set up, get all the necessary libvirt packages installed and configured, all the necessary bits required to get VMs up and running (storage pools, virtual volumes, virtual networks), VMs get spun up, they connect to razor and get their OSes installed and everything configured through Puppet.

And the only thing I have to do is turn on the bare metal servers and sign the puppet certs, although I will likely set the Puppet server to autosign those as well so everything is automated. This way we can have a set of lab systems that can be brought up and torn down as quickly as possible.

[Garrett Honeycutt]

On 4/28/14, 3:34 PM, Alex Scoble wrote:
> Hi All,
>
> I'm working on a module that builds KVM/libvirt hosts and populates them
> with predefined VMs.
>
> So far I have the module to where it can create any number of virtual
> nets, storage pools and volumes using virsh, but it isn't pretty.
>
> I've read on various threads here that create_resources is not a good
> function to use. This was stated quite emphatically by R.I. Pienaar and
> others.
>
> Already I've run into the situation where it's hard to control order in
> which two separate create_resources functions are run and I've seen some
> kludges to fix it, but I'm looking for a better way and hoping that it
> doesn't involve the use of custom types because that's currently more
> than I want to deal with.
>
> Here's what I have and hopefully someone can help me do this in a
> cleaner way. Sorry that it's not in Github as I'm not ready to put it
> there yet.
>

> *This is an example of the hiera data that I'm working with:*

>
> kvm::servertype:'kvm'
> kvm::virtnet_name:'br0'
> kvm::virtnet_forwardmode:'bridge'
> kvm::virtbridge_name:'br0'
> kvm::virtnet_macaddress:'52:54:00:1F:95:6C'
> kvm::virtpool_hash:
> default:
> virtpool_size:'429496729601'
> iso-images:
> virtpool_target:'/var/lib/libvirt/iso-images'
> virtpool_format:'iso'
> kvm::virtvol_hash:
> dtlrazorts1.img:
> volcapacity:'80G'
> volformat:'qcow2'
> dtlrepots1.img:
> volcapacity:'80G'
> volformat:'qcow2'
> dtlwebvirtmants1.img:
> volcapacity:'60G'
> volformat:'qcow2'
>

> *My init.pp*

> *The virtpool.pp file:*

>
> define kvm::virtpool (
> $virtpool = $name,
> $virtpool_size = undef,
> $virtpool_target = '/var/lib/libvirt/images',
> $virtpool_type = 'dir',
> $virtpool_format = 'raw',
> ) {
>
> file { "${virtpool_target}":
> ensure => directory,
> } ->
> exec { "virsh pool-define-as ${name} --target ${virtpool_target}
> --type ${virtpool_type} --source-format ${virtpool_format} && virsh
> pool-autostart ${name} && virsh pool-start ${name}":
> path => '/usr/bin',
> unless => "virsh pool-list | /bin/grep ${name}",
> # refreshonly => true,
> }
>
> }
>

> *The virtnet.pp file:*

>
> class kvm::virtnet inherits kvm {
>
> file {
> "/var/opt/lib/pe-puppet/temp/${hostname}_virtnet_${virtnet_name}.xml":
> ensure => file,
> content => template('kvm/virtnet.xml.erb'),
> owner => 'pe-puppet',
> group => 'pe-puppet',
> } ->
> exec { 'virsh net-destroy default && virsh net-undefine default':
> path => '/usr/bin',
> onlyif => 'virsh net-list | /bin/grep default',
> } ->
> exec { "virsh net-define
> /var/opt/lib/pe-puppet/temp/${hostname}_virtnet_${virtnet_name}.xml &&
> virsh net-autostart ${virtnet_name} && virsh net-start ${virtnet_name}":
> path => '/usr/bin',
> unless => "virsh net-list | /bin/grep ${virtnet_name}",
> }
>
> }
>

> *The virtvol.pp file:*

>
> define kvm::virtvol (
> $virtvol = $name,
> $pool = 'default',
> $volcapacity = '60G',
> $volformat = 'qcow2',
> ) {
>
> exec { "virsh vol-create-as ${pool} ${name} ${volcapacity} --format
> ${volformat}":
> path => '/usr/bin',
> unless => "virsh vol-list ${name} | /bin/grep ${name}",
> onlyif => "virsh pool-list ${pool} | /bin/grep ${pool}",
> }
>
> }
>

> *The virtnet.xml.erb:*

>
> <network>
> <name><%= @virtnet_name %></name>
> <forward mode='<%= @virtnet_forwardmode %>' />
> <bridge name='<%= @virtbridge_name %>'/>
> <mac address='<%= @virtnet_macaddress %>'/>
> </network>
>
> -----------------------------------------------------------------------------
> end files
> -----------------------------------------------------------------------------
>
> It would be super helpful if anyone could point me to a puppet module on
> github that presents me with a better pattern to use with the hiera data.
>
> Any other help or criticisms are also welcome.
>
> Thanks,
>
> Alex

Hi,

The create_resources() function allows you to have a data driven design.
This gives you the ability to specify arbitrary data in Hiera and
decouple data from modules. It means you can change the data without the
code. I would argue that any define in your module should have a
corresponding create_resources() function, so that it can be accessed
through data in Hiera.

As for ordering, your defines should probably take care of that, though
you can use collections with chaining to help, such as

Virt::Foo <||> -> Virt::Bar <||>

When you are using create_resources(), I recommend also specifying an
option to use hiera_hash() to collect your hashes. This does a deep hash
merge and will return values from all levels of the hierarchy. My types
module[1] is an example of this that lets you define file, mount, and
cron resources as hashes.

[1] - https://github.com/ghoneycutt/puppet-module-types

Best regards,
-g

--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[Alex Scoble]

Hi Garrett,

Thanks for the response. The module you posted does indeed help as it shows how to deal with hashes of hiera data without using the create_resources() function.

[Alex Scoble]

I think I get it now.

The types module is actually a wrapper for tying hiera hash data to the three built in types, but this pattern could also be used with a defined type perhaps?

Thanks,

Alex

[Alex Scoble]

Sorry, this is what happens when I'm too tired. I see all the pieces now. The types module is using create_resources() as well as the hiera_hash() function.

I'll read the collectors documentation again.

Any idea why R.I. Pienaar thinks that it's a bad idea to use the create_resources() function?

Thanks,

Alex

[jcbollinger]

On Monday, April 28, 2014 2:34:43 PM UTC-5, Alex Scoble wrote:

Hi All,

I'm working on a module that builds KVM/libvirt hosts and populates them with predefined VMs.

So far I have the module to where it can create any number of virtual nets, storage pools and volumes using virsh, but it isn't pretty.

I've read on various threads here that create_resources is not a good function to use. This was stated quite emphatically by R.I. Pienaar and others.

If anyone has been vocal (here) about Puppet functions not to use then it's me, and create_resources() is not one I rant about.  As a general rule I do prefer expressing configuration in DSL where that's feasible, and you can, in fact, do the job of create_resources() almost entirely with ordinary Puppet DSL (you need to add only stdlib's keys() function).  Nevertheless, I'm not persuaded that going to substantial effort and length to avoid create_resources() is necessarily a win.

On the other hand, I have come out rather strongly against the ensure_resource() function.  Were you perhaps thinking of that?

 

Already I've run into the situation where it's hard to control order in which two separate create_resources functions are run and I've seen some kludges to fix it, but I'm looking for a better way and hoping that it doesn't involve the use of custom types because that's currently more than I want to deal with.

Indeed, one of the issues that does exist with create_resources() is that you cannot express resource references in Hiera data (though I have seen one recent report that claims otherwise).  When part of the point of using that function in the first place is that you don't want to have to know exactly which resources you are actually declaring, that can present a challenge.  It is, nevertheless, a challenge that can be worked around.  In some cases, the "workaround" is a mechanism, such as collector chaining,  that you might reasonably have used anyway.

> It would be super helpful if anyone could point me to a puppet module on github that presents me with a better pattern to use with the hiera data.

[...]

> Any other help or criticisms are also welcome.

That looks pretty clean to me already.  It does appear that maybe you want to express relationships between your Kvm::Virtpool and Kvm::Virtvol instances, but Garrett appears already to have gotten you pointed in a useful direction with that (collector chaining).

If you want responses to particular commentary about the create_resources() function then it would be helpful to provide a pointer to that commentary.  R.I. is a pretty smart guy, and he has thought a lot about how Puppet can and should use data, so I'm not remotely going to try to respond to anything he may have said on the topic without actually reading it for myself

Additional, minor comments about your code:

1. For any but the most simple Execs, I recommend using descriptive resource titles, and expressing the commands themselves via Exec's 'command' property.  It is much, much easier that way to understand their purpose, especially in log output.
2. There are only two reasons for using class inheritance: to make a variable of the inherited class reliably available for use as a class parameter default, and to enable overriding properties of a resource declared by the inherited class.  The latter is the original purpose, and the former is merely a side effect, but the former is by far the more common reason for class inheritance these days.  In all other cases, an ordinary class declaration (include 'foo') can and should be used instead, if indeed anything of the sort is needed at all.  Avoid class inheritance where it is unneeded.  Especially avoid inheriting from a parameterized class.
   
3. Your class kvm has the strung 'undef' as the defined default value for a bunch of parameters.  That's quite different from the undef keyword; is it what you really mean?  If it were then I would expect to see some kind of test for those values, but I don't.
4. More generally, don't define class and type parameter default values unless you really mean it.  If you forget to declare a needed resource or class parameter then Puppet will diagnose the issue if and only if the parameter has no defined default value.  Do not write definitions or classes such that failure to declare all the needed parameters will silently yield an invalid configuration, as appears may be the case for some of your defined parameters.
   

John

[Alex Scoble]

Hi John,

Thanks so much for your feedback. It's extremely useful for me at this stage of my education in the Puppet DSL.

Here is the Puppet Users group thread where R.I. Pienaar said that he felt that using create_resources() was bad: https://groups.google.com/forum/?fromgroups#!searchin/puppet-users/create_resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ

Responses to comments:

1. That makes perfect sense. I'll do that from now on. Thanks.

2. As far as I can see, I'm using class inheritance solely to make sure that class parameters are available to the main and sub classes. Maybe you are seeing that I'm using inheritance where it's not yet strictly needed? I'll have to look more closely at my code and fix any instances of inheritance where it's unnecessary. Thanks.

3. You are right. I'm using the wrong kind of 'undef' here.

4. Yep, I'll have to look at the parameters and clean this up as well.

Thanks much!

Alex

[jcbollinger]

On Tuesday, April 29, 2014 11:37:02 AM UTC-5, Alex Scoble wrote:

Hi John,

Thanks so much for your feedback. It's extremely useful for me at this stage of my education in the Puppet DSL.

Here is the Puppet Users group thread where R.I. Pienaar said that he felt that using create_resources() was bad: https://groups.google.com/forum/?fromgroups#!searchin/puppet-users/create_resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ

For the record, then, R.I. said "I dont tend to use create_resources as I consider it about as bad as eval() :) ".  I hardly find that an emphatic and categorical  condemnation.  In fact, I think that lines up relatively well with my own position on the matter, expressed earlier in this thread.
 

Responses to comments:

2. As far as I can see, I'm using class inheritance solely to make sure that class parameters are available to the main and sub classes. Maybe you are seeing that I'm using inheritance where it's not yet strictly needed? I'll have to look more closely at my code and fix any instances of inheritance where it's unnecessary. Thanks.

I think I was looking primarily at class kvm::virtnet.  It is not parameterized itself and it does not contain any resource parameter overrides, so it cannot have any need to inherit from class kvm.  Moreover, class kvm is parameterized, so nobody should inherit from it.  It looks like you may be inheriting simply to avoid prefixing class kvm's variables with 'kvm::' where you reference them, but that's poor form even when you do use inheritance.

If you are concerned about a possibility that class kvm::virtnet will be evaluated before class kvm, so that the references to kvm's variables are not resolved correctly, then 'include' class kvm instead of inheriting from it.  But that's dangerous and nearly pointless.  It is dangerous because if class kvm is declared via a parameterized-style declaration, and class kvm::virtnet really is evaluated first, then catalog compilation will fail -- as it would also do under the same circumstances with classes as you present.  It is nearly pointless because if you can rely on class kvm being evaluated first so that catalog compilation succeeds, then class kvm::virtnet doesn't need to inherit, include, or do anything else along those lines to work properly.

The caveat is that if class kvm is declared via 'include' or (I think) hiera_include(), and class kvm::virtnet is also independently declared, then it is possible for class kvm::virtnet to be evaluated first without the catalog compiler crashing later.  One possible approach to that problem is basically just to say "don't do that".  If class kvm::virtnet is intended only for internal use by the module, as appears may be the case, then document that clearly and be done with it.  None of this is an issue with respect to class kvm::virtnet being declared by class kvm, by the way; if that's the only declaration of class kvm::virtnet then class kvm is certain to be evaluated first.


John

[R.I.Pienaar]

----- Original Message -----
> From: "jcbollinger" <John.Bo...@stJude.org>
> To: puppet...@googlegroups.com
> Sent: Wednesday, April 30, 2014 3:15:03 PM
> Subject: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources
>
>
>
> On Tuesday, April 29, 2014 11:37:02 AM UTC-5, Alex Scoble wrote:
> >
> > Hi John,
> >
> > Thanks so much for your feedback. It's extremely useful for me at this
> > stage of my education in the Puppet DSL.
> >
> > Here is the Puppet Users group thread where R.I. Pienaar said that he felt
> > that using create_resources() was bad:
> > https://groups.google.com/forum/?fromgroups#!searchin/puppet-users/create_resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ
> >
>
>
> For the record, then, R.I. said "I dont tend to use create_resources as I
> consider it about as bad as eval() :) ". I hardly find that an emphatic
> and categorical condemnation. In fact, I think that lines up relatively
> well with my own position on the matter, expressed earlier in this thread.

And I still think that, unfortunately there are often few other options.

Today you should just suck it up and use the future parser and loop the data
though, it seems to be speedy enough and relatively stable, syntax has been
locked down etc.

I've not followed this thread too closely but I'd hope the f.parser iterating
your hashes/arrays should do all you need. If not it's probably worth highlighting
those.

ie. the closer to reality the f.parser becomes the less you should be using
create_resources.

[Alex Scoble]

Hi John,

Thanks for the info.

By the way, I started using the inherit pattern with params because of the myriad of Splunk modules out there that do it, including this one from Puppet Labs https://github.com/puppetlabs/puppetlabs-splunk

I understand why the pattern is used, it makes it a bit easier to deal with variables, particularly with hiera. I also understand what you are saying. This is the sort of thing that can drive one nuts, heh. It would be nice if there was only one way to do things and the one way was also the right way, but such nirvana is hard to find.

Thanks,

Alex

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/QNZyd4ipB0U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/35a23857-cc2c-445e-adb0-aa8ce404e760%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alex Scoble]

Thanks, R.I.Pienaar,

I sometimes wonder how the moving target that is the Puppet DSL slows adoption of the product.

I'll keep an eye on the future parser. Hopefully, it's in PE soon. Of course, then I'll need to wait until someone creates a useful module that shows how to use the pattern. I'm not quite smart enough (or knowledgeable enough yet) to create a new pattern on my own.

Thanks,

Alex

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/QNZyd4ipB0U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1444896807.322.1398867543907.JavaMail.zimbra%40devco.net.

[jcbollinger]

On Wednesday, April 30, 2014 9:23:49 AM UTC-5, Alex Scoble wrote:

Hi John,

Thanks for the info.

By the way, I started using the inherit pattern with params because of the myriad of Splunk modules out there that do it, including this one from Puppet Labs https://github.com/puppetlabs/puppetlabs-splunk

Where that PL module uses class inheritance, it is for one or both of the two reasons I gave earlier.

Some of the classes inherit class splunk::params in order to use variables of that class as default values in their own parameter lists.  Inheriting splunk::params ensures that that class has already been evaluated when the child class is evaluated.  Where the child classes in fact refer to variables from splunk::params, they do so via their qualified names.

The classes in the 'platform' subpackage inherit splunk::virtual.  They do so in order to override parameters of resources declared by splunk::virtual.  As far as I can tell, they do not refer to any of splunk::virtual's class variables via unqualifed names.

So actually, that module provides a pretty good example and model of several of my above points about class inheritance.  It does have a few flaws, though -- most notably, it parameterizes class splunk::params that is intended to be inherited.  Notwithstanding my practical arguments, the language reference explicitly forbids inheriting from parameterized classes, so the PL crew really should know better.

 

I understand why the pattern is used, it makes it a bit easier to deal with variables, particularly with hiera.

If that's what you're gleaning from the PL module then no, it seems that you don't understand why the pattern is used (there).

 

I also understand what you are saying. This is the sort of thing that can drive one nuts, heh. It would be nice if there was only one way to do things and the one way was also the right way, but such nirvana is hard to find.

I do not claim to be keeper of the One True Way, but I am fairly well informed about this community's current opinions about best practices for Puppet manifest code (and at times I have disagreed with aspects of them).  You will certainly code things as you think most appropriate, but you should make the best-informed decisions you can about how that actually is.  Myself, I am not much swayed by the "Somebody else is doing it" rationale.


John

[Felix Frank]

On 04/30/2014 04:28 PM, Alex Scoble wrote:
> I sometimes wonder how the moving target that is the Puppet DSL slows
> adoption of the product.

Well, I don't know what part of the discussion led you to the conclusion
of the Puppet DSL being a "moving target", but I can confidently state
that it is just not true.

Yes, the language is seeing constant additions. This is inherently a
good thing, because those strive to ever make the user's life easier.
Such changes are usually opt-in, you don't have to use things you don't
need.

Sure, users face recurring issues that get iterated time and again on
the list, because solving them is a less than trivial design problem.
When those can eventually be solved, the offending parts may be taken
out of the language (see Dynamic Scoping), but never without an
appropriate deprecation phase. The dev team goes to great lengths to
ensure that your manifest will not just break.

As for the future parser, that is just an elegant way to allow people to
preview the Puppet 4 language additions before its release (yes, Puppet
4 *will* have an enhanced DSL - I would argue against the view that
turning a new page for a major release makes you a "moving target").

I feel compelled to add - the future parser is a preview system. It is
*not* something you should enable on a production master for a realistic
field test. Production grade for this parser will be reached in Puppet 4.

Regards,
Felix

[Alex Scoble]

If you are trying to follow Puppet Labs recommended best practices, it's definitely a moving target.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/QNZyd4ipB0U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/536785FA.5030203%40alumni.tu-berlin.de.

[jcbollinger]

On Monday, May 5, 2014 9:32:05 AM UTC-5, Alex Scoble wrote:

If you are trying to follow Puppet Labs recommended best practices, it's definitely a moving target.

Obviously, "best practices" are not at all the same thing as the DSL itself.

Still, I'm curious what you define as "PuppetLabs recommended best practices".  PL has indeed at times published various best practices guides on their site, but the current one is so high-level as to hardly recommend anything close enough to the DSL as to lead anyone to confuse the two.  The guides they have published in the past have come and gone as Puppet has grown and experience with it has grown, both at PL and in the community.

PL is indeed a fast-moving company, though, and Puppet is a faster-moving product than many.  Nevertheless, Felix is right: the Puppet core is pretty stable.  The key to best practices, whether with Puppet or any other complex system, is to understand the system well enough to know why one practice is better than another.  That's something we can help you with here, though you will have to contend with a plurality of opinions.


Best,

John

[Ramin K]

On 5/5/2014 7:32 AM, Alex Scoble wrote:
> If you are trying to follow Puppet Labs recommended best practices, it's
> definitely a moving target.

I find it best not to change my workflow or methodology until it makes
sense on my system regardless of what the community or even Puppet Labs
has said.

A few of the things I have completely ignored,
- inheriting params.pp class to get local variables
- ENC
- parameterized classes
- 99.8% of Forge modules
- data bindings

A few of the things I do/have done that Puppet no longer recommends or
has never recommended,
- use stages
- include subclasses to change behavior of modules, class
postfix::config::relay inherits postfix::config {
- specific hiera calls
- used RVM to run my Puppet master

On the flip side I did have a 2000+ line commit last year when I
switched over to role/profile. However the massive benefits in
organization (at least for me) were worth the time investment. At some
point I'll probably bite the parameterized classes/data binding bullet
as well.

I ran across a paper recently with the following quote. I think keeping
this definition of Engineering in mind would be a good thing for the
Puppet community instead of searching for the elusive best solution.

http://www.lunduniversity.lu.se/o.o.i.s?id=24965&postid=2968043
"Engineering has no hint of the absolute, the deterministic, the
guaranteed, the true. Instead it fairly reeks of the uncertain, the
provisional and the doubtful. The engineer instinctively recognizes this
and calls his ad hoc method “doing the best you can with what you’ve
got,” “finding a seat-of-the-pants solution,” or just “muddling
through”. (Koen, 1985, p. 23)."

Ramin

[Felix Frank]

On 05/07/2014 12:52 AM, Ramin K wrote:
>
> I find it best not to change my workflow or methodology until it
> makes sense on my system regardless of what the community or even Puppet
> Labs has said.

Ramin, I could hardly agree more. Even your ignored practices resemble
my own personal choices very closely (those perhaps come rather natural
to old schoolers).

If I understood Alex right though, he also feels that the apparent flux
might be hindering broader acceptance of Puppet. If that is indeed the
case, we have a problem that we should talk about. (Note that apparent
stability is more important than the technicalities.)

However, it has been my feeling that general adoption is not one of
Puppet's problems. On the contrary, Puppet users usually form the
largest crowd in any kind of forum concerned with the configuration
management problem. The user base keeps growing and the community is
literally buzzing with activity.

Alex, are there concrete issues that you have faced concerning the ease
of adoption?

Thanks,
Felix

[Alex Scoble]

I'm sorry, I misspoke, I should have said that for us Puppet (PE actually) has been a moving target in a number of ways.

For instance, we started out heavily using the PE dashboard to define what classes specific nodes would get and related variables, pretty quickly realized that that wasn't very scalable or repeatable and found hiera and environments.

Puppet does a decent job of explaining environments, but their documentation in regards to integrating with git and gitolite is inadequate, particularly considering that they strongly recommend going down that path. Oh and while you are researching how to make dynamic environments work, you invariably run into R10k which sounds really awesome, but then you wonder how do you use it?

Dynamic environments are great, extremely useful, but centrally managing what nodes are in what environment wasn't possible until we discovered the console_env module, which unfortunately isn't working for us after we upgraded to PE 3.2 (and to make a point about how the product is a moving target, they are currently working on a new way of assigning roles to systems). Hiera is also great as it's much faster to work with than the dashboard, but it's a lot easier to make mistakes with hiera. Also, for many situations, you need a hiera yaml file for individual nodes (like when you want a group of systems to use a specific class). So now there's the roles and profiles pattern, which Puppet recommends using along with hiera, but they don't provide good documentation at all on how to use both together.

Plus if you pay attention to R.I. Pienaar (and you definitely should, in my opinion), you see that he's proposed a new pattern where you use hiera data within modules, but getting that to work requires using one of the modules that aren't yet in a finished state...

I've talked to people who don't think that Puppet has handled the way they've added features and patterns and deprecated other features and patterns, but we haven't personally run in to that.

All in all, I love the product, don't get me wrong, and maybe it looks pretty stable when you're at the guru level and figuring out new stuff is fairly easy, but for me, it's just me and a coworker trying to mature our linux and puppet infrastructure, processes and workflows. Neither of us are developers, so are learning that side of things as best as we can while we learn Puppet, git, hiera, Geppetto (which is the bee's knees, just be super careful when you try to merge two significantly different branches together using it), beaker, rspec, ruby and so on.

I'm always questioning our choice in tools because that's how you stay ahead in this game and when I go looking for why people are using Salt Stack and Ansible instead of Puppet or Chef, the number one reason I've seen is complexity of the code needed to do something.

Having said that, now that I've figured out a number of things and can pretty much do anything I want with Puppet, albeit crudely (you can see my kvm/webvirtmgr module as evidence of this), I don't see a reason to switch, but I can certainly understand why some people may look at the DevOps space and gravitate towards Salt Stack and Ansible over Puppet.

Thanks,

Alex

[Alex Scoble]

And this is aside from the significant resistance we've been getting from our traditional Windows admins over using Puppet, which is a problem that Puppet definitely can't solve. Their partnership with Microsoft is a very good thing, in my opinion though.

Oh, and Chocolatey. :D

--Alex

[Felix Frank]

On 05/07/2014 07:04 PM, Alex Scoble wrote:
> All in all, I love the product, don't get me wrong, and maybe it looks
> pretty stable when you're at the guru level and figuring out new stuff
> is fairly easy, but for me, it's just me and a coworker trying to mature
> our linux and puppet infrastructure, processes and workflows. Neither of
> us are developers, so are learning that side of things as best as we can
> while we learn Puppet, git, hiera, Geppetto (which is the bee's knees,
> just be super careful when you try to merge two significantly different
> branches together using it), beaker, rspec, ruby and so on.

Thanks for the in-debt write-up, Alex.

So what I took from it is that, for all the documentation and buzzing
blog-sphere, it can be daunting to pick the patterns, solutions and tool
chain that are best fitted for one's particular needs and goals. Is that
a fair summary?

I can get behind that, I guess, but I have a hard time coming up with a
real good strategy to handle this. Yes, it's a rich ecosystems, and that
has its own downsides. Perhaps the Learning Puppet course should
culminate in the ultimate advice to join the list and ask for hints
regarding one's own goals and environment?

But then such threads run an elevated risk of starving, I feel, because
some new users tend to ask quite broadly with too little to go on for
the senior members to take interest in resolving the challenge.

</ramble>

Felix