Hiera, define, class, putting all together

[Helmut Schneider]

Hi,

I'm new to puppet and read docs as much as possible, so forgive some
confusion in my code.

ubuntu-common.yaml:
---
packages:
- bc
- bsd-mailx
- fail2ban
- logrotate
- open-vm-tools
- openssh-server
- rsyslog
- rsync
- sudo
- snmpd
apacheModules:
- auth_kerb
- authnz_ldap
- status
classes:
- ubuntu-common-files

nodes/xyz.yaml:
packages:
- postfix
- wget
apacheModules:
- mpm_prefork
- php
- ssl

apache24::modules.pp:
define modules ($module = $title) {
class { "apache::mod::$module": }
}

install-packages::packages.pp:
define install-packages ($package = $title) {
if $package != undef {
package { $package:
ensure => installed
}
# I have some classes named as packages that copy files e.g.
if defined ("$package") {
class { "$package": }
}
}
}

nodes.pp:
case $operatingsystem {
/^(Debian|Ubuntu)$/: {
$ubuntuDefaultPackages = hiera_array ('packages', '',
'ubuntu-common')
$ubuntuExtraPackages = hiera_array ('packages', '', "nodes/$fqdn")
$ubuntuPackages = [ $ubuntuDefaultPackages, $ubuntuExtraPackages ]
$ubuntuDefaultApacheModules = hiera_array ('apacheModules', '',
'ubuntu-common')
$ubuntuExtraApacheModules = hiera_array ('apacheModules', '',
"nodes/$fqdn")
$ubuntuApacheModules = [ $ubuntuDefaultApacheModules,
$ubuntuExtraApacheModules ]
hiera_include ('classes', '', 'ubuntu-common')
}
default: {
}
}
hiera_include ('classes', '', "nodes/$fqdn")
[...]
install-packages { [ $ubuntuDefaultPackages, $packages ]: } <= works
apache24::modules { $ubuntuApacheModules: } <= fails

The error is:

Error: Could not retrieve catalog from remote server: Error 400 on
SERVER: Duplicate declaration: Apache24::Modules[mpm_prefork] is
already declared in file
/etc/puppet/environments/production/manifests/nodes.pp:54; cannot
redeclare at /etc/puppet/environments/production/manifests/nodes.pp:54
on node xyz

Apache24::Modules[mpm_prefork] varies. I'm using the apache class from
puppetlabs. Apache24::Modules is called only from nodes.pp, there are
no duplicates.

My question is: Is the approach itself correct? I want to pass
$packages and $apachemodules to a class that does the install.

Thanks.

[jcbollinger]

On Saturday, June 4, 2016 at 7:20:19 PM UTC-5, Helmut Schneider wrote:

Hi,

I'm new to puppet and read docs as much as possible, so forgive some
confusion in my code.

[...]
 

nodes/xyz.yaml:
packages:
  - postfix
  - wget
apacheModules:
  - mpm_prefork
  - php
  - ssl

[...]
 

nodes.pp:
case $operatingsystem {
  /^(Debian|Ubuntu)$/: {

 

[...]

    $ubuntuDefaultApacheModules = hiera_array ('apacheModules', '',
'ubuntu-common')
    $ubuntuExtraApacheModules = hiera_array ('apacheModules', '',
"nodes/$fqdn")
    $ubuntuApacheModules = [ $ubuntuDefaultApacheModules,
$ubuntuExtraApacheModules ]
    hiera_include ('classes', '', 'ubuntu-common')
  }
  default: {
           }
}
hiera_include ('classes', '', "nodes/$fqdn")
[...]
install-packages { [ $ubuntuDefaultPackages, $packages ]: } <= works
apache24::modules { $ubuntuApacheModules: }                 <= fails

The error is:

Error: Could not retrieve catalog from remote server: Error 400 on
SERVER: Duplicate declaration: Apache24::Modules[mpm_prefork] is
already declared in file
/etc/puppet/environments/production/manifests/nodes.pp:54; cannot
redeclare at /etc/puppet/environments/production/manifests/nodes.pp:54
on node xyz

Apache24::Modules[mpm_prefork] varies. I'm using the apache class from
puppetlabs. Apache24::Modules is called only from nodes.pp, there are
no duplicates.

If there are no duplicates among your Hiera data, then it follows that you are creating the duplication in your manifests, which indeed appears to be plausible.  In any event, you do appear to be using Hiera in a less than efficient manner.  Here are some important considerations that bear on the question:

• The choice among functions hiera(), hiera_array(), and hiera_hash() has little to do with the expected type of the value returned.  Although the latter two do return an array and a hash, respectively, the plain hiera() function will also return one of those if that is the type of the data it finds associated with the given key.  The primary distinction among these is the lookup strategy.  In particular, the hiera_array() function performs an "array-merge" lookup, which collects data from the whole hierarchy, whereas the plain hiera() function performs a priority lookup, returning only the value from the highest-priority hierarchy level that provides one.
• The third parameter to the hiera-family functions is optional, and usually unneeded.  Additionally, I suspect its docs are misleading for hiera_hash() and hiera_array(): whereas it does cause the specified hierarchy level to be consulted, first, it does not makes sense for these two functions for the search to stop there in the event that the key is found in the override level.  That ought to apply only to the hiera() function, from which the docs for that argument in the other two functions appear to have been copied.
  

 

My question is: Is the approach itself correct? I want to pass
$packages and $apachemodules to a class that does the install.

There's nothing inherently wrong with constructing those arrays and using them as resource titles.  There's nothing any more wrong about using array-titled, resource-like class declarations than there is about using resource-like class declarations in general.  If you use array titles, however, then you undertake responsibility to ensure that the arrays contain no duplicate elements.  In this case, it appears that you have ensured the opposite.

Consider this call:

hiera_array ('apacheModules', '', 'ubuntu-common')

It performs an array-merge lookup for key 'apacheModules', with hierarchy level 'ubuntu-common' inserted at the top of the hierarchy.  That level is already in the hierarchy for the node, or so I must presume.  Now remember that an array-merge lookup collects data from every level in the hierarchy, so if that duplicate level contains any data for the requested key then you will automatically get dupes for all those data.

Even if you weren't getting dupes by virtue of having the same hierarchy level consulted twice, it looks like you have a second avenue for dupes.  You are performing two array-merge lookups and then concatenating the results.  Since each array-merge lookup will collect data from the whole hierarchy (plus, in your case, an extra level), the two sets of results you are concatenating will have many elements in common, so concatenating them produces dupes.

The bottom line is that you're making this a lot harder than it needs to be.  You have no need to use a manual hierarchy override if you simply define the hierarchy appropriately, as the details of your error suggest you have already done.  And you do not need separate array-merge lookups for each key: with the hierarchy defined correctly, a single one will do what you seem to be trying to do manually.  In fact, even your 'case' statement is a bit suspect.  With appropriate use of fact interpolation in your hierarchy definition, you should be able to reduce [the section you presented of] your nodes.pp to just this:

$packages = hiera_array ('packages', [])
$apacheModules = hiera_array ('apacheModules', [])
hiera_include ('classes')
install-packages { $packages: }
apache24::modules { $apacheModules: }

John

[Helmut Schneider]

jcbollinger wrote:

> On Saturday, June 4, 2016 at 7:20:19 PM UTC-5, Helmut Schneider wrote:
> >
> > Hi,
> >
> > I'm new to puppet and read docs as much as possible, so forgive
> > some confusion in my code.

[Code]

> Consider this call:
>
> hiera_array ('apacheModules', '', 'ubuntu-common')
>
> It performs an array-merge lookup for key 'apacheModules', with
> hierarchy level 'ubuntu-common' inserted at the top of the hierarchy.
> That level is already in the hierarchy for the node, or so I must
> presume. Now remember that an array-merge lookup collects data from
> every level in the hierarchy, so if that duplicate level contains any
> data for the requested key then you will automatically get dupes for
> all those data.
>
> Even if you weren't getting dupes by virtue of having the same
> hierarchy level consulted twice, it looks like you have a second
> avenue for dupes. You are performing two array-merge lookups and
> then concatenating the results. Since each array-merge lookup will
> collect data from the whole hierarchy (plus, in your case, an extra
> level), the two sets of results you are concatenating will have many
> elements in common, so concatenating them produces dupes.

This in fact wasn't clear to me, I thought it would restrict the serach
to ubuntu-common.yaml and not extend it.

> In fact, even your 'case' statement is a bit suspect.

Without the case statement, how can I make sure that ubuntu only
receives classes for ubuntu and not e.g. for Windows then?

> With appropriate use of fact interpolation in your hierarchy
> definition, you should be able to reduce [the section you presented
> of] your nodes.pp to just this:
>
> $packages = hiera_array ('packages', [])
> $apacheModules = hiera_array ('apacheModules', [])
> hiera_include ('classes')
> install-packages { $packages: }
> apache24::modules { $apacheModules: }

Works like a charm btw.

[jcbollinger]

On Tuesday, June 7, 2016 at 8:02:24 AM UTC-5, Helmut Schneider wrote:

jcbollinger wrote:

Without the case statement, how can I make sure that ubuntu only
receives classes for ubuntu and not e.g. for Windows then?

That is where your data hierarchy comes in.  You have a level named "ubuntu-common"; it therefore stands to reason that if you have any Windows machines under management then there is also a "windows-common" (even if it's only notional).  These would be *alternatives* for a particular level of your hierarchy, and you would select which to use based on node facts.  That's precisely what you presently do via the 'case' statement, but you can, and probably should, do it directly in your hierarchy definition.  This is one of the primary uses of Hiera interpolation tokens.  The Hiera docs for defining a hierarchy discuss this and provide an example.  (I've linked to the Hiera 3.1 docs, but substantially the same thing applies to older Hiera as well.)

John

[Helmut Schneider]

jcbollinger wrote:

>
> On Tuesday, June 7, 2016 at 8:02:24 AM UTC-5, Helmut Schneider wrote:
> >
> > jcbollinger wrote:
> >
> >
> Without the case statement, how can I make sure that ubuntu only
> > receives classes for ubuntu and not e.g. for Windows then?
> >
> >
>
> That is where your data hierarchy comes in. You have a level named
> "ubuntu-common"; it therefore stands to reason that if you have any
> Windows machines under management then there is also a
> "windows-common" (even if it's only notional). These would be

> alternatives for a particular level of your hierarchy, and you would

> select which to use based on node facts. That's precisely what you
> presently do via the 'case' statement, but you can, and probably
> should, do it directly in your hierarchy definition. This is one of
> the primary uses of Hiera interpolation tokens. The Hiera docs for
> defining a hierarchy

> <https://docs.puppet.com/hiera/3.1/hierarchy.html> discuss this and

> provide an example. (I've linked to the Hiera 3.1 docs, but
> substantially the same thing applies to older Hiera as well.)

I changed my config accordingly:

:hierarchy:
- "nodes/%{::fqdn}"
- "%{::operatingsystem}"
- "%{::osfamily}"
- common

Thank you!