Overriding specific values within a Hiera hash

[Matthew Burgess]

Hi all,

This is probably a real newbie question, but I'm having trouble overriding a value in Hiera that is defined in my common.yaml in a more specific role-based yaml file:

common.yaml -

lvm::volume_groups:
  rootvg:
    physical_volumes:
      - /dev/sda3
    logical_volumes:
      root_vol:
        size: 1G
        mountpath: '/'
        mountpath_require: true
      var_vol:
        size: 4G
        mountpath: '/var'
        mountpath_require: true
      ...

The following works, in that the var_vol volume is resized to 18G, but it ends up overriding the entire hash (and therefore all the other LV's defined in common.yaml are no longer managed).

web.yaml -

lvm::volume_groups:
  rootvg:
    physical_volumes:
      - /dev/sda3
    logical_volumes:
      var_vol:
        size: 18G
        mountpath: '/var'
        mountpath_require: true

Is it possible to only override the size parameter of the var_vol entry?

Thanks,

Matt

[Andreas Ntaflos]

On 2014-05-02 23:57, Matthew Burgess wrote:
> Hi all,
>
> This is probably a real newbie question, but I'm having trouble
> overriding a value in Hiera that is defined in my common.yaml in a more
> specific role-based yaml file:
>

[...]

>
> Is it possible to only override the size parameter of the var_vol entry?

Sound like you want to install deep-merge (packaged by Puppetlabs for
Debian as "ruby-deep-merge" and for RedHat as "rubygem-deep-merge") on
the Puppet master, set ":merge_behavior: deeper" in
/etc/puppet/hiera.yaml (and/or /etc/hiera.yaml) and restart the Puppet
master.

You can find documentation about Hiera merge types and behaviour here:
http://docs.puppetlabs.com/hiera/1/lookup_types.html#hash-merge

HTH Andreas

[Matthew Burgess]

On 2 May 2014 23:04, Andreas Ntaflos <da...@pseudoterminal.org> wrote:

Sound like you want to install deep-merge (packaged by Puppetlabs for
Debian as "ruby-deep-merge" and for RedHat as "rubygem-deep-merge") on
the Puppet master, set ":merge_behavior: deeper" in
/etc/puppet/hiera.yaml (and/or /etc/hiera.yaml) and restart the Puppet
master.

​Hmm, I already have that configuration setup (Hiera 1.2.1 with deepmerge installed and 'deeper' specified in hiera.yaml). Other Hiera hashes are getting merged correctly, although they never try to override an element of the hash, they simply define additional elements of the hash.  Indeed that works with the example above; if I specify something like an 'app_vol' LV, which doesn't exist in the common.yaml has, the server ends up with all of the common.yaml entries + the extra app_vol.  This failing case is where I have the var_vol LV defined in *both* levels of the hierarchy, at which point Hiera decides to replace the entire hash with the highest priority version of the hash instead of recursively merging the values defined therein.

Just googling a bit leads me to a really small nuance in the text in the link you posted.  The merge behaviour described on that page is for 'hash merge lookups' and not every Hiera lookup is that.  In particular, automated class parameter binding uses hiera() not hiera_hash() so doesn't do a hash merge lookup.  If I've read the code in lvm/init.pp correctly, that appears to be the reason for what I'm seeing.

So, I guess there's 3 questions now:

1) Have I understood the code and documentation correctly and come to the correct conclusion about why I'm seeing the behaviour I am?

2) Is there a reason why the automatic class parameter binding uses hiera() and not hiera_hash() when the parameter is a hash?

3) Is there anything I can do to 'fix' this particular issue? I don't mind carrying a local patch to puppetlabs/lvm if that's what's required, or a config change.  If not, I'll just have to resign myself to having data duplication; the number of places we have to override the size of a common filesystem is very low, as are the chances that the common filesystem sizes will change in common.yaml.

Thanks,

Matt.

[Andreas Ntaflos]

On 2014-05-03 00:40, Matthew Burgess wrote:
> On 2 May 2014 23:04, Andreas Ntaflos <da...@pseudoterminal.org

> <mailto:da...@pseudoterminal.org>> wrote:
>
> Sound like you want to install deep-merge (packaged by Puppetlabs for
> Debian as "ruby-deep-merge" and for RedHat as "rubygem-deep-merge") on
> the Puppet master, set ":merge_behavior: deeper" in
> /etc/puppet/hiera.yaml (and/or /etc/hiera.yaml) and restart the Puppet
> master.
>

> Just googling a bit leads me to a really small nuance in the text in the
> link you posted. The merge behaviour described on that page is for
> 'hash merge lookups' and not every Hiera lookup is that. In particular,
> automated class parameter binding uses hiera() not hiera_hash() so
> doesn't do a hash merge lookup. If I've read the code in lvm/init.pp
> correctly, that appears to be the reason for what I'm seeing.

Right, hiera() does priority lookups, and does not merge data.
hiera_hash() and hiera_array() can and will merge data if
":merge_behavior" is set to "deeper" and the deep-merge Gem is
installed. I think the documentation of Hiera
(http://docs.puppetlabs.com/hiera/1/index.html) covers these differences
and nuances nicely.

> So, I guess there's 3 questions now:
>
> 1) Have I understood the code and documentation correctly and come to
> the correct conclusion about why I'm seeing the behaviour I am?

I believe so, yes.

> 2) Is there a reason why the automatic class parameter binding uses
> hiera() and not hiera_hash() when the parameter is a hash?

This is by design
(http://docs.puppetlabs.com/hiera/1/puppet.html#limitations), but I
don't know much about the reasons behind it.

> 3) Is there anything I can do to 'fix' this particular issue? I don't
> mind carrying a local patch to puppetlabs/lvm if that's what's required,
> or a config change. If not, I'll just have to resign myself to having
> data duplication; the number of places we have to override the size of a
> common filesystem is very low, as are the chances that the common
> filesystem sizes will change in common.yaml.

This is usually a case for a profile class that calls hiera_hash() and
create_resources(). Here's one way to go about it (note that I don't
know or use the puppetlabs-lvm module myself, and the following is
untested):

# /etc/puppet/environment/production/modules/profile/manifests/lvm.pp
class profile::lvm {
include '::lvm'

$physical_volumes = hiera_hash('profile::lvm::physical_volumes', {})
create_resources('physical_volume', $physical_volumes)

$volume_groups = hiera_hash('profile::lvm::volume_groups', {})
create_resources('volume_group', $volume_groups)

$logical_volume = hiera_hash('profile::lvm::logical_volumes', {})
create_resources('logical_volume', $logical_volumes)
}

# /etc/puppet/hieradata/common.yaml
---
profile::lvm::physical_volumes:
'/dev/sda3':
ensure: present

profile::lvm::volume_groups:
'rootvg':
ensure: present
physical_volume: /dev/sda3

profile::lvm::logical_volumes:
root_vol:
size: 1G
volume_group: rootvg

mountpath: '/'
mountpath_require: true
var_vol:
size: 4G

volume_group: rootvg
mountpath: '/var'
mountpath_require: true

# /etc/puppet/hieradata/web.yaml
---
profile::lvm::logical_volumes:
var_vol:
size: 18G

The above defines a profile::lvm class which you can include on nodes
where you manage LVM. It uses hiera_hash() to look up hashes of data
that can be fed into the puppetlabs-lvm types physical_volume,
volume_group and logical_volume, and calls create_resources() on these
types with that data. See
http://docs.puppetlabs.com/references/latest/function.html#createresources
for details on create_resources().

In common.yaml we defined that data in the form of YAML hashes under the
lookup keys the profile::lvm class expects. In web.yaml we override the
size parameter for the var_vol LV, so a node that includes profile::lvm
and gets its data from the hierarchy level on which web.yaml resides
will have an 18GB LV named "var_vol" in the VG "rootvg", mounted under /var.

A more complete, practical and real-world example of roles and profiles
with Hiera can be found here:
<https://ask.puppetlabs.com/question/1655/an-end-to-end-roleprofile-example-using-hiera/>.
It uses the same concepts as the profile::lvm class above.

Andreas

[jcbollinger]

On Friday, May 2, 2014 5:40:24 PM UTC-5, Matthew Burgess wrote:

On 2 May 2014 23:04, Andreas Ntaflos <da...@pseudoterminal.org> wrote:

Sound like you want to install deep-merge (packaged by Puppetlabs for
Debian as "ruby-deep-merge" and for RedHat as "rubygem-deep-merge") on
the Puppet master, set ":merge_behavior: deeper" in
/etc/puppet/hiera.yaml (and/or /etc/hiera.yaml) and restart the Puppet
master.

​Hmm, I already have that configuration setup (Hiera 1.2.1 with deepmerge installed and 'deeper' specified in hiera.yaml). Other Hiera hashes are getting merged correctly, although they never try to override an element of the hash, they simply define additional elements of the hash.  Indeed that works with the example above; if I specify something like an 'app_vol' LV, which doesn't exist in the common.yaml has, the server ends up with all of the common.yaml entries + the extra app_vol.  This failing case is where I have the var_vol LV defined in *both* levels of the hierarchy, at which point Hiera decides to replace the entire hash with the highest priority version of the hash instead of recursively merging the values defined therein.

Just googling a bit leads me to a really small nuance in the text in the link you posted.  The merge behaviour described on that page is for 'hash merge lookups' and not every Hiera lookup is that.  In particular, automated class parameter binding uses hiera() not hiera_hash() so doesn't do a hash merge lookup.  If I've read the code in lvm/init.pp correctly, that appears to be the reason for what I'm seeing.

So, I guess there's 3 questions now:

1) Have I understood the code and documentation correctly and come to the correct conclusion about why I'm seeing the behaviour I am?

Yes, class parameter binding proceeds via hiera() in all cases.  Hash merge behavior is obtained only by explicitly calling hiera_hash().

 

2) Is there a reason why the automatic class parameter binding uses hiera() and not hiera_hash() when the parameter is a hash?

Puppet doesn't know the value's type until it looks it up.  Furthermore, it does not follow from the value being a hash that hash-merge is the desired lookup flavor.  Generally, that doesn't even follow from details of the class in question -- it is strictly a data definition matter.

 

3) Is there anything I can do to 'fix' this particular issue? I don't mind carrying a local patch to puppetlabs/lvm if that's what's required, or a config change.  If not, I'll just have to resign myself to having data duplication; the number of places we have to override the size of a common filesystem is very low, as are the chances that the common filesystem sizes will change in common.yaml.

I think you could achieve this without modifying your data by turning on 'deeper' merging and patching class lvm in lvm/manifests/init.pp like so:

class lvm {
  $volume_groups = hiera_hash('lvm::volume_groups', {})
  validate_hash($volume_groups)

  create_resources('lvm::volume_group', $volume_groups)
}

Note that that ties you to drawing the volume group information from Hiera (whereas otherwise you could, in principle, specify it in DSL at the point where you declare class lvm).  That's in fact what you are doing, however, and indeed what you should be doing.


John