Hiera question: Nested hash

[Sans]

In my module, I have something like this to installed some base packages:

package {
    'rpm-mon-ntp':
    ensure   => latest,
    require  => Yumrepo[ "mon-repo-test" ];

    'rpm-mon-utils':
    ensure   => installed,
    require  => Yumrepo[ "mon-utils-test" ];
   
   'rpm-mon-patch':
    ensure   => installed,
    require  => [
                 Yumrepo[ "mon-repo-test" ],
                 Package[ 'mon-tools', 'patch-utils' ],
                ];
}

I want to use hiera to supply the values - repo-name, rpm-name, version etc. Is it possible at all?
I suppose, in the .yaml I can have something like this (in {repo-name{rpm-name: version}} format) :

rpm_mon_base_pack:
    mon-repo-test:
        rpm-mon-ntp: latest
        rpm-mon-patch: installed

    mon-utils-test:
        rpm-mon-utils: latest

but then, how do I do to make the package{} to get the values from hiera? How do I handle the case when Package[] is also required? Any help/pointer?

Best!

[jcbollinger]

You can use the create_resources() function to declare resources based on hashes similar in form to those you present.  You cannot, however, express resource references in hiera data, and you need resource references to express your resource relationships.  That can be worked around.  In this case, you also structure your data in a form that is not particularly conducive to the lookups you want to perform, but that, too can be worked around.

As is often the case in Puppet DSL, a suitable defined type can serve here as an adaptor, and suitable templates can do the needed processing.  I might do something like this:

define site::hiera_package($package_pack) {
  $repo = inline_template("<%= @package_pack.values.each { |repo| if repo.has_key?(@name) { repo; break } } %>")
  if $repo == '' {
    fail("No package '@name' defined in the provided pack")
  }
  package { $name:
    ensure => $package_pack[$repo][$name]
    require => Yumrepo[$repo]
  }
}

That will work with the data structure you presented, but will not provide relationships on other Packages.  However, since you are assuming the Yum package provider, you should not need to define package relationships to Puppet.  These should already be adequately described in your packages' internal metadata, which Yum uses to install any needed dependencies automatically.

If you nevertheless want data-driven explicit dependencies between packages, then you can probably obtain that.  One approach would involve making the values of the inner hashes be a third level of hash wherein you describe possibly-multiple properties of the package named by the key.  One of those properties would be the state to ensure, and another could be an array of the names of packages it should depend on.  You could then use an additional define to process the array and declare appropriate resource relationships via a chain operator.  Alternatively, the iteration facilities of the Future parser might provide a more direct route to expressing the needed chain operations.


John

[Sans]

Thanks John for showing me a way to do it. I haven't tried that yet but meanwhile, I was also trying the "define type" but different way. Do you think this is a better data structure?

rpm_mon_base_pack:
    base-pkg1:
        rep: mon-repo-test
        rpm: rpm-mon-ntp
        ver: latest
    base-pkg2:     
        rep: mon-utils-test:
        rpm: rpm-mon-utils
        ver: latest

then, in my .pp:

class packages::rpm::monitoring-utils {

    $rpm_pkgs_list = hiera_array('rpm_mon_base_pack', undef) 

    define monitoring_utils_pkgs() {

        $i_rpm = $rpm_pkgs_list[$name]
        $u_rpm = inline_template("<%= @i_rpm['rpm'].to_s.chomp %>")
        $u_ver = inline_template("<%= @i_rpm['ver'].to_s.chomp %>")
        $u_rep = inline_template("<%= @i_rpm['rep'].to_s.chomp %>")
   
        package { "$u_rpm":
            ensure   => $u_ver,
            require  => [
                          Yumrepo[ "$u_rep" ],

                          Package[ 'mon-tools', 'patch-utils' ],
                        ];
        }
    }

    monitoring_utils_pkgs { $rpm_pkgs_list: }
}

But I end up with the error:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Resource title must be a String, not Hash

 What's doing wrong here? I'll try your suggestion to night as well. Best!

[jcbollinger]

On Monday, May 5, 2014 12:44:59 PM UTC-5, Sans wrote:

Thanks John for showing me a way to do it. I haven't tried that yet but meanwhile, I was also trying the "define type" but different way. Do you think this is a better data structure?

rpm_mon_base_pack:
    base-pkg1:
        rep: mon-repo-test
        rpm: rpm-mon-ntp
        ver: latest
    base-pkg2:     
        rep: mon-utils-test:
        rpm: rpm-mon-utils
        ver: latest

Yes, I think that's a better data structure.  The import criterion here is that the desired per-package data be selectable directly, without scanning the whole top-level hash.

 

then, in my .pp:

class packages::rpm::monitoring-utils {

    $rpm_pkgs_list = hiera_array('rpm_mon_base_pack', undef) 

Using hiera_array() does ensure that the value retrieved is an array, but it's not at all what you want for this job or for either data structure.  In this case, the returned array will contain one element -- the same hash that would be returned by the ordinary hiera() function.  If the 'rpm_mon_base_pack' key appeared at multiple levels of your hierarchy, then the array returned by hiera_array() would contain the values from the other levels, too -- that's the main purpose of the funciton.

 

    define monitoring_utils_pkgs() {

        $i_rpm = $rpm_pkgs_list[$name]
        $u_rpm = inline_template("<%= @i_rpm['rpm'].to_s.chomp %>")
        $u_ver = inline_template("<%= @i_rpm['ver'].to_s.chomp %>")
        $u_rep = inline_template("<%= @i_rpm['rep'].to_s.chomp %>")
   
        package { "$u_rpm":
            ensure   => $u_ver,
            require  => [
                          Yumrepo[ "$u_rep" ],
                          Package[ 'mon-tools', 'patch-utils' ],
                        ];
        }
    }
    monitoring_utils_pkgs { $rpm_pkgs_list: }
}

But I end up with the error:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Resource title must be a String, not Hash

 What's doing wrong here? I'll try your suggestion to night as well. Best!

The error message pretty much says it.  Resource titles must be strings, but the one element of $rpm_pkgs_list is a hash. Under slightly different circumstances, the hash might have been automatically flattened to a string, which would fail a bit more interestingly.  An approach along those lines could be made to work, however, by making two main changes:

1. Look up the data via hiera() rather than hiera_array(), so that the hash is not needlessly wrapped in an array.
2. Use use just the keys of the hash as your resource titles.  You can extract an array of the keys via the stdlib's keys() function, and that's exactly what you would want to use in your declaration:

$package_names = keys($rpm_pkgs_list)
monitoring_utils_pkgs { $package_names: }


John

[Sans]

Hi John,

I haven't use create_resources() before, so following the puppetlab docs. After having the data structure in my default.yaml, I have this  in the nodes.pp:

$hr_mon_base_pkgs = hiera_hash('rpm_mon_base_pack', undef)

and then in my utils.pp, I have this:

class monitoring::mon-utils {

    define mon_pkgs($pkgs_pack) {
        $repo = inline_template("<%= @pkgs_pack.values.each { |repo| if repo.has_key?(@name) { repo; break } } %>")

        if $repo == '' {
            fail("No package '@name' defined in the provided pack")
        }
        package { $name:

            ensure   => $pkgs_pack[$repo][$name],
            require  => Yumrepo[ $repo ],
        }
    }
    create_resources(  monitoring::mon-utils::mon_pkgs, $hr_mon_base_pkgs )
}

but ends up with the error:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter rpm-mon-ntp on node mon105.back.local
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

which is one of the packages in the hiera hash. Am I doing it right at all??

Best,
San

[Sans]

On Tuesday, May 6, 2014 3:36:13 PM UTC+1, jcbollinger wrote:

Using hiera_array() does ensure that the value retrieved is an array, but it's not at all what you want for this job or for either data structure.  In this case, the returned array will contain one element -- the same hash that would be returned by the ordinary hiera() function.  If the 'rpm_mon_base_pack' key appeared at multiple levels of your hierarchy, then the array returned by hiera_array() would contain the values from the other levels, too -- that's the main purpose of the funciton.

Just noticed the typo: I was actually using hiera_hash() in stead of hiera_array(). This is for reason - I have four environments and the idea is override the default values from the environment specific yaml. Using :merge_behavior: deeper to merge  the values levels.

Best,
San