"Best" way to relate keys, or stuff multiple values into a single key in hiera?

[Wolf Noble]

So… I'm coming across a situation where I foresee myself having a collection of related k/v pairs in hiera. I want to create a thing… lets say a yumrepo for example's sake, but this same idea translates to a vhost, a web app configuration , a virtual domain, etc.

lets assume a hierarchy of:

- %{region}/%{dc}/%{env}/%{function}/%{fqdn}
- %{region}/%{dc}/%{env}/%{function}/common
- %{region}/%{dc}/%{env}/common
- %{region}/%{dc}/common
- %{region}/common

in the us-east region common yaml file, should I create the following two repos values:

repo_descr: eastcoast-pub
repo_baseurl: http://myfoo.com/eastcoastgoodtimes
repo_gpgkey: http://myfoo.com/eastcoastgoodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0

repo_descr: eastcoast-internal
repo_baseurl: http://myfoo.com/eastcoast
repo_gpgkey: http://myfoo.com/eastcoast/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0


in the foo DC common file I have a repo for that datacenter:

repo_descr: foo
repo_baseurl: http://myfoo.com/goodtimes
repo_gpgkey: http://myfoo.com/goodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0



in the 'qa' env common yaml file

repo_descr: fooqarepo
repo_baseurl: http://myfoo.com/qa_crispybacon/goodtimes
repo_gpgkey: http://myfoo.com/qa_crispybacon/goodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0



in the webservers function common yaml file, I have another repo:

repo_descr: webservers
repo_baseurl: http://myfoo.com/bacon/goodtimes
repo_gpgkey: http://myfoo.com/bacon/goodtimes/key
repo_gpgcheck: 1
repo_enabled: 1
repo_enablegroups: 0






so I want all the qa web servers in the foo DC in the east coast to have all five of those repo files defined.. except for web02, of course, because he's special… looking at it this way makes my head hurt. I don't see a good way to actualize these repos onto a node, or exclude them from a unique snowflake if necessary, which sort-of defeats the purpose of hiera altogether.

Do I not create them as separate keys, but as a single 'repo' array, and in my manifest use a template to iterate through the array and associate the values contained within the array with values needed for the addition of a repo?

What have people found works "best"?





________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.

[jcbollinger]

For complex data you need to put some structure in your YAML and process it accordingly in your manifests.  In particular, in this case it would be to your advantage to use hiera's (and YAML's) support for arrays and / or hashes as values.  For example:

in us-east/common.yaml:
======
region_repos:
  eastcoast_pub_repo: {
      descr: eastcoast-pub,
      baseurl: http://myfoo.com/eastcoastgoodtimes,
      gpgkey: http://myfoo.com/eastcoastgoodtimes/key,
      gpgcheck: 1,
      enabled: 1,
      enablegroups: 0
    },
  eastcoast_internal_repo: {
      descr: eastcoast-internal,
      baseurl: http://myfoo.com/eastcoast,
      gpgkey: http://myfoo.com/eastcoast/key,
      gpgcheck: 1,
      enabled: 1,
      enablegroups: 0
    }
======

That's a hash of hashes, associated with the key "region_repos".  You could use that in your manifests like so:

mymodule::repos {
    create_resources('yumrepo', hiera('region_repos'))
    # perhaps also create_resources('yumrepo', hiera('dc_repos')), etc.
}

If that looks simple, that's because I tailored the YAML data structure to the expectations of the create_resources() function.

It's also because the structure assumes that you will explicitly load the data for each hierarchy level you want, instead of, for example, using hiera_array() or hiera_hash() to assemble an even more complex aggregate data structure from various levels of your hierarchy.  I think that's reasonable for your use case, especially given that you anticipate certain machines will need exceptions.  You're still getting indirection out of it because hiera is choosing the appropriate region-, dc-, and environment-specific data files.

There are many other ways to do it, but that one has the advantage of being simple (relatively speaking) and fairly easy to understand.


John

[Wolf Noble]

Hi John,

Thanks a bunch. This seems incredibly sensible.

If I'm understanding your suggestion properly, I interpret the best thing to do as something like:


mymodule::repos {
$region_repos = hiera('region_repos' , '' )
$dc_repos = hiera('dc_repos' , '' )

if $region_repos {
create_resources('yumrepo', $region_repos))
}
if $dc_repos {
create_resources('yumrepo', $dc_repos))
}
#...
}

and then I could override them like

us-east/foo/qa/webservers/web02.qa.myfoo.com.yaml

region_repos: ""
dc_repos:
foodc_qa_repo: {
descr: foodc-qa,
baseurl: http://myfoo.com/foodc-repo,
gpgkey: http://myfoo.com/foodcrepo/key,

gpgcheck: 1,
enabled: 1,
enablegroups: 0
}

with the known caveat that I'd have to explicitly redeclare the entire hash at the most relevant tier if I wanted to change any element of the hash already defined.


Did I miss anything?

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/Rug-iffTkLkJ.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[jcbollinger]

I don't see any obvious errors or omissions.  In particular, you are correct that overriding repos declared at a lower level will involve declaring an entirely new hash (of hashes) at the appropriate higher level.  That's one of the costs attending complex data.  All in all, it looks nicely fleshed out.


John

[Wolf Noble]

Awesome!

I wonder about one other thing though, Is create_resources likely to balk if there are values in the hash that are irrelevant for the resource type being fed?

ie: if I have a region_repos hash that has both yum values, and apt values… (assuming that a debian sibling to the yumrepo type comes along at some point in the future) and feed them both via the same key, is that likely going to cause me pain? or should the key be geared more to one or the other, i.e.:

region_yumrepo:
foo_repo: {
..
}

region_aptsource
foo_aptsource: {
…
}

just trying to think about where the pointy/sharp parts of this might be.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/Ts3RcypLci0J.

[jcbollinger]

On Wednesday, June 13, 2012 10:21:19 AM UTC-5, Wolf Noble wrote:

I wonder about one other thing though,  Is create_resources likely to balk if there are values in the hash that are irrelevant for the resource type being fed?

 

I don't know off-hand, but I wouldn't be at all surprised if create_resources() fails when its input contains mappings for properties that the target resource type does not actually have.  If you put such additional data into your repository structure then you'll probably need to process the result to select the desired portions before feeding it to create_resources().  You might need to write a custom function for that purpose.  Alternatively, you could associate data for different resource types with different hiera keys.


John