Accessing other parts of the Hiera tree

[Robin Lee Powell]

This seems to come up for me a lot. As an example, my Hiera data
includes both hypervisors and the VMs that they contain. It would
be very useful to have the VMs say "I'm a VM on host X" and, in
templates in host X, be able to say "What are all my VMs?".

Doing this via storeconfigs rather defeats the purpose of my Hiera
tree, which is to be the single source of truth.

The way I'm currently doing it is to have a script that collates all
my Hiera host data into a special YAML file, and then have templates
read that YAML.

I'm wondering if there's a better/more correct/more idiomatic way to
engage in this sort of referential linking across the Hiera tree?

-Robin

--
http://intelligence.org/ : Our last, best hope for a fantastic future.
.i ko na cpedu lo nu stidi vau loi jbopre .i danfu lu na go'i li'u .e
lu go'i li'u .i ji'a go'i lu na'e go'i li'u .e lu go'i na'i li'u .e
lu no'e go'i li'u .e lu to'e go'i li'u .e lu lo mamta be do cu sofybakni li'u

[jcbollinger]

On Monday, June 17, 2013 12:28:18 AM UTC-5, Robin Powell wrote:

This seems to come up for me a lot.  As an example, my Hiera data
includes both hypervisors and the VMs that they contain.  It would
be very useful to have the VMs say "I'm a VM on host X" and, in
templates in host X, be able to say "What are all my VMs?".

It is inherently non-idiomatic for your templates (or manifests) to pose such a question, or really any question.  Rather than asking a question, the templates may need to say "my VMs are [...]".  I know I'm being nit-picky with the wording, but people really need to get a firm grip on the concept in order to master Puppet.

 

Doing this via storeconfigs rather defeats the purpose of my Hiera
tree, which is to be the single source of truth.

I think you're focusing on the wrong thing when you say that.  To the extent that it's applicable to the problem, storeconfigs is simply the infrastructure that makes exported resources work.  It is in no way an independent source of truth, because everything in it comes from declarations made in one manifest or another, based on node facts and possibly on data from some other source.  Put another way, in its application to problems of this kind, storeconfigs serves merely as an asynchronous communication channel.

Focusing on exported resources, then, these are simply a mechanism for making declarations during the compilation of one node's catalog that can later be incorporated by reference into any node's catalog.  Such declarations draw on whatever source(s) of truth you choose, and no others, just like any other declarations your manifests make.  Collecting exported resources later does not introduce any new sources of truth.
 

The way I'm currently doing it is to have a script that collates all
my Hiera host data into a special YAML file, and then have templates
read that YAML.

So you in fact don't have a single source of truth currently.  Not any more so than you would have with exported resources, at least.  Your collation script and special YAML file simply take the place of master-mediated resource export and the storeconfigs database, respectively.  Perhaps you have other reasons to want to avoid storeconfigs, but otherwise I cannot imagine why you think what you've come up with is superior to using exported resources.

 

I'm wondering if there's a better/more correct/more idiomatic way to
engage in this sort of referential linking across the Hiera tree?

It depends in part on whether you intend to extend your single-source-of-truth requirement to include DRY requirements on your hiera data.  One possibility would be to provide indices in your hiera data -- for example, for each VM host, an array of the names of its VMs.  That's not DRY, however, if it duplicates information in VMs' individual data files.

If you want all needed data stored in hiera, with no redundancy and no use of intermediate collation, then all the data need to be provided to (at least) all the associated nodes; that is, to the VM host and all its VMs.  They all need to get all the data, else there is no escaping a need for some kind of data duplication.

You might find it convenient and logical to structure it as one large, complex, nested value, from which the individual components would select the pieces they need.  For example, a hash with VM hostnames as keys, and hashes of VM names to VM parameter hashes as values (i.e. a trebly-nested hash).  Alternatively, there are ways you could spread out the data over multiple Hiera keys, but the overall logical structure is going to be similar.

It should be obvious at this point is that with this kind of data structure, you are not going to be able to use automated class parameter binding to assist in declaring most of the various interrelated bits and pieces.

Given those choices, I would probably opt for exported resources, myself.


John

[Robin Lee Powell]

[snipped heavily, hopefully nothing important]

On Mon, Jun 17, 2013 at 09:48:48AM -0700, jcbollinger wrote:
>
> On Monday, June 17, 2013 12:28:18 AM UTC-5, Robin Powell wrote:
> >
> > Doing this via storeconfigs rather defeats the purpose of my
> > Hiera tree, which is to be the single source of truth.
> >
>
>
> I think you're focusing on the wrong thing when you say that. To
> the extent that it's applicable to the problem, storeconfigs is
> simply the infrastructure that makes exported resources work. It
> is in no way an independent source of truth, because everything in
> it comes from declarations made in one manifest or another, based
> on node facts and possibly on data from some other source.

I didn't see the point in getting into this before, but you raise
valid issues, so:

That would be true, if facts automatically expired when not renewed,
but they don't. I've had *no end* of trouble with lies sticking
around because they were in storeconfigs. I don't mean to propose
that they *should* auto-expire, as that doesn't fit the model at
all, but it's still a very real problem.

> You might find it convenient and logical to structure it as one
> large, complex, nested value, from which the individual components
> would select the pieces they need. For example, a hash with VM
> hostnames as keys, and hashes of VM names to VM parameter hashes
> as values (i.e. a trebly-nested hash). Alternatively, there are
> ways you could spread out the data over multiple Hiera keys, but
> the overall logical structure is going to be similar.

*nodnod* OK, that seems sane, thanks.

> It should be obvious at this point is that with this kind of data
> structure, you are not going to be able to use automated class
> parameter binding to assist in declaring most of the various
> interrelated bits and pieces.

Enh; I've already gotten around that with my users setup by using
create_resources(...)

> Given those choices, I would probably opt for exported resources,
> myself.

It's been so bad for me. ;_; After the 4th time of "Why won't
nagios stop alerting on this host I turned off and removed from all
my configs!?! ... Oh.", I got really sick of it. The *main goal*
for me in using Hiera is to never use storeconfigs ever again.

-Robin

[Robin Lee Powell]

> > You might find it convenient and logical to structure it as one
> > large, complex, nested value, from which the individual
> > components would select the pieces they need. For example, a
> > hash with VM hostnames as keys, and hashes of VM names to VM
> > parameter hashes as values (i.e. a trebly-nested hash).
> > Alternatively, there are ways you could spread out the data over
> > multiple Hiera keys, but the overall logical structure is going
> > to be similar.
>
> *nodnod* OK, that seems sane, thanks.

Thanks again for the push there; it really helped.

FWIW, what I've got is actually a hybrid, combining a structured
giant hiera data tree with hierarchical class overrides; this seems
useful so I'll show y'all.

Here's a snippet of the hosts structure, which lives in hosts.yaml:

- ----------------

hosts:
dev01.[snip]:
# type, environment, and site are all used to *find* what we
# inherit from in hiera, and hence cannot themselves be inherited,
# or at least not in a reasonable fashion
fqdn: dev01.[snip]
type: cytoweb
type_version: 3
environment: dev
site: dev01
ipv6: false
ipv4: true
ipv4_ip: [snip]
openvzid: 6222
openvzhost: ds518
qa01.[snip]:
# type, environment, and site are all used to *find* what we
# inherit from in hiera, and hence cannot themselves be inherited,
# or at least not in a reasonable fashion
fqdn: qa01.[snip]
type: cytoweb
type_version: 2
environment: qa
site: qa01
ipv6: false
ipv4: true
ipv4_ip: [snip]
openvzid: 8589
openvzhost: ds988

- ----------------

Here's the relevant bit of site.pp; *all* other puppet decisions are
hiera driven:

- -------------

$hosts = hiera_hash('hosts')
$host_type = $hosts[$fqdn]['type']
$host_type_version = $hosts[$fqdn]['type_version']

# Make sure our type can be loaded
$type_test = hiera('type_loaded')
notify{ "sphtf": message => "site.pp: host type file: hiera/types/${::host_type}_v$host_type_version" }

$host_environment = $hosts[$fqdn]['environment']
$host_site = $hosts[$fqdn]['site']

node default {
notify{ "spht": message => "site.pp: host type: $::host_type" }
notify{ "sphtv": message => "site.pp: host type version: $::host_type_version" }

hiera_include('classes')
}

- -------------

So we grab some specific bits out of the host structure for the
current host, but then, and this is the fun part, we *turn them back
into hiera selectors*; this is my hiera.yaml:

- -------------

---
:backends:
- yaml
:yaml:
:datadir: /opt/puppet3/etc/hiera
:hierarchy:
- node/%{::fqdn}
- types/%{::host_type}_v%{::host_type_version}
- site/%{::host_site}
- environments/%{::host_environment}
- os/%{::operatingsystem}
- osfamily/%{::osfamily}
- hosts
- sites
- users
- common
# options are native, deep, deeper
:merge_behavior: deeper

- -------------

So this means that qa01.[snip] can have behaviour driven by
information in the global hosts structure, which can be accessed by
everybody for global things like generating an /etc/hosts file,
*and* it can have hiera/node/qa01.[snip] , which can have class
overrides like:

sudo::dev_sudo: true

or whatever.

Note in addition that with "deeper" merging setup, if you use
hiera_hash as I've done, you can actually override bits of the
structure.

We don't use this for the hosts structure, but we *do* use it for
users:

create_resources( users::user, hiera_hash('users::users') )

is how that structure gets used, and individual hosts can have
things like:

---
users::users:
stu:
ensure: present
until: 'Mon Jun 24 12:59:12 PDT 2013'
andrew:
ensure: present
until: 'Mon Jun 24 12:59:12 PDT 2013'

in their node/hostname.[snip] files, which will override those parts
of the users::user structure, in which those users default to
"enable: false". users::user, like hosts, is a big giant nested
hash.

Hopefully this all is of use to someone.

-Robin