Using regex to match hostnames in hiera

[Edwin Starkey]

Hi, I'm having a problem with extlookup not respecting the 'certname' parameter[1].  When executing a puppet run with either the --certname or --fqdn parameters, it ends up using the specified SSL certificate and gets the correct node definition applied from the puppetmaster.  However, it still retrieves extlookup data using the node's actual FQDN, not the one manually specified using the parameter.

Anyway, I thought this might be a bug in the extlookup code so I decided to try out hiera as an alternative.  It seems like a nice tool, but in my case there is a very big downside - the inability to match the fqdn based on a regular expression.  This was brought up in a previous thread[2].

As an example, here is my existing extlookup configuration from site.pp:

$hostgroup = regsubst($hostname, '-*\d+$', '')
$extlookup_datadir = "/etc/puppet/environments/${environment}/manifests/extdata"
$extlookup_precedence = [ 'hostnames/%{fqdn}', 'hostgroups/%{hostgroup}', 'common' ]

This layout is quite beautiful, as it has three possible matching scenarios.

1. A CSV file for a specific hostname exists.  Example: hostnames/web1.mydomain.com.csv applies to a single host - web1.mydomain.com.
2. A CSV file for a regular expression based on hostname exists.  Example: hostgroups/web.csv applies to all hosts whose hostname begins with 'web' - web1.mydomain.com or web999.mydomain.com.
3. No matches are found, so default values from common.csv are used.

I'd like to replicate this behavior using hiera.  Is it possible?

P.S.  The first reply to the aforementioned mailing list thread suggested creating a custom fact and using that to specify the node's hostgroup.  Please don't recommend that!  To begin with, such measures aren't necessary with extlookup.  Hiera is the shiny/new/better successor to extlookup, I shouldn't have to create a custom fact to reproduce the old functionality.  Secondly, I have another good reason but it is complicated and would take too long to explain.  Just take my word for it :-)

Thank you.

[1] https://projects.puppetlabs.com/issues/17198
[2] https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/aGFSQ2SYgL8

[R.I.Pienaar]

Create a simple custom fact that has the same logic as your $hostgroup
variable, use that in the hierarchy

> P.S.  The first reply to the aforementioned mailing list thread
> suggested creating a custom fact and using that to specify the
> node's hostgroup.  Please don't recommend that!  To begin with, such
> measures aren't necessary with extlookup.  Hiera is the
> shiny/new/better successor to extlookup, I shouldn't have to create
> a custom fact to reproduce the old functionality.  Secondly, I have
> another good reason but it is complicated and would take too long to
> explain.  Just take my word for it :-)

hiera can access any variable in its hierarchy, so set it however you
want, however facts are there for a reason to solve this exact problem.
you should use them.

[Calvin Walton]

On Thu, 2012-10-25 at 20:45 -0700, Edwin Starkey wrote:
> Hi, I'm having a problem with extlookup not respecting the 'certname'
> parameter[1]. When executing a puppet run with either the --certname
> or --fqdn parameters, it ends up using the specified SSL certificate
> and gets the correct node definition applied from the puppetmaster.
> However, it still retrieves extlookup data using the node's actual
> FQDN, not the one manually specified using the parameter.

> As an example, here is my existing extlookup configuration from
> site.pp:
>
> $hostgroup = regsubst($hostname, '-*\d+$', '')
> $extlookup_datadir = "/etc/puppet/environments/${environment}/manifests/extdata"
> $extlookup_precedence = [ 'hostnames/%{fqdn}', 'hostgroups/%{hostgroup}', 'common' ]

The certname value is exposed in the puppet manifests in the $clientcert
variable. Something like the following might work:

$hostgroup = regsubst($clientcert, '-*\d+$', '')

$extlookup_datadir = "/etc/puppet/environments/${environment}/manifests/extdata"

$extlookup_precedence = [ 'hostnames/%{clientcert}', 'hostgroups/%{hostgroup}', 'common' ]

> This layout is quite beautiful, as it has three possible matching
> scenarios.
>
> 1. A CSV file for a specific hostname exists. Example:
> hostnames/web1.mydomain.com.csv applies to a single host -
> web1.mydomain.com.
> 2. A CSV file for a regular expression based on hostname exists.
> Example: hostgroups/web.csv applies to all hosts whose hostname begins
> with 'web' - web1.mydomain.com or web999.mydomain.com.
> 3. No matches are found, so default values from common.csv are used.
>
> I'd like to replicate this behavior using hiera. Is it possible?

Absolutely. Hiera lookups can be done using arbitrary variables defined
in your puppet manifests. Something like the following should be
sufficient:

in site.pp toplevel:
$hostgroup = regsubst($clientcert, '-*\d+$', '')

in hiera.yaml:
:yaml:
:datadir: /etc/puppet/environments/${environment}/manifests/hiera
:hierarchy:
- hostnames/%{clientcert}
- hostgroups/%{hostgroup}
- common

Then you can use puppet3's automatic class parameter lookup, or call the
hiera() functions anywhere in your manifest.

--
Calvin Walton <calvin...@kepstin.ca>