defaults for virtual defined resources from hiera.

[Steve Traylen]

Hi,

This follows on a bit from the previous thread 'trouble with hiera and puppet defines' [1]

Up to now I've had a large file of virtual resources and then enabled them on demand

on various services. The very standard.

@metric{'1234:

      one => 1

      two => [1,2]

}

@metric{'abcd':

      one => a, 

      two => [b,c]

}

and then somewhere realize potentially with an override <| title == 1234 |> { one => 2  }

metric is something like 

define metric ($one, $two) {

   # do stuff.

}

Attempting to move the data of virtual resources above into hiera now  and a few questions.

metric1234:

     one: 1

     two:

           -'1'

           -'2'

metricabcd:

     one: a

     two:

          -'b'

          -'c'

I accept  to have to use that hierarchy rather than the perhaps more obvious

metric:

   1234: ...

    abcd: ...

since hiera can only select on the first level as the key?, i.e hiera(metic[1234]) or something is not posible is it?

To  realize in a manifest the following works:

$h = hiera_hash('metric4102')

$g = { '4102' => $h }

create_resources('lemon::metric',$g)

however what I would like to do is get the defaults from hiera within the defined resource so it becomes 

like the following which is incorrect but hopefully explains it.

metric{4102: one=> 'override'}

where metric is defined as 

define metric (hiera("lemon${title}") {

  # do stuff.

}

I can understand why that does not make sense, the define is only going to evaluated once and badly.

Only option I can think of is to override the paramters inside the define if they are non-sensical

defaults.

define metric(one=>'UNSET', two=>'UNSET) {

  $h = hiera("lemon${title}")

  if $one == 'UNSET' {

      $one = $h['one']

  }

  if $two == 'UNSET' {

      $two = $['two']

  }

}

but this is it a bit bizarre.

I am looking for a better way for a defined resource to get its defaults dynamically from hiera?

The motivation is that I probably only realize 20 of a potential > 1000 resources 

on a particular host so dropping that yaml file data in a database backend to hiera would

make it easier.

I realize I can override with an earlier priority yaml file and hiera_hash but would like

the paramatized declaration to work also.

  Steve.

[1] https://groups.google.com/forum/?fromgroups#!topic/puppet-users/Ou9F3KdzqeE

[jcbollinger]

On May 14, 2:27 pm, Steve Traylen <steve.tray...@cern.ch> wrote:
> Hi,
> This follows on a bit from the previous thread 'trouble with hiera and
> puppet defines' [1]
>
> Up to now I've had a large file of virtual resources and then enabled them
> on demand
> on various services. The very standard.
>
> @metric{'1234:
>       one => 1
>       two => [1,2]}
>
> @metric{'abcd':
>       one => a,
>       two => [b,c]
>
> }
>
> and then somewhere realize potentially with an override <| title == 1234 |>
> { one => 2  }

Overriding parameter values at realization time is mildly evil. You
need at least to understand that doing this prevents you from
realizing the overridden resource in more than one place, which
otherwise is safe and sometimes useful.

> metric is something like
> define metric ($one, $two) {
>    # do stuff.
>
> }
>
> Attempting to move the data of virtual resources above into hiera now  and
> a few questions.
>
> metric1234:
>      one: 1
>      two:
>            -'1'
>            -'2'
>
> metricabcd:
>      one: a
>      two:
>           -'b'
>           -'c'
>
> I accept  to have to use that hierarchy rather than the perhaps more obvious
> metric:
>    1234: ...
>     abcd: ...
>
> since hiera can only select on the first level as the key?, i.e
> hiera(metic[1234]) or something is not posible is it?

Not exactly, but similar should be possible. Hiera can return hashes
as values, so you ought to be able to do something like

$metric_data = hiera('metric')
$metric_1234 = $metric_data['1234']

That sort of structure might also play well with Puppet's
create_resources() function, which I see you already know about.

> To  realize in a manifest the following works:
>
> $h = hiera_hash('metric4102')
> $g = { '4102' => $h }
> create_resources('lemon::metric',$g)

Ok. You need to know, if you don't already, that the hiera() function
will return a hash if that's the type of value associated with the
specified key. Hiera_hash() also returns a hash, but its main purpose
is to collect and merge data from your whole hierarchy, instead of
from just the highest-priority level where the key is found. I don't
see any evidence from your example that the latter is what you're
really after in that case.

> however what I would like to do is get the defaults from hiera within the
> defined resource so it becomes
> like the following which is incorrect but hopefully explains it.
>
> metric{4102: one=> 'override'}
>
> where metric is defined as
>
> define metric (hiera("lemon${title}") {
>   # do stuff.
>
> }
>
> I can understand why that does not make sense, the define is only going to
> evaluated once and badly.

What does not make sense is "default values" that vary with the
resource instance you are declaring. At least, that doesn't match up
with what Puppet means by that term.

> Only option I can think of is to override the paramters inside the define
> if they are non-sensical
> defaults.
>
> define metric(one=>'UNSET', two=>'UNSET) {
>
>   $h = hiera("lemon${title}")
>   if $one == 'UNSET' {
>       $one = $h['one']
>   }
>   if $two == 'UNSET' {
>       $two = $['two']
>   }
>
> }
>
> but this is it a bit bizarre.

And it wouldn't work as written, since you cannot redefine variables
in Puppet, but something similar could be done. You could do
something a little cleaner and clearer with the aid of a couple of
functions. For example,

define metric($one => 'UNSET', $two => 'UNSET') {
$passed_params = remove_value({ 'one' => $one, 'two' => $two },
'UNSET')
$standard_params = hiera("metric_${title}")
$resolved_params = merge($standard_params, $passed_params)
# do something with $resolved_params['one']
# do something with $resolved_params['two']
}

There is just such a merge() function in the Puppetlabs stdlib
module. You would have to write remove_values() yourself, but it
ought to be pretty simple.

> I am looking for a better way for a defined resource to get its defaults
> dynamically from hiera?

If the "default" values were associated with top-level keys in your
data store, then perhaps you could do something like this:

define metric($one => hiera("metric${title}_one"), $two =>
hiera("metric${title}_one")) {
# ...
}

Again, however, part of the problem is that you are trying to use
Puppet's default value facility in a way that was not intended.
Generally speaking, default values are defaults for the *type*, not
separate defaults for each instance.

> The motivation is that I probably only realize 20 of a potential > 1000
> resources
> on a particular host so dropping that yaml file data in a database backend
> to hiera would
> make it easier.
>
> I realize I can override with an earlier priority yaml file and hiera_hash
> but would like
> the paramatized declaration to work also.

I think you're creating a mess for yourself, especially with parameter
overrides at realization time. You would be better off moving all the
way to hiera. In fact, you could leverage hiera further to pull those
1000 virtual resource declarations out of your manifests altogether.
For example:

class metric::instances {
# Retrieve an array of the names of the metric
# instances wanted for this node
$metrics = hiera('metrics')

# Declare all the wanted 'metric' instances (concretely).
# The definition relies on hiera for all needed data,
# likely by merging data from two or more levels via
# hiera_hash()
metric::metric { $metrics: }
}

Isn't that better than thousands of lines of virtual resource
declarations? It will be a lot lighter on your puppetmaster, too.


John

[Steve Traylen]

On Tuesday, 15 May 2012 16:48:48 UTC+2, jcbollinger wrote:

On May 14, 2:27 pm, Steve Traylen <steve.tray...@cern.ch> wrote:
> Up to now I've had a large file of virtual resources and then enabled them
> on demand
> on various services. The very standard.
>
> @metric{'1234:
>       one => 1
>       two => [1,2]}
>
> @metric{'abcd':
>       one => a,
>       two => [b,c]
> }
> and then somewhere realize potentially with an override <| title == 1234 |>
> { one => 2  }

Overriding parameter values at realization time is mildly evil.  You
need at least to understand that doing this prevents you from
realizing the overridden resource in more than one place, which
otherwise is safe and sometimes useful.

Agreed, i need to set paramaters at creation time, the parameters just may

differ in place to place at creation time sometimes.

Okay, I guess i am just using hiera as a data store because its easy to access that

and let it deal with the complexity.

What I went for in end which was a merge of some of your comments was

define metric( $param => undef ) {

    $metricid = $title

    $metrichash = hiera("lemon_${metricid}")

    # Set $my_param to first passed parameter, then hiera value then default value.

    if ( $param != undef ) 

      $my_param = $param 

    } elsif ( has_key($metrichas,'param') 

       $my_param = $metrichash['hash']

    } else {

       $my_param = undef

    }

    

   And then use $my_param in the erb template.       

 

> The motivation is that I probably only realize 20 of a potential > 1000
> resources
> on a particular host so dropping that yaml file data in a database backend
> to hiera would
> make it easier.
>
> I realize I can override with an earlier priority yaml file and hiera_hash
> but would like
> the paramatized declaration to work also.


I think you're creating a mess for yourself, especially with parameter
overrides at realization time.  You would be better off moving all the
way to hiera.  In fact, you could leverage hiera further to pull those
1000 virtual resource declarations out of your manifests altogether.
For example:

I now have no virtual resources, I just load the bits of data from hiera as and when needed.

Thanks for the input , it helped me a lot with my understanding of the finer points.

I am at the moment fairly happy with the result simply because it allows one person to maintain

the library of metrics in YAML and other people to enable what they want. .. And they can still

do override of the data which is especially useful when you want to override based

on a fact for instance which can only be set in a manifest.

For reference: https://github.com/CERN-Ops/puppet-lemon