Puppet array merge failure?

[leam hall]

Note that I'm still learning so this could easily be operator error. I'd like to get all 4 ntp servers in in the array. That is, I have other things that need to be in merged hashes and arrays and I'm using ntp_servers as a learning tool.

Thanks!

Leam

#####

hiera.yaml
--------------
:backends:
  - yaml
  - json
:hierarchy:
  - "node/%{fqdn}"
  - "%{osfamily}"
  - "%{clientcert}"
  - "%{environment}"
  - global
  - defaults

:yaml:
  :datadir: "/var/lib/hiera"
:json:
  :datadir: "/var/lib/hiera"

:merge_behavior: deep



Hiera tests on puppet master
-----------------------------------------

Note that remotenode.example.com is in the sandbox environment.

#  hiera -a --array ntp_servers fqdn=remotenode.example.com
["aquarius.example.com", "defiant.example.com"]


#  hiera  -a ntp_servers environment=sandbox
['ntp1.example.com", "ntp2.example.com"]

[Matthew Burgess]

On 30 Jan 2015 14:09, "leam hall" <leam...@gmail.com> wrote:
>
> :merge_behavior: deep

See https://docs.puppetlabs.com/hiera/1/lookup_types.html#deep-merging-in-hiera--120. In particular you probably want 'deeper' not 'deep' behavior.

Matt.

[leam hall]

Hey Matt, good catch. However, I changed to deep_merge in hiera.yaml, restarted the pe-puppet service, and am still getting the behavior. I think we have the deep_merge gem installed:

/opt/puppet/lib/ruby/gems/1.9.1/gems/deep_merge-1.0.0

Not sure if the gems just get picked up or have to be called specifically.

Thanks!

Leam

[Sans]

Did you check if have ruby-deep-merge (or the equivalent to your OS) installed?

[jcbollinger]

On Friday, January 30, 2015 at 11:33:37 AM UTC-6, leam hall wrote:

On Friday, January 30, 2015 at 12:11:20 PM UTC-5, Matthew Burgess wrote:

On 30 Jan 2015 14:09, "leam hall" <leam...@gmail.com> wrote:
>
> :merge_behavior: deep

See https://docs.puppetlabs.com/hiera/1/lookup_types.html#deep-merging-in-hiera--120. In particular you probably want 'deeper' not 'deep' behavior.

Matt.

Hey Matt, good catch. However, I changed to deep_merge in hiera.yaml [...]

I suppose you mean you changed to

:merge_behavior: deeper

If not, then you did not make the change Matthew suggested.

 

I think we have the deep_merge gem installed:

/opt/puppet/lib/ruby/gems/1.9.1/gems/deep_merge-1.0.0

Not sure if the gems just get picked up or have to be called specifically.

Since it seems you are using Puppet PE, the installer should have ensured you have all necessary gems, in the right places.

I suppose the relevant question at this point is why you expect different behavior than you see?  You did not present your data files, but I infer from what you did present that you have at least these:

/var/lib/hiera/node/remotenode.example.com.yaml
/var/lib/hiera/sandbox.yaml

You might have .json files instead of or in addition to the .yaml files, and you might have more files than those, such as a global.yaml and/or a defaults.yaml, but only the fqdn- and environment-specific data sources have values associated with key 'ntp_servers'.  The "node/remotenode.example.com" data source(s) provides the "defiant" and "aquarius" values, and the "sandbox" data source(s) provides the "ntp1" and "ntp2" values.

Now consider this lookup you report performing:

hiera -a --array ntp_servers fqdn=remotenode.example.com

Why do you give the "fqdn=..." argument?  Clearly you discovered, one way or another, that that is how you can present fact values to Hiera when you invoke it from the command line.  In this case, that allows Hiera to interpolate the given fact value into your hierarchy definition, so that it will consult your "node/%{fqdn}" data source(s) for that FQDN.

Similarly, with this command ...

hiera  -a ntp_servers environment=sandbox

... the "environment=sandbox" defines the "environment" fact to Hiera, so that it chooses the "sandbox" data source(s) in the "%{environment}" slot.

By this point I'm sure you see where I'm going: in each of those cases, you define only one fact to Hiera.  You do not in either case give it the information to choose both data sources you want it to use.  Thus, this is not an array merge failure, it is a data source specification failure.  I predict you would see the results you want with this:

hiera --array ntp_servers environment=sandbox fqdn=remotenode.example.com

There are several things to note here:

• The -a and --array options are synonymous.  You don't need to specify both.
• The native / deep / deeper merge behavior pertains to hash-merge lookups only.  You are performing array-merge lookups, so that option is irrelevant to your tests.
• Hiera does not consider it an error for any data source to be missing -- it simply skips any it cannot open.  That is intentional, and generally what you want when the data source is genuinely missing, but it can produce surprising results if the data source is present but inaccessible (e.g. because of ownership / permissions / mandatory access controls, etc.).
• When Puppet invokes Hiera via its Ruby API, all the Puppet variables that are in scope are available to Hiera for interpolation, including in particular all fact values reported by the target node.  You won't encounter an issue of the type that tripped you up in conjunction with Hiera usage by Puppet.

John

[leam hall]

Hey John, thanks!

To add to your points, I'm referencing
https://docs.puppetlabs.com/hiera/1/lookup_types.html#array-merge

Your recommended Hiera command works fine on the master. Thanks! All
NTP servers show up.

However, they are not getting put into the ntp.conf file. I've
modified both the sandbox and remotenode json files to add extra
servers. The new servers show up in a hiera call on the puppet master
but are not showing up in the remotenode ntp config file.

Back to trouble-shooting...

Leam

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/bB-wGAOQAHI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/360ff971-55e8-421e-b793-cd82c26050ad%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Mind on a Mission

[jcbollinger]

On Monday, February 2, 2015 at 10:19:33 AM UTC-6, leam hall wrote:

Hey John, thanks!

To add to your points, I'm referencing
https://docs.puppetlabs.com/hiera/1/lookup_types.html#array-merge

Your recommended Hiera command works fine on the master. Thanks! All
NTP servers show up.

However, they are not getting put into the ntp.conf file. I've
modified both the sandbox and remotenode json files to add extra
servers. The new servers show up in a hiera call on the puppet master
but are not showing up in the remotenode ntp config file.

Back to trouble-shooting...

At this point, it sounds like we can conclude that the Hiera data files are not your problem.  If you want further help troubleshooting then we'll need to see the resource declaration by which the target file is being managed.  If you are templating the file then the relevant part(s) of the template, too.


John

[leam hall]

We're using the PuppetLabs ntp module with minor changes.

environments/sandbox/modules/ntp/manifests:

|-- config.pp
|-- init.pp
|-- install.pp
|-- params.pp
`-- service.pp

In init.pp we're using servers in the class parameters:

$servers = $ntp::params::servers,

Didn't change the $servers validation:

validate_array($servers)


In params.pp we use a hiera call to build $servers:

$servers = hiera('ntp_servers')

and commented out the centos pool servers under the 'RedHat' $osfamily
case statement.

Is that what you were looking for?

Leam

[leam hall]

And on that note I moved the

$servers = hiera('ntp_servers')

to

$servers = hiera_array('ntp_servers')

and resolved the issue. Woo-hoo!

[jcbollinger]

On Thursday, February 5, 2015 at 10:30:52 AM UTC-6, leam hall wrote:

And on that note I moved the

  $servers           = hiera('ntp_servers')

to

  $servers           = hiera_array('ntp_servers')

and resolved the issue. Woo-hoo!

Indeed.  As you discovered, the hiera() function (and automated data binding) perform priority lookups.  If you want array- or hash-merge lookups then you need to use the appropriate function specific to that kind of lookup.  This is an acknowledged limitation of Hiera -- in practice, the appropriate type of lookup for a given key is usually a property of the data, but there is no built-in way to represent that alongside the data.  The data consumer somehow just has to know.  This is an especial problem for automated data binding, where you cannot even directly control the type of lookup performed.

Anyway, I'm glad you've got it sorted.  Yay!


John