Jira (PUP-8405) Knock-out-prefix fails with more than 3 hierarchy levels

Josh Cooper (Jira)

unread,

May 24, 2022, 8:11:03 PM5/24/22

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

PUP-8405

Knock-out-prefix fails with more than 3 hierarchy levels

Change By:	Josh Cooper
Summary:	Knock-out-prefix fails with more than 3 merges hierarchy levels

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)

Josh Cooper (Jira)

unread,

May 24, 2022, 8:12:01 PM5/24/22

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

PUP-8405

Knock-out-prefix fails with more than 3 hierarchy levels

Change By:	Josh Cooper

When defining multiple levels of hierarchy in hiera.yaml ( more than 3 or more ), and requesting hiera to lookup a key that is found on more than at least 3 levels, and lookup options for that key are configured to merge deep with a knockout prefix, the merge fails to knockout the key matching the higher precedence knockout.

Steps to reproduce:

{code:yaml}
hiera.yaml:
version: 5
defaults:
   datadir: data
   data_hash: yaml_data

hierarchy:
   - name: node specific data
     path: node/%{::site.node}.yaml

   - name: product data
     path: product/%{::site.product}.yaml

   - name: os data
     path: os/%{::os.family}.yaml

   - name: global data
     path: common.yaml

common.yaml:
  lookup_options:
    classes:
      merge:
        strategy: deep
        knockout_prefix: --
  classes:
    - profiles::motd
    - profiles::network

os/RedHat.yaml:
  classes:
    - profiles::yum

product/product.yaml
classes:
   - profiles::application

node/node.yaml:
classes:
   - --profiles::network
   - network
{code}

*Expected*

{code:java}
puppet lookup --knock-out-prefix '--' --merge deep classes
---
- profiles::motd
- profiles::yum
- profiles::application
- network
{code}

*Actual*

{code:java}
puppet lookup --knock-out-prefix '--' --merge deep classes
---
- profiles::motd
- profiles::network
- profiles::yum
- profiles::application
- network
{code}

Removing a layer from the hierarchy causes the merge to behave as expected. The difference being:

{code:java}
puppet lookup --knock-out-prefix '--' --merge deep classes | grep 'Found key:' | wc -l
4
{code}
vs
{code:java}
puppet lookup --knock-out-prefix '--' --merge deep classes | grep 'Found key:' | wc -l
3
{code}

*Verified Affected Versions*
Puppet 4.10.0
Puppet 4.10.9
Puppet 5.3.1

Add Comment

Josh Cooper (Jira)

unread,

May 24, 2022, 8:37:03 PM5/24/22

to puppe...@googlegroups.com

Josh Cooper commented on

PUP-8405

Re: Knock-out-prefix fails with more than 3 hierarchy levels

I updated the title and description to capture some small, but important details. This problem can be triggered if the lookup key appears in 3 (or more) places in the hierarchy(s). So if I have 3 levels for a.yaml, b.yaml and common.yaml, and the classes key appears in all of them:

a.yaml

---

classes:

  - --profiles::network

  - network

b.yaml

---

classes:

  - profiles::application

common.yaml

---

lookup_options:

  classes:

    merge:

      strategy: deep

      knockout_prefix: --

classes:

  - profiles::motd

  - profiles::network

Then there will be 2 merges. First a.yaml and b.yaml are merged:

classes merging ["--profiles::network", "network"] and ["profiles::application"]

Then that result is merged with common.yaml:

classes merging ["profiles::application", "network"] and ["profiles::motd", "profiles::network"]

The problem is as Thomas and Henrik described earlier. The anti-value "--profiles::network" is only applied during the first merge and is "lost" by the time the second merge takes place, so the value "profiles::network" is not removed/knocked out.

This problem can also occur across environment and module layers. However, it is only triggered if there are multiple merges (once in the environment layer and again across layers).

Also the issue won't occur if the key only occurs in two places in the hierarchy, since there's only one merge and the value and anti-value cancel each other.

Add Comment

Christopher Allen (Jira)

unread,

May 27, 2022, 8:28:03 AM5/27/22

to puppe...@googlegroups.com

Christopher Allen commented on

PUP-8405

Re: Knock-out-prefix fails with more than 3 hierarchy levels

In support to what is mention above (years ago) but with additions, I think that:

a) The current behaviour should not be changed

b) A new merge strategy should be introduced

This should allow backwards compatibility and a way forward for those that assume the knockouts occur after the merge of all lists have been completed.