Jira (PUP-8405) Knock-out-prefix fails with more than 3 merges

[Henrik Lindberg (JIRA)]

Henrik Lindberg moved an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges Change By: Henrik Lindberg Method Found: Needs Assessment Issue Type: Task Bug Key: HI PUP - 595 8405 Project: Hiera Puppet Add Comment

This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)

[Henrik Lindberg (JIRA)]

Henrik Lindberg assigned an issue to Thomas Hallgren

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: Henrik Lindberg Assignee: Thomas Hallgren

Add Comment

[Ryan Struber (JIRA)]

Ryan Struber updated an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: Ryan Struber When defining multiple levels of hierarchy in hiera.yaml (more than 3), and requesting hiera to lookup a key that is found on more than 3 levels, and lookup options for that key are configured to merge deep with a knockout prefix, the merge fails to knockout the key matching the higher precedence knockout. Steps to reproduce: {code:yaml} hiera.yaml:  version: 5  defaults:             datadir: data    data_hash: yaml_data  hierarchy:    - name: node specific data      path: node/%{::site.node}.yaml    - name: product data      path: product/%{::site.product}.yaml    - name: os data      path: os/%{::os.family}.yaml    - name: global data      path: common.yaml common.yaml:   lookup_options:     classes:       merge:         strategy: deep         knockout_prefix: --   classes:     - profiles::motd     - profiles::network os/RedHat.yaml:   classes:     - profiles::yum product/product.yaml  classes:    - profiles::application node/node.yaml:  classes:    - profiles::puppet-agent    - --profiles::network    - network {code} *Expected* {code:java} puppet lookup --knock-out-prefix '--' --merge deep classes --- - profiles::motd - profiles::yum - profiles::application - network {code} *Actual* {code:java} puppet lookup --knock-out-prefix '--' --merge deep classes --- - profiles::motd - profiles::network - profiles::yum - profiles::application - network {code} Removing a layer from the hierarchy causes the merge to behave as expected. The difference being: {code:java} puppet lookup --knock-out-prefix '--' --merge deep classes | grep 'Found key:' | wc -l 4 {code} vs {code:java} puppet lookup --knock-out-prefix '--' --merge deep classes | grep 'Found key:' | wc -l 3 {code} Verified in puppet version 4.10.0, 4.10.9

Add Comment

[Ryan Struber (JIRA)]

Verified in puppet version *Affect Versions* Puppet  4.10.0 , Puppet  4.10.9

Add Comment

[Ryan Struber (JIRA)]

* Affect Affected  Versions* Puppet 4.10.0 Puppet 4.10.9

Add Comment

[Ryan Struber (JIRA)]

* Verified Affected Versions* Puppet 4.10.0 Puppet 4.10.9

Add Comment

[Ryan Struber (JIRA)]

*Verified Affected Versions* Puppet 4.10.0 Puppet 4.10.9 Puppet 5.3.1

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges This is per design. A knock-out prefix is only capable of doing one knock-out. Once the knock-out is performed, it's not remembered.

Add Comment

[Ryan Struber (JIRA)]

Ryan Struber commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

I'm not sure I got the concept across correctly given your comment. I'm not reporting the knock-out is failing to perform multiple times. Rather if you merge together too many levels, knock-out fails to perform at all.

Add Comment

[Ryan Struber (JIRA)]

Ryan Struber updated an issue

Puppet / PUP-8405

Knock-out-prefix fails with more than 3 merges

Change By: Ryan Struber

-  -- profiles::network

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges I understand that, but the knock-out is not remembered after a merge. It's a concept implemented by the underlying deep_merge operation and forgotten once a merge has taken place. It doesn't matter if values were actually knocked out or not during that merge. The knock-out is only applicable to the immediate merge with the next level, not with levels beyond that.

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

Running the lookup with a --explain will provide information on what happens in each merge. The knock-outs are not specifically reported but you see the inputs and the result of each individual merge.

Add Comment

[R.I.Pienaar (JIRA)]

R.I.Pienaar commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

Imo this makes this feature more or less useless. It works too differently and have very surprising restrictions orthogonal to the rest of hiera If it can’t be fixed it should be removed rather than just be super weird

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

Is it true to say that the desired (unsurprising behaviour) would be that a knockout knocks out a key if it was present in the result from lower precedence entries in the hierarchy?

Add Comment

[R.I.Pienaar (JIRA)]

R.I.Pienaar commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

Personally to me that would be less surprising yes

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

We have several options: 1. Change the current behavior to something else, like knocking out keys of lower precedence than were the lookup_option was introduced. Question is though, what this precedence means given that a lookup_option that is found in a module actually has impact on entries found at higher levels. I.e. an environment can override a key defined in a module without overriding a lookup_option for that same key. The option will still be in effect but now be applied on the key from the environment instead. In any case, it will be a breaking change. 2. Remove the knockout_prefix altogether. Again, breaking change. 3. Deprecate the knockout_prefix and introduce something better. A filter proposal discussed before was tabled due to bad complexity/demand ratio. 4. Do nothing and instead give examples and recommendations on how lookup functions can be written in Puppet to achieve the desired results. 5. Something else...

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

But... the knockout prefix in option just turns on the feature and specifies what the prefix is. Don't think it is ever an issue to use different knockout prefixes for the same key. So... it is really a matter of merging the data and knocking out the value if there is an "anti-value"... no?

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

Yes, that's correct. But if we start talking about layers and precedence, then users might thing that the option is turned on in the module layer but have no effect on higher layers, which would be a false assumption but perhaps the distinction between options and values and their precedence is described well enough already. In any case, changing the behavior so that anti-values survive when they don't knock things out is a breaking change. We've discussed this before and decided it was not a direction we'd want to go. Please see PUP-7438 .

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

I have no worry over the lookup_options here. The knockout_prefix feature is kind of useless as it stands. Although the filter capability we discussed is nice it is not as convenient as the knockout prefix since you see the later in your data. The PUP-7438 ticket was closed in the big triage, don't think all that much thought went into that decision. We may want to change the behavior for puppet 6 (it is breaking, but I suspect the change will go unnoticed, since use of knockout that works will continue to work, so only affects those that may have garbage knockouts that they really did not want).

Add Comment

[Ryan Struber (JIRA)]

Ryan Struber commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

Just an aside, we've decided to move forward getting the difference of an exclusion key: classes: - network - agent   exclude_classes: - network   $classes = [ lookup(classes) - lookup(exclude_classes) ] Perhaps an example (more flushed out) like this in docs could be helpful given the functionality confusion.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-8405

Knock-out-prefix fails with more than 3 merges

Change By: Josh Cooper Sub-team: Language

Add Comment

This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574)

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: Josh Cooper Team: Platform Core

Add Comment

[Dominik Schuster (JIRA)]

Dominik Schuster commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges Thanks Ryan Struber that really unlocked me from the problem.

Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Kenyon Ralph (JIRA)]

Kenyon Ralph commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges

This seems to be a duplicate of PUP-7428 (and this is a bug that just bit me as well, trying to remove a class in a node that's added by the common level).

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-8405

Knock-out-prefix fails with more than 3 merges

Change By: Josh Cooper Team: Froyo

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)

[David McTavish (Jira)]

David McTavish updated an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: David McTavish Priority: Normal Low

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: Josh Cooper Component/s: Hiera & Lookup

Add Comment

[Alvin Rodis (Jira)]

Alvin Rodis updated an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: Alvin Rodis Zendesk Ticket Count: 1 Zendesk Ticket IDs: 47851

Add Comment

[Alvin Rodis (Jira)]

Alvin Rodis updated an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: Alvin Rodis Labels: jira_escalated

Add Comment

[Stephen Jennings (Jira)]

Stephen Jennings updated an issue

Puppet / PUP-8405 Knock-out-prefix fails with more than 3 merges

Change By: Stephen Jennings Priority: Low Major

Add Comment

[Stephen Jennings (Jira)]

Stephen Jennings commented on PUP-8405

Re: Knock-out-prefix fails with more than 3 merges Increase priority to major (customer sat). Unclear as to this or https://tickets.puppetlabs.com/browse/PUP-7409 are primary issues

Add Comment