Jira (PUP-7409) Knockout prefix does not remove an entire key:value entry from a hash

[Stefan Pijnappels (JIRA)]

Stefan Pijnappels created an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash Issue Type: Bug Assignee: Unassigned Created: 2017/03/30 3:29 AM Environment: Test Vagrant environment using PE2017.1.1 Priority: Normal Reporter: Stefan Pijnappels While doing some testing on the behaviour of deep_merge using Hiera, it was noted that there was some unexpected behaviour when using a knockout_prefix in the lookup options. The common.yaml files were defined as follows: [root@pe-201711-master production]# cat data/common.yaml --- profile::test::hashdeep: --key1: 'a' key2: '--f' --key3: '--g' key5: 'val5' array1: - --a - b - e - f [root@pe-201711-master production]# cat modules/profile/data/common.yaml --- profile::test::hashdeep: key1: 'a' key2: 'b' key3: 'c' key4: 'd' array1: - a - b - c - d lookup_options: profile::test::hashdeep: merge: strategy: deep knockout_prefix: -- sort_merge_arrays: true [root@pe-201711-master production]# The expected behaviour was that either using puppet lookup or using a notify statement, key1 and key3 would not be included in the returned data, and key2 would return an empty, undefined value. However, it appears that using the knockout_prefix on a key name simply prepends the prefix to the key name in the output, rather than removing the key and value completely. Using the knockout_prefix on the value for a given key works as expected, with the value being removed and rest to an undefined, empty value. [root@pe-201711-master production]# puppet lookup profile::test::hashdeep --explain Warning: /etc/puppetlabs/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (in /etc/puppetlabs/puppet/hiera.yaml) Warning: Undefined variable 'pp_application'; (file & line not available) Warning: Undefined variable 'pp_role'; (file & line not available) Warning: Undefined variable 'pp_environment'; (file & line not available) Warning: Undefined variable 'pp_datacenter'; (file & line not available) Warning: Undefined variable 'pp_zone'; (file & line not available) Warning: Undefined variable '::pp_application'; (file & line not available) Warning: Undefined variable '::pp_role'; (file & line not available) Warning: Undefined variable '::pp_environment'; (file & line not available) Warning: Undefined variable '::pp_datacenter'; (file & line not available) Warning: Undefined variable '::pp_zone'; (file & line not available) Searching for "lookup_options" Module "profile" Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/code/environments/production/modules/profile/hiera.yaml" Hierarchy entry "Module Hiera Data" Merge strategy hash Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS-7.yaml" Original path: "%{facts.os.name}-%{facts.os.release.major}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS.yaml" Original path: "%{facts.os.name}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/modules/profile/data/RedHat.yaml" Original path: "%{facts.os.family}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/modules/profile/data/common.yaml" Original path: "common.yaml" Found key: "lookup_options" value: { "profile::test::hashdeep" => { "merge" => { "strategy" => "deep", "knockout_prefix" => "--", "sort_merge_arrays" => true } } } Path "/etc/puppetlabs/code/environments/production/modules/profile/data/profile.yaml" Original path: "profile.yaml" Path not found Merged result: { "profile::test::hashdeep" => { "merge" => { "strategy" => "deep", "knockout_prefix" => "--", "sort_merge_arrays" => true } } } Using merge options from "lookup_options" hash Searching for "profile::test::hashdeep" Merge strategy deep Options: { "knockout_prefix" => "--", "sort_merge_arrays" => true } Global Data Provider (hiera configuration version 3) Using configuration "/etc/puppetlabs/puppet/hiera.yaml" Hierarchy entry "yaml" Merge strategy deep Options: { "knockout_prefix" => "--", "sort_merge_arrays" => true } Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/pe-201711-master.puppetdebug.vlan.yaml" Original path: "nodes/%{::trusted.certname}" Path not found Path "/etc/puppetlabs/code/environments/production/hieradata/common.yaml" Original path: "common" No such key: "profile::test::hashdeep" Environment Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml" Hierarchy entry "Global yaml hierarchy" Merge strategy deep Options: { "knockout_prefix" => "--", "sort_merge_arrays" => true } Path "/etc/puppetlabs/code/environments/production/data/nodes/pe-201711-master.puppetdebug.vlan.yaml" Original path: "nodes/%{::trusted.certname}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/role/--.yaml" Original path: "role/%{::pp_application}-%{::pp_role}-%{::pp_environment}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/role/-.yaml" Original path: "role/%{::pp_application}-%{::pp_role}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/role/-.yaml" Original path: "role/%{::pp_application}-%{::pp_environment}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/role/.yaml" Original path: "role/%{::pp_application}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/location/-.yaml" Original path: "location/%{::pp_datacenter}-%{::pp_zone}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/location/.yaml" Original path: "location/%{::pp_datacenter}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/location/.yaml" Original path: "location/%{::pp_zone}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/data/common.yaml" Original path: "common.yaml" Found key: "profile::test::hashdeep" value: { "--key1" => "a", "key2" => "--f", "--key3" => "--g", "key5" => "val5", "array1" => [ "b", "e", "f" ] } Merged result: { "--key1" => "a", "key2" => "--f", "--key3" => "--g", "key5" => "val5", "array1" => [ "b", "e", "f" ] } Module "profile" Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/code/environments/production/modules/profile/hiera.yaml" Hierarchy entry "Module Hiera Data" Merge strategy deep Options: { "knockout_prefix" => "--", "sort_merge_arrays" => true } Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS-7.yaml" Original path: "%{facts.os.name}-%{facts.os.release.major}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS.yaml" Original path: "%{facts.os.name}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/modules/profile/data/RedHat.yaml" Original path: "%{facts.os.family}.yaml" Path not found Path "/etc/puppetlabs/code/environments/production/modules/profile/data/common.yaml" Original path: "common.yaml" Found key: "profile::test::hashdeep" value: { "key1" => "a", "key2" => "b", "key3" => "c", "key4" => "d", "array1" => [ "a", "b", "c", "d" ] } Path "/etc/puppetlabs/code/environments/production/modules/profile/data/profile.yaml" Original path: "profile.yaml" Path not found Merged result: { "key1" => "a", "key2" => "b", "key3" => "c", "key4" => "d", "array1" => [ "a", "b", "c", "d" ] } Merged result: { "key1" => "a", "key2" => "", "key3" => "c", "key4" => "d", "array1" => [ "b", "c", "d", "e", "f" ], "--key1" => "a", "--key3" => "", "key5" => "val5" } [root@pe-201711-master production]# Is this a bug or is it by design, and if it is by design, how could an entire key:value pair be removed from the output using a knockout_prefix value? Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Stefan Pijnappels (JIRA)]

Stefan Pijnappels updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Stefan Pijnappels Affects Version/s: PUP 4.9.4

Add Comment

[Stefan Pijnappels (JIRA)]

Stefan Pijnappels updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Stefan Pijnappels Labels: hiera

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg assigned an issue to Thomas Hallgren

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Henrik Lindberg Assignee: Thomas Hallgren

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Henrik Lindberg Team: Puppet Developer Experience Sprint: PDE Triage

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Henrik Lindberg Sprint: Forge - To Accept

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Henrik Lindberg Sprint: PDE Triage

Add Comment

[Corey Hickey (Jira)]

Corey Hickey commented on PUP-7409

Re: Knockout prefix does not remove an entire key:value entry from a hash Is this ticket actually done? The resolution is "Done", but I still can't make knockout_prefix remove a hash key with puppet 7.14.0. I seem to end up finding this ticket once a year or so when I go looking for a way to do that... If the actual resolution is "WONTFIX" due to lack of support in the deep_merge gem, then I can understand that, but if so, can you please update the ticket "Resolution" field? I can't see DOC-3150, but I do think the documentation is still unclear: A string prefix to indicate a value should be removed from the final result. If a value is exactly equal to the prefix, it will knockout the entire element. Pertaining to a hash, I don't know what an "element" is; this term is never used in: https://puppet.com/docs/puppet/7/lang_data_hash.html ...so I would tend to guess that "element" means a key/value pair, but that does not match the observed behavior of knockount_prefix only clearing the value but leaving the key remaining. Thanks, Corey

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)

[Austin Boyd (Jira)]

Austin Boyd updated an issue

Puppet / PUP-7409

Knockout prefix does not remove an entire key:value entry from a hash

Change By: Austin Boyd Labels: hiera jira_escalated

Add Comment

[Jarret Lavallee (Jira)]

Jarret Lavallee updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Jarret Lavallee Zendesk Ticket Count: 1 Zendesk Ticket IDs: 47851

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Josh Cooper

Sprint: Forge - To Accept

Add Comment

[Josh Cooper (Jira)]

Josh Cooper assigned an issue to Unassigned

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Josh Cooper Assignee: Thomas Hallgren

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Josh Cooper Team: Puppet Developer Experience

Add Comment

[Jarret Lavallee (Jira)]

Jarret Lavallee commented on PUP-7409

Re: Knockout prefix does not remove an entire key:value entry from a hash It looks like the issue here is that the knockout prefix does not remove entries across different hiera layers. Here is an example. Here is an example of it working fine within the environment layer. # cat common.yaml --- mymodule::test:

- a - b - c

# cat nodes/pe-2019810-master.puppetdebug.vlan.yaml --- lookup_options: mymodule::test: merge: strategy: deep knockout_prefix: '--' sort_merged_arrays: true mymodule::test: - '--a' - d # puppet lookup mymodule::test ---

- b - c - d

When I add in the module layer data, I am unable to knockout the elements from the module layer. # cat data/common.yaml --- mymodule::test:

- a - b - c

# cat data/nodes/pe-2019810-master.puppetdebug.vlan.yaml --- lookup_options: mymodule::test: merge: strategy: deep knockout_prefix: '--' sort_merged_arrays: true mymodule::test: - '--a' - d # cat modules/mymodule/data/common.yaml --- mymodule::test: - x - y - z - a # puppet lookup mymodule::test ---

- a - b - c - d

- x - y - z The lookup explain shows that the environment knockout prefix is applied and the elements from the environment layer have the knockout prefix applied within it. It removes the --a correctly.

Environment Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"

Merge strategy deep Options: { "sort_merged_arrays" => true, "merge_hash_arrays" => false, "knockout_prefix" => "--" } Hierarchy entry "Per-node data (yaml version)" Path "/etc/puppetlabs/code/environments/production/data/nodes/pe-2019810-master.puppetdebug.vlan.yaml"

Original path: "nodes/%{::trusted.certname}.yaml"

Found key: "mymodule::test" value: [ "d" ] Hierarchy entry "Other YAML hierarchy levels"

Path "/etc/puppetlabs/code/environments/production/data/common.yaml" Original path: "common.yaml"

Found key: "mymodule::test" value: [

"a", "b", "c"

] Merged result: [

"b", "c", "d" ]

Since the environment layer is then merged with the module layer, it uses the merged environment result to merge with the module layer merged result. The problem is that the merged result from the environment layer does not include the --a element to knockout the data from the module layer, so no knockout happens across the layer boundary. Module "mymodule" Data Provider (hiera configuration version 5) Using configuration "/etc/puppetlabs/code/environments/production/modules/mymodule/hiera.yaml" Hierarchy entry "Module Defaults" Path "/etc/puppetlabs/code/environments/production/modules/mymodule/data/common.yaml" Original path: "common.yaml" Found key: "mymodule::test" value: [ "x", "y", "z", "a" ] Merged result: [

"a", "b", "c",

"d", "x", "y", "z" ] This is a limitation where the knockout items are limited to within a hiera layer, or the logic should change to knockout the items between the hiera layers at the final merging. There are easy workarounds for this in the Puppet code, so it should not be a blocker. Workaround #1: Do a post hiera lookup knockout filter. $final_array = $hiera_array.filter |$item| { $item !~ /^--.*/ } Workaround #2: Use another array as removals $final_array = $hiera_array - $removals

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7409

Knockout prefix does not remove an entire key:value entry from a hash

Change By: Josh Cooper Team: Phoenix

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Josh Cooper Component/s: Hiera & Lookup

Add Comment

[Stephen Jennings (Jira)]

Stephen Jennings updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Stephen Jennings Priority: Normal Major

Add Comment

[Morgan Rhodes (Jira)]

Morgan Rhodes updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Morgan Rhodes Epic Link: PUP-11619

Add Comment

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)

[Morgan Rhodes (Jira)]

Morgan Rhodes updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: Morgan Rhodes Priority: Major Normal

Add Comment

[David Piekny (Jira)]

David Piekny updated an issue

Puppet / PUP-7409 Knockout prefix does not remove an entire key:value entry from a hash

Change By: David Piekny Epic Link: PUP- 11619 11658

Add Comment