Jira (PUP-10726) hiera lookup removes duplicate array elements during deep merge

7 views
Skip to first unread message

Reinhard Vicinus (Jira)

unread,
Oct 22, 2020, 9:46:03 AM10/22/20
to puppe...@googlegroups.com
Reinhard Vicinus created an issue
 
Puppet / Bug PUP-10726
hiera lookup removes duplicate array elements during deep merge
Issue Type: Bug Bug
Affects Versions: PUP 6.19.0
Assignee: Unassigned
Components: Hiera & Lookup
Created: 2020/10/22 6:45 AM
Priority: Normal Normal
Reporter: Reinhard Vicinus

Puppet Version: 6.19.0
Puppet Server Version: not installed
OS Name/Version: Ubuntu 20.04

hiera lookup removes duplicate array elements during deep merge.

hiera.yaml:

version: 5
 
defaults:
 
  data_hash: yaml_data
 
  datadir: data
 
hierarchy:
 
  - name: 'level1'
 
    path: level1.yaml
 
  - name: 'level2'
 
    path: level2.yaml

data/level1.yaml:

test:
 
  array:
 
    - 1
 
    - 1
 
    - 2

data/level2.yaml:

lookup_options:
 
  test:
 
    merge: 'deep'
 
 
 
test: {}

puppet lookup --explain test:

Searching for "lookup_options"
 
  Global Data Provider (hiera configuration version 5)
 
    No such key: "lookup_options"
 
  Environment Data Provider (hiera configuration version 5)
 
    Using configuration "/home/rvicinus/.puppetlabs/etc/code/environments/production/hiera.yaml"
 
    Merge strategy hash
 
      Hierarchy entry "level1"
 
        Path "/home/rvicinus/.puppetlabs/etc/code/environments/production/data/level1.yaml"
 
          Original path: "level1.yaml"
 
          No such key: "lookup_options"
 
      Hierarchy entry "level2"
 
        Path "/home/rvicinus/.puppetlabs/etc/code/environments/production/data/level2.yaml"
 
          Original path: "level2.yaml"
 
          Found key: "lookup_options" value: {
 
            "test" => {
 
              "merge" => "deep"
 
            }
 
          }
 
      Merged result: {
 
        "test" => {
 
          "merge" => "deep"
 
        }
 
      }
 
Using merge options from "lookup_options" hash
 
Searching for "test"
 
  Merge strategy deep
 
    Global Data Provider (hiera configuration version 5)
 
      No such key: "test"
 
    Environment Data Provider (hiera configuration version 5)
 
      Using configuration "/home/rvicinus/.puppetlabs/etc/code/environments/production/hiera.yaml"
 
      Merge strategy deep
 
        Hierarchy entry "level1"
 
          Path "/home/rvicinus/.puppetlabs/etc/code/environments/production/data/level1.yaml"
 
            Original path: "level1.yaml"
 
            Found key: "test" value: {
 
              "array" => [
 
                1,
 
                1,
 
                2
 
              ]
 
            }
 
        Hierarchy entry "level2"
 
          Path "/home/rvicinus/.puppetlabs/etc/code/environments/production/data/level2.yaml"
 
            Original path: "level2.yaml"
 
            Found key: "test" value: {}
 
        Merged result: {
 
          "array" => [
 
            1,
 
            2
 
          ]
 
        }
 
    Merged result: {
 
      "array" => [
 
        1,
 
        2
 
      ]
 
    }

Desired Behavior:

Return array with all 3 values: [ 1, 1, 2 ]

Actual Behavior:

Returns array with only unique values: [ 1, 2 ]

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Henrik Lindberg (Jira)

unread,
Oct 24, 2020, 7:58:04 AM10/24/20
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-10726
 
Re: hiera lookup removes duplicate array elements during deep merge

Yes it does. I believe it has always done that and what is wanted in most situations as you would otherwise end up with duplicates.
Don't think the default behaviour can be changed as that would be a breaking change for many users. Maybe this could be controlled with a lookup option to not unique merged arrays. It then depends on if this is supported in the deep merge gem that is used to perform the deep merge.

Mihai Buzgau (Jira)

unread,
Dec 1, 2020, 11:24:03 AM12/1/20
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
 
Change By: Mihai Buzgau
Labels: help-wanted

Mihai Buzgau (Jira)

unread,
Dec 1, 2020, 11:26:04 AM12/1/20
to puppe...@googlegroups.com
Mihai Buzgau commented on Bug PUP-10726
 
Re: hiera lookup removes duplicate array elements during deep merge

Thank you for filing this issue. We agree it is likely an improvement, but due to other issues demanding precedence, we don’t anticipate being able to address this any time soon. If you are interested in submitting a patch please open a pull request.

Andrew Lipscomb (Jira)

unread,
Jun 25, 2021, 2:20:01 AM6/25/21
to puppe...@googlegroups.com

Hi guys - I've made some headway with this once we found it in our codebase. Seems like an odd choice to treat arrays as sets rather than ... arrays in the `deep_merge` gem but hey - is what it is. 

I've made a branch with relevant code at https://github.com/ALTinners/puppet/tree/PUP-10726-keep-array-duplicates . No tests yet - I am a C++ coder by trade and am still figuring out Puppet's ruby codebase

However - in the process of figuring this out I also had to solve a related bug related to merging when combining an array with a nil

https://github.com/danielsdeleo/deep_merge/pull/48

Which I would assume needs a release before a Pull req is worth making as its a core `vendor_ruby` dependency. Given that lib hasn't had a release since 2017 (but had activity last year) I'm not sure how likely it is to get one. 

FWIW - I've stuck the changes in for this as a local patch on our main Puppet server and it does what we'd expect, so any one who needs a quick dirty fix can modify the handful of files required by hand. Obviously not a recommended solution - but a solution nonetheless. 

 
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages