Jira (PUP-8887) puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

4 views
Skip to first unread message

Stefan Andres (JIRA)

unread,
May 24, 2018, 5:30:03 AM5/24/18
to puppe...@googlegroups.com
Stefan Andres created an issue
 
Puppet / Bug PUP-8887
puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2018/05/24 2:29 AM
Priority: Normal Normal
Reporter: Stefan Andres

Puppet Version: 5.5.1

We are using array-iteration in our hiera.yaml to implement some kind of roles-concept.

For this we have nodes that can define an array of roles.

Those roles will be added to the hiera hierarchy by using the mapped_paths. The %{roles} variable is defined in our site.pp by using the lookup() function.

Prior to 5.5.1 we could could define lookup_options in roles dynamically loaded roles and define merge behavior or hashes defined in thoses roles. With 5.5.1 this does not seem to work anylonger. Values can still be looked up though. 

Below is a minimal reproducible case:

root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat ../hiera.yaml
 
---
 
version: 5
 
defaults:
 
hierarchy:
 
  - name: "node"
 
    path: node.yaml
 
  - name: "roles"
 
    mapped_paths:
 
      - roles
 
      - role
 
      - "%{role}.yaml"
 
  - name: "Other YAML hierarchy levels"
 
    paths:
 
      - "common.yaml"
 
      
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat ../data/common.yaml
 
 
 
myhash:
 
  key1: value1
 
  key2: value2
 
  
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat ../data/firstrole.yaml
 
lookup_options:
 
  myhash:
 
    merge:
 
      strategy: deep
 
 
 
myhash:
 
  role1: value1
 
  role2: value2
 
  
role1: role1
 
 
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat ../data/secondrole.yaml
 
myhash:
 
  role1: value1
 
  role3: value3
 
  
role2: role2
 
 
 
 
 
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat site.pp
 
$roles = lookup( 'roles', Array[String], 'unique', [], )
 
notice("roles: $roles")
 
$ret = lookup('myhash')
 
notice("myhash: $ret")
 
 
 
$role1 = lookup('role1')
 
$role2 = lookup('role2')
 
notice("role1 var: $role1")
 
notice("role2 var: $role2")
 
 
 
 
 
puppet-agent-5.4.0:
 
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# puppet apply site.pp
 
Warning: Undefined variable 'roles'; \n   (file & line not available)
 
Notice: Scope(Class[main]): roles: [firstrole, secondrole]
 
Notice: Scope(Class[main]): myhash: {key1 => value1, key2 => value2, role1 => value1, role3 => value3, role2 => value2}
 
Notice: Scope(Class[main]): role1 var: role1
 
Notice: Scope(Class[main]): role2 var: role2
 
Notice: Compiled catalog for c312a1b640ef in environment production in 0.03 seconds
 
Notice: Applied catalog in 0.01 seconds
 
 
 
puppet-agent-5.5.1
 
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# puppet apply site.pp
 
Warning: Undefined variable 'roles'; \n   (file & line not available)
 
Notice: Scope(Class[main]): roles: [firstrole, secondrole]
 
Notice: Scope(Class[main]): myhash: {role1 => value1, role2 => value2}
 
Notice: Scope(Class[main]): role1 var: role1
 
Notice: Scope(Class[main]): role2 var: role2
 
Notice: Compiled catalog for c312a1b640ef in environment production in 0.02 seconds
 
Notice: Applied catalog in 0.02 seconds

 

When putting the lookup_options in the common.yaml, the deep merge will happen also.

 

I couldn't find any related change in the Changelog for this change.

Desired Behavior:

I'd expect to lookup_options still to work.

Actual Behavior:

 

puppet lookup ignores the lookup_options defined in the roles.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Stefan Andres (JIRA)

unread,
May 24, 2018, 5:34:02 AM5/24/18
to puppe...@googlegroups.com
Stefan Andres commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

With PUP-8715, this prevents us from upgrading the puppet-agents on our puppetservers (this is where the logic is only used) to puppet-agent-5.5.1.

Stefan Andres (JIRA)

unread,
May 24, 2018, 5:57:04 AM5/24/18
to puppe...@googlegroups.com
Stefan Andres commented on Bug PUP-8887

The commit that introduced this new behavior is https://github.com/puppetlabs/puppet/commit/36cb716a37e091215ea5f2f9cdb8a29bff11e7b5#diff-9b5fcadd98b4a6f7f471fc1a8d3186dc in the file lib/puppet/pops/lookup/lookup_adapter.rb. When using an a revision older it still works.

Henrik Lindberg (JIRA)

unread,
May 24, 2018, 9:38:03 AM5/24/18
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Thomas Hallgren
 
Change By: Henrik Lindberg
Assignee: Thomas Hallgren
Sub-team: Language
Team: Platform Core

Henrik Lindberg (JIRA)

unread,
May 24, 2018, 9:38:03 AM5/24/18
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Affects Version/s: PUP 5.5.1

Thomas Hallgren (JIRA)

unread,
May 24, 2018, 10:08:02 AM5/24/18
to puppe...@googlegroups.com
Thomas Hallgren commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

The example seems incomplete. I don't understand how the above code can produce the notice:

roles: [firstrole, secondrole]

Where did that array come from? Lookup will not find a 'roles' key in any of the listed data files and the provided default value is an empty array.

I get the impression that $roles gets its value from a lookup and then is used to actually configure lookup. If that's true, then it's classic chicken-and-egg problem. Constructs like that should be avoided at length.

Henrik Lindberg (JIRA)

unread,
May 24, 2018, 10:19:03 AM5/24/18
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Stefan Andres
 
Change By: Henrik Lindberg
Assignee: Thomas Hallgren Stefan Andres

Stefan Andres (JIRA)

unread,
May 24, 2018, 10:20:03 AM5/24/18
to puppe...@googlegroups.com
Stefan Andres commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

Ah, I forgot to paste the node.yaml, of course - this is where our roles for nodes are defined:

 

root@c312a1b640ef:/etc/puppetlabs/code/environments/production# cat data/node.yaml
 
roles:
 
  - firstrole
 
  - secondrole

Stefan Andres (JIRA)

unread,
May 24, 2018, 10:22:03 AM5/24/18
to puppe...@googlegroups.com
Stefan Andres updated an issue
 
Change By: Stefan Andres
*Puppet Version: 5.5.1*


We are using array-iteration in our hiera.yaml to implement some kind of roles-concept.

For this we have nodes that can define an array of roles.

Those roles will be added to the hiera hierarchy by using the mapped_paths. The %\{roles} variable is defined in our site.pp by using the lookup() function.

Prior to 5.5.1 we could could define lookup_options in roles dynamically loaded roles and define merge behavior or hashes defined in thoses roles. With 5.5.1 this does not seem to work anylonger. Values can still be looked up though. 

Below is a minimal reproducible case:
{code:java}
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat ../hiera.yaml
---
version: 5
defaults:
hierarchy:
  - name: "node"
    path: node.yaml
  - name: "roles"
    mapped_paths:
      - roles
      - role
      - "%{role}.yaml"
  - name: "Other YAML hierarchy levels"
    paths:
      - "common.yaml"
      
root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat ../data/ node.yaml

roles:
  - firstrole
  - secondrole

root@c312a1b640ef:/etc/puppetlabs/code/environments/production/manifests# cat ../data/ common.yaml
{code}

 

When putting the lookup_options in the common.yaml, the deep merge will happen also.

 

I couldn't find any related change in the Changelog for this change.

*Desired Behavior:*


I'd expect to lookup_options still to work.

*Actual Behavior:*


 

puppet lookup ignores the lookup_options defined in the roles.

Stefan Andres (JIRA)

unread,
May 24, 2018, 10:37:03 AM5/24/18
to puppe...@googlegroups.com
Stefan Andres assigned an issue to Unassigned
Change By: Stefan Andres
Assignee: Stefan Andres

Henrik Lindberg (JIRA)

unread,
May 24, 2018, 11:37:02 AM5/24/18
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Thomas Hallgren
Change By: Henrik Lindberg
Assignee: Thomas Hallgren

Thomas Hallgren (JIRA)

unread,
May 24, 2018, 11:47:03 AM5/24/18
to puppe...@googlegroups.com
Thomas Hallgren commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

Henrik Lindberg, the commit that breaks this workflow, can it be reverted? Judging from the comment, it's an optimization.

Thomas Hallgren (JIRA)

unread,
May 24, 2018, 11:47:04 AM5/24/18
to puppe...@googlegroups.com
Thomas Hallgren assigned an issue to Henrik Lindberg
 
Change By: Thomas Hallgren
Assignee: Thomas Hallgren Henrik Lindberg

Henrik Lindberg (JIRA)

unread,
May 24, 2018, 1:32:03 PM5/24/18
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

Not sure if it will work to just revert it, and it makes convert_to more expensive (yet another lookup of lookup_options).

Henrik Lindberg (JIRA)

unread,
May 25, 2018, 6:12:04 AM5/25/18
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Stefan Andres
 
Change By: Henrik Lindberg
Assignee: Henrik Lindberg Stefan Andres

Henrik Lindberg (JIRA)

unread,
May 25, 2018, 6:12:04 AM5/25/18
to puppe...@googlegroups.com
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

Looked at this again. I think the issue is that the feature convert_to requires getting the lookup options and doing that changes at what time the lookup options gets bound. Don't think that will change by reverting the commit in question.

The fact remains that it is a really bad idea to have the hiera.config depend on the data it references. It is not designed to handle that. The input to the hiera.config is using must to come from facts, an ENC or from variables set in topscope in site.pp. The variables in top scope that the hiera config makes use of should not be set via a call to lookup as that makes the definition recursive/ambiguous.

To make sure that the problem I describe in this comment is what is what is causing the problem - a quick test is to assign the $roles statically in site.pp and observe how that works.

Stefan Andres (JIRA)

unread,
May 28, 2018, 9:16:03 AM5/28/18
to puppe...@googlegroups.com
Stefan Andres commented on Bug PUP-8887

Thank you for taking the time to answer this!

 

As a matter of fact, we've migrated all nodes from a 100% ENC (own development and therefore hard to maintain vs. just using hiera) environment to a 100% hiera environment.

Therefore we are really not looking into an option to implement some kind of ENC logic again (especially because hiera fulfilled all our needs until now).

 

When assigning the roles variable not by using lookup(), but hardcoding its values it indeed works for >puppet-agent-5.5

root@c312a1b640ef:/etc/puppetlabs/code/environments/production# cat manifests/site.pp
 
#$roles = lookup( 'roles', Array[String], 'unique', [], )
 
$roles = ['firstrole', 'secondrole']
 
notice("roles: $roles")
 
$ret = lookup('myhash')
 
notice("myhash: $ret")
 
$role1 = lookup('role1')
 
$role2 = lookup('role2')
 
notice("role1 var: $role1")
 
notice("role2 var: $role2")
 
 
 
root@c312a1b640ef:/etc/puppetlabs/code/environments/production# puppet apply /etc/puppetlabs/code/environments/production/manifests/site.pp
 
Notice: Scope(Class[main]): roles: [firstrole, secondrole]
 
Notice: Scope(Class[main]): myhash: {key1 => value1, key2 => value2, role1 => value1, role3 => value3, role2 => value2}
 
Notice: Scope(Class[main]): role1 var: role1
 
Notice: Scope(Class[main]): role2 var: role2
 
Notice: Compiled catalog for c312a1b640ef in environment production in 0.02 seconds
 
Notice: Applied catalog in 0.02 seconds

 

What we are wondering is that a Y release breaks our backwards compatibility. We'd expect such a change in a X release (or better not at all in our case). 

https://puppet.com/docs/puppet/5.5/release_notes.html

Puppet’s version numbers use the format X.Y.Z, where:
 
* X must increase for major backward-incompatible changes
 
* Y can increase for backward-compatible new functionality or significant bug fixes
 
* Z can increase for bug fixes

 

Stefan Andres (JIRA)

unread,
May 28, 2018, 9:35:01 AM5/28/18
to puppe...@googlegroups.com
Stefan Andres assigned an issue to Unassigned
Change By: Stefan Andres
Assignee: Stefan Andres

Henrik Lindberg (JIRA)

unread,
May 28, 2018, 10:05:02 AM5/28/18
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

Stefan Andres Thanks for test - that confirms that there is nothing else that is causing the problem you observed. The change to hiera was unintentional in this respect. We think that the changed logic causes an earlier binding of the lookup_options. This breaks your case since you rely on them being bound on subsequent lookups when roles have been established. We believe that the change in time for when lookup_options get bound uncovered another problem - that lookup_options are not evicted from the cache when the inputs the cached result depends on has changed. We will look into that problem as we think your use case should work (even though we do not recommend hiera configuring itself because self recursion may lead to confusion and loss of performance (the cache must be evicted).

Should you want to change, you would need for the nodes to report their roles in a fact if you do not want to use an ENC.

Henrik Lindberg (JIRA)

unread,
May 28, 2018, 10:06:02 AM5/28/18
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Thomas Hallgren
 
Change By: Henrik Lindberg
Assignee: Thomas Hallgren

Matthias Baur (JIRA)

unread,
Jun 29, 2018, 7:30:03 AM6/29/18
to puppe...@googlegroups.com
Matthias Baur commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

Thanks! Is there anything we can do to get this fixed? This is blocking us from upgrading to 5.5.x on our Puppetservers.

Stefan Andres (JIRA)

unread,
Sep 18, 2018, 4:39:04 AM9/18/18
to puppe...@googlegroups.com
Stefan Andres commented on Bug PUP-8887

We've developed a workaround for us that prevents caching the $roles variable by using loadyaml() directly in die site.pp

Here's the workaround code:

# old
 
$roles = lookup( 'roles', Array[String], 'unique', [], ) # old style  
 
 
# new
 
$nodeyaml = loadyaml("/etc/puppetlabs/code/environments/production/manifests/../data/node.yaml")
 
if $nodeyaml['roles'] {
 
  if $nodeyaml['roles'] !~ Array[String[1]] {
 
    fail('roles: Needs to be an array of strings!')
 
  } else {
 
    $roles = unique($nodeyaml['roles'])
 
  }
 
} else {
 
  $roles = []
 
}

We'd still appreciate a fix of this regression.

Here's a script that can trigger the regression. 

 
 
prod=/etc/puppetlabs/code/environments/production# Create test datarm -rf /etc/puppetlabs/code/
 
puppet module install puppetlabs-stdlib >/dev/nullmkdir -p "$prod"/{manifests,modules,data}
 
cat > "$prod/hiera.yaml"  <<EOF
 
---
 
version: 5
 
defaults:
 
hierarchy:
 
  - name: "node"
 
    path: node.yaml
 
  - name: "roles"
 
    mapped_paths:
 
      - roles
 
      - role
 
      - "%{role}.yaml"
 
  - name: "Other YAML hierarchy levels"
 
    paths:
 
      - "common.yaml"EOF
 
cat > "$prod/data/node.yaml" <<EOF
 
roles:
 
  - firstrole
 
  - secondrole
 
EOF
 
cat > "$prod/data/common.yaml" <<EOF
 
myhash:
 
  key1: value1
 
  key2: value2
 
EOF
 
cat > "$prod/data/firstrole.yaml" <<EOF
 
lookup_options:
 
  myhash:
 
    merge:
 
      strategy: deepmyhash:
 
  role1: value1
 
  role2: value2role1: role1EOFcat > "$prod/data/secondrole.yaml" <<EOF
 
myhash:
 
  role1: value1
 
  role3: value3role2: role2
 
EOF
 
cat > "$prod/manifests/site.pp" <<'EOF'
 
$roles = lookup( 'roles', Array[String], 'unique', [], ) # old style
 
#$nodeyaml = loadyaml("/etc/puppetlabs/code/environments/production/manifests/../data/node.yaml")
 
#if $nodeyaml['roles'] {
 
#  if $nodeyaml['roles'] !~ Array[String[1]] {
 
#    fail('roles: Needs to be an array of strings!')
 
#  } else {
 
#    $roles = unique($nodeyaml['roles'])
 
#  }
 
#} else {
 
#  $roles = []
 
#}
 
notice("roles: $roles")
 
$myhash = lookup('myhash')
 
notice("myhash: $myhash")$role1 = lookup('role1')
 
$role2 = lookup('role2')
 
notice("role1 var: $role1")
 
notice("role2 var: $role2")# tests
 
if $roles != ['firstrole', 'secondrole'] {
 
  fail('$roles does not contain firstrole, secondrole')
 
}
 
if $myhash[key1] != 'value1' {
 
  fail('$myhash_key1 does not contain value1')
 
}
 
if $myhash['key2'] != 'value2' {
 
  fail('$myhash_key1 does not contain value1')
 
}
 
if $myhash['role1'] != 'value1' {
 
  fail('$myhash_role1 does not contain value1')
 
}
 
if $myhash['role2'] != 'value2' {
 
  fail('$myhash_role2 does not contain value2')
 
}
 
if $myhash['role3'] != 'value3' {
 
  fail('$myhash_role3 does not contain value3')
 
}
 
EOF
 
puppet apply "$prod/manifests/site.pp"rm -rf "$prod"

 

David McTavish (Jira)

unread,
Jan 12, 2022, 1:34:01 PM1/12/22
to puppe...@googlegroups.com
David McTavish updated an issue
 
Change By: David McTavish
Labels: final_triage has_workaround
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

David McTavish (Jira)

unread,
Jan 12, 2022, 1:34:01 PM1/12/22
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Priority: Normal Low

Josh Cooper (Jira)

unread,
Jan 13, 2022, 1:02:02 AM1/13/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-8887
 
Re: puppet-5.5.0 breaks lookup_options when used post-loading dynamic items from hiera.yaml

So to summarize the issue, lookup_options set in common_yaml apply, so you get the merged roles:

Notice: Scope(Class[main]): myhash: {key1 => value1, key2 => value2, role1 => value1, role3 => value3, role2 => value2}

But if the lookup_options are set in firstrole.yaml then the merge doesn't happen:

Notice: Scope(Class[main]): myhash: {role1 => value1, role2 => value2}

Note there's an different lookup_options caching issue in PUP-9792. In that case, a variable affecting how the hiera config is interpolated changes, but the hiera config & lookup options aren't refreshed.

Molly Waggett (Jira)

unread,
Feb 22, 2022, 1:51:01 PM2/22/22
to puppe...@googlegroups.com
Molly Waggett updated an issue
 
Change By: Molly Waggett
Labels: final_triage has_workaround

Molly Waggett (Jira)

unread,
Feb 22, 2022, 1:51:01 PM2/22/22
to puppe...@googlegroups.com
Molly Waggett updated an issue
Change By: Molly Waggett
Team: Froyo Phoenix
Reply all
Reply to author
Forward
0 new messages