Jira (PUP-11437) puppet lookup fails to interpolate topscope variables when an environment is specified

8 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Jan 27, 2022, 7:12:03 PM1/27/22
to puppe...@googlegroups.com
Josh Cooper created an issue
 
Puppet / Bug PUP-11437
puppet lookup fails to interpolate topscope variables when an environment is specified
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2022/01/27 4:11 PM
Priority: Normal Normal
Reporter: Josh Cooper

A regression introduced in PUP-7479 causes "puppet lookup --environment <env> <key>" to no longer set node parameters as topscope variables. If hiera.yaml is configured to interpolate based on topscope variables, such as "nodes/%{fqdn}.yaml", then the variable will be interpolated as an empty string.

Given a basic setup:

# tree /etc/puppetlabs/code/environments/
/etc/puppetlabs/code/environments/
├── production
└── test
    ├── data
    │   ├── common.yaml
    │   └── iron-judgement.yaml
    └── hiera.yaml
 
3 directories, 3 files
# cat /etc/puppetlabs/code/environments/test/hiera.yaml 
---
version: 5
hierarchy:
  - name: "node"
    paths:
      - "%{hostname}.yaml"
  - name: "common"
    paths:
      - "common.yaml"
# cat /etc/puppetlabs/code/environments/test/data/common.yaml 
---
foo: defined in common
# cat /etc/puppetlabs/code/environments/test/data/iron-judgement.yaml 
--- {}

Then a warning is generated trying to resolve the `hostname` variable:

# bundle exec puppet lookup -E test foo --explain
Warning: Undefined variable 'hostname'; 
   (file & line not available)
Searching for "lookup_options"
  Global Data Provider (hiera configuration version 5)
    No such key: "lookup_options"
  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/test/hiera.yaml"
    Merge strategy hash
      Hierarchy entry "node"
        Path "/etc/puppetlabs/code/environments/test/data/.yaml"
          Original path: "%{hostname}.yaml"
          Path not found
      Hierarchy entry "common"
        Path "/etc/puppetlabs/code/environments/test/data/common.yaml"
          Original path: "common.yaml"
          No such key: "lookup_options"

Note the path "/etc/puppetlabs/code/environments/test/data/.yaml"

The issue does not occur when an environment is omitted from the command line or when hiera interpolation uses facts such as "nodes/%{facts.hostname}.yaml".

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Josh Cooper (Jira)

unread,
Jan 27, 2022, 7:27:01 PM1/27/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11437
 
Re: puppet lookup fails to interpolate topscope variables when an environment is specified

The problem is that if --environment is specified, then we create a Node, but never call Node#facts_merge which is what populates the node's parameters. Normally this occurs in the plain or classifier node termini.

Josh Cooper (Jira)

unread,
Jan 31, 2022, 11:30:08 AM1/31/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Feb 1, 2022, 4:32:01 PM2/1/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Known Issue
Release Notes Summary: Puppet 6.26 and 7.14 will fail to resolve toplevel facts in hiera configs when using the --environment option for puppet lookup. As a workaround use trusted facts or specify the fact value using the "facts" hash, such as  "%{facts.hostname}"

Josh Cooper (Jira)

unread,
Feb 1, 2022, 4:33:03 PM2/1/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 7.15.0
Fix Version/s: PUP 6.27.0

Heston Hoffman (Jira)

unread,
Feb 1, 2022, 6:52:02 PM2/1/22
to puppe...@googlegroups.com
Heston Hoffman updated an issue
Change By: Heston Hoffman
Release Notes Summary: (Note from Heston: I've added the known issue to the docs, it will need to be removed when the fix goes live in the versions above.)

Puppet 6.26 and 7.14 will fail to resolve toplevel facts in hiera configs when using the --environment option for puppet lookup. As a workaround use trusted facts or specify the fact value using the "facts" hash, such as  "%{facts.hostname}"

Nirupama Mantha (Jira)

unread,
Feb 2, 2022, 11:49:01 AM2/2/22
to puppe...@googlegroups.com

Nirupama Mantha (Jira)

unread,
Feb 2, 2022, 11:51:01 AM2/2/22
to puppe...@googlegroups.com

Lisa Ross (Jira)

unread,
Feb 9, 2022, 12:47:01 PM2/9/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Feb 10, 2022, 2:26:01 AM2/10/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11437
 
Re: puppet lookup fails to interpolate topscope variables when an environment is specified

I think we want to do something like the following (ignoring whitespace changes):

diff --git a/lib/puppet/application/lookup.rb b/lib/puppet/application/lookup.rb
index cfbcfab81e..b9e6f22c87 100644
--- a/lib/puppet/application/lookup.rb
+++ b/lib/puppet/application/lookup.rb
@@ -374,12 +374,9 @@ Copyright (c) 2015 Puppet Inc., LLC Licensed under the Apache 2.0 License
 
     unless node.is_a?(Puppet::Node) # to allow unit tests to pass a node instance
       facts = retrieve_node_facts(node, given_facts)
-      if Puppet.settings.set_by_cli?('environment')
-        node = Puppet::Node.new(node, :classes => nil, :parameters => nil, :facts => facts, :environment => Puppet.settings.value('environment'))
-      else
       ni = Puppet::Node.indirection
       tc = ni.terminus_class
-        if options[:compile]
+      if options[:compile] && !Puppet.settings.set_by_cli?('environment')
         if tc == :plain
           node = ni.find(node, facts: facts)
         else
@@ -402,10 +399,9 @@ Copyright (c) 2015 Puppet Inc., LLC Licensed under the Apache 2.0 License
         end
       else
         ni.terminus_class = :plain
-          node = ni.find(node, facts: facts)
+        node = ni.find(node, facts: facts, environment: Puppet[:environment])
         ni.terminus_class = tc
       end
-      end
     else
       node.add_extra_facts(given_facts) if given_facts
     end

So if an environment is specified on the command line, then we always use the plain node terminus *and pass in our current environment* so that the returned node object has all of the correct node parameters. If the environment doesn't exist locally then the command should fail (with and without the --compile) option.

Nirupama Mantha (Jira)

unread,
Feb 10, 2022, 11:52:01 AM2/10/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Feb 15, 2022, 3:40:03 PM2/15/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Acceptance Criteria: The puppet lookup command should resolve topscope variables when an environment is specified on the command line in the same way it does when the environment is specified in puppet.conf.

Christopher Thorn (Jira)

unread,
Mar 1, 2022, 3:23:03 PM3/1/22
to puppe...@googlegroups.com

Nirupama Mantha (Jira)

unread,
Mar 2, 2022, 11:11:01 AM3/2/22
to puppe...@googlegroups.com

Christopher Thorn (Jira)

unread,
Mar 9, 2022, 3:08:02 PM3/9/22
to puppe...@googlegroups.com

Christopher Thorn (Jira)

unread,
Mar 9, 2022, 3:09:02 PM3/9/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 14, 2022, 12:23:02 PM3/14/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 14, 2022, 12:25:02 PM3/14/22
to puppe...@googlegroups.com

Parker Leach (Jira)

unread,
Mar 14, 2022, 12:38:02 PM3/14/22
to puppe...@googlegroups.com

Parker Leach (Jira)

unread,
Mar 14, 2022, 1:30:02 PM3/14/22
to puppe...@googlegroups.com

Parker Leach (Jira)

unread,
Mar 15, 2022, 1:47:03 PM3/15/22
to puppe...@googlegroups.com

Christopher Thorn (Jira)

unread,
Mar 18, 2022, 12:01:01 PM3/18/22
to puppe...@googlegroups.com
Christopher Thorn updated an issue
Change By: Christopher Thorn
Affects Version/s: PUP 6.26.0
Affects Version/s: PUP 7.14.0

Tim Meusel (Jira)

unread,
Jun 20, 2022, 9:01:06 AM6/20/22
to puppe...@googlegroups.com
Tim Meusel commented on Bug PUP-11437
 
Re: puppet lookup fails to interpolate topscope variables when an environment is specified

Hi josh,
I'm currently testing on PE 2019 with Puppet Agent 6.27.0 and PE 2021 with Puppet Agent 7.16.0. I have topscope variables set in nodegroups in the PE console. I use those in hiera.yaml. Normal puppet agent runs works fine. I'm not able to lookup keys in Hiera with `puppet lookup` because it cannot resolve the topscope variables. I think fixing that wasn't the intention of the PR? puppet lookup now resolves facts, that are written as topscope variables, but not all topscope variables?

If my assumption is true I think the text in the release notes should be updated, it's a bit missleading.

Alvin Rodis (Jira)

unread,
Jul 21, 2022, 10:43:02 PM7/21/22
to puppe...@googlegroups.com
Alvin Rodis updated an issue
 
Change By: Alvin Rodis
Labels: docs_reviewed jira_escalated
This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)
Atlassian logo

Alvin Rodis (Jira)

unread,
Jul 21, 2022, 10:43:03 PM7/21/22
to puppe...@googlegroups.com
Alvin Rodis updated an issue
Change By: Alvin Rodis
Zendesk Ticket Count: 1
Zendesk Ticket IDs: 49053
Reply all
Reply to author
Forward
0 new messages