Jira (PUP-7507) puppet lookup --facts <json file> can't modify $facts hash

6 views
Skip to first unread message

Nicholas Fagerlund (JIRA)

unread,
May 9, 2017, 8:49:03 PM5/9/17
to puppe...@googlegroups.com
Nicholas Fagerlund created an issue
 
Puppet / Bug PUP-7507
puppet lookup --facts <json file> can't modify $facts hash
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2017/05/09 5:48 PM
Labels: hiera
Priority: Normal Normal
Reporter: Nicholas Fagerlund

I was trying to simulate a change to my $facts['networking'] fact, so I ran facter --json > ~/Desktop/facts.json, edited the appropriate spot, and tried to pass that to puppet lookup --facts. No dice: my change was ignored.

That's because I was using facts.networking in my hierarchy, and apparently --facts doesn't actually set the $facts variable — it just sets top-scope variables with names from the JSON.

  • Problem one: That's unexpected, and it interferes with using facter --json as a basis for simulating other nodes.

So, I tried wrapping the facts json in another object with only a "facts": key. Which resulted in the following error: "Error: Could not run: Attempt to assign to a reserved variable name: 'facts'"

  • Problem two: If your hierarchy uses the $facts variable like we recommend (instead of ::factname vars), it's impossible to simulate any changes with the puppet lookup CLI. I expect this blocks setting other values for $trusted or $server_facts, as well.

The first problem goes into the bucket of things we need to discuss around puppet lookup's UX. The second problem seems like a straight-up defect.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
May 15, 2017, 5:00:02 AM5/15/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Component/s: Hiera & Lookup

Josh Cooper (JIRA)

unread,
May 16, 2017, 6:35:03 PM5/16/17
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Labels: hiera  triaged

Josh Cooper (JIRA)

unread,
May 16, 2017, 6:35:04 PM5/16/17
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

puppet lookup --facts <file> should consume facts that are generated from facter --json

Moses Mendoza (JIRA)

unread,
May 18, 2017, 2:07:36 PM5/18/17
to puppe...@googlegroups.com
Moses Mendoza updated an issue
 
Change By: Moses Mendoza
Team: Agent

Henrik Lindberg (JIRA)

unread,
Oct 25, 2017, 7:15:03 AM10/25/17
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

A unit or acceptance test that shows the problems (or that asserts that they are fixed?) would be great.

Craig Dunn (JIRA)

unread,
Nov 16, 2017, 4:06:03 AM11/16/17
to puppe...@googlegroups.com
Craig Dunn commented on Bug PUP-7507

I'm seeing this problem too.

Configuration of Hiera

[root@puppet /]# cat /etc/puppetlabs/code/environments/production/hiera.yaml
 
 
 
---
 
version: 5
 
 
 
defaults: 
  datadir: data
 
  data_hash: yaml_data
 
 
 
hierarchy:
 
  - name: "env"
 
    path: "%{facts.myenv}.yaml"

The data

[root@puppet /]# cat /etc/puppetlabs/code/environments/production/data/uat.yaml
 
---
 
ntp::servers: 1.2.3.4

The faked scope

Using facts file:

[root@puppet /]# cat facts.yaml
 
---
 
myenv: uat

Reproducing

When I perform the lookup I expect the fact 'myenv' to be set to 'uat' but the fact is not interpolated:

[root@puppet /]# puppet lookup ntp::servers --environment production --facts ./facts.yaml  --explain
 
...
 
    Hierarchy entry "env"
 
      Path "/etc/puppetlabs/code/environments/production/data/.yaml"
 
        Original path: "%{facts.myenv}.yaml"
 
        Path not found

Remediation

When I modify the hierarchy in the Hiera config to use top level vars....

...
 
    path: "%{::myenv}.yaml"
 
...

The lookup now resolves.

[root@puppet /]# puppet lookup ntp::servers --environment production --facts ./facts.yaml  --explain
 
...
 
    Hierarchy entry "env"
 
      Path "/etc/puppetlabs/code/environments/production/data/uat.yaml"
 
        Original path: "%{::myenv}.yaml"
 
        Found key: "ntp::servers" value: "1.2.3.4"

This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

Gerhardus Geldenhuis (JIRA)

unread,
Nov 16, 2017, 4:40:04 AM11/16/17
to puppe...@googlegroups.com

I can confirm that the same problem exists for $trusted hash. Passing trusted facts via --facts gets ignored when running puppet lookup from the command line.

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 4:56:04 AM11/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
 
Change By: Henrik Lindberg
Fix Version/s: PUP 5.4.0

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 4:56:05 AM11/16/17
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Thomas Hallgren
Change By: Henrik Lindberg
Assignee: Thomas Hallgren

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 4:57:04 AM11/16/17
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

Thanks for the reproducer Craig Dunn! That helps us look into this.

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 5:05:03 AM11/16/17
to puppe...@googlegroups.com

Depending on what the issue is, I would like this fixed on 4.x.

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 5:05:04 AM11/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Fix Version/s: PUP 4.10.z

Thomas Hallgren (JIRA)

unread,
Nov 16, 2017, 11:42:04 AM11/16/17
to puppe...@googlegroups.com
Thomas Hallgren commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

Solving the problem properly setting the facts hash is relatively easy but I have a question regarding the expectations on trusted hash.

I tested by passing a

{ 'clientcert' => 'some.domain.name' }

as a fact and by doing so, the explain for the 'trusted' value outputs:

            Found key: "trusted" value: {
 
              "authenticated" => "local",
 
              "certname" => "some.domain.name",
 
              "extensions" => {},
 
              "hostname" => "some",
 
              "domain" => "domain.name'"
 
            }

This looks OK to me. Gerhardus Geldenhuis, do you expect to be able to pass trusted facts verbatim in the facts file or is what I just described OK?

Thomas Hallgren (JIRA)

unread,
Nov 16, 2017, 11:46:03 AM11/16/17
to puppe...@googlegroups.com
Thomas Hallgren assigned an issue to Henrik Lindberg
 
Change By: Thomas Hallgren
Assignee: Thomas Hallgren Henrik Lindberg

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 12:00:19 PM11/16/17
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Unassigned
Change By: Henrik Lindberg
Assignee: Henrik Lindberg

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 12:01:04 PM11/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Release Notes Summary: The {{lookup}} command line tool did not set facts given on the command line such that they ended up in {{$facts}} variable.
Release Notes: Bug Fix

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 12:18:04 PM11/16/17
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Sprint: Platform Core KANBAN

Gerhardus Geldenhuis (JIRA)

unread,
Nov 16, 2017, 12:19:06 PM11/16/17
to puppe...@googlegroups.com
Gerhardus Geldenhuis commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

Thomas Hallgren
I was expecting the following to be made available to my hiera hierarchy via a custom file passed via --facts :
```
—
trusted:
extensions:
Company: Puppet
pp_zone: INTERNAL
pp_datacenter: London
pp_environment: Production
```

I am using a number of custom extensions and an extract from my hierarchy looks as follows:
hierarchy:
```
-name: "Company Group Name"
path: "company/%

{trusted.extensions.Company}

.yaml"
-name: "Datacenter"
path: "datacenter/%

{trusted.extensions.pp_datacenter}

.yaml"
```

I have named my custom extensions using $confdir/custom_trusted_oid_mapping.yaml but that should not matter.

The original intend was to use puppet lookup as a cheaper way of testing and verifying my hierarchy since currently beaker( which is probably better) has a bit of a steep learning curve, not helped by very poor documentation.

Henrik Lindberg (JIRA)

unread,
Nov 16, 2017, 12:28:03 PM11/16/17
to puppe...@googlegroups.com

Thomas Hallgren I think we ended up with the idea of allowing the key trusted in the given facts, to be treated as the information that ends up in $trusted - that would be the same way this gets done when a node's data is obtained from PDB. Not sure that is tested though.

Gerhardus Geldenhuis (JIRA)

unread,
Nov 16, 2017, 12:31:04 PM11/16/17
to puppe...@googlegroups.com

Perhaps to be absolutely clear, my expectation is that I should be able to fully simulate an actual hiera lookup with a combination of facts provided via --facts which should be compatible with what I would get if I did a facter -y or -j and also be compatible with the output of any fact hashes like trusted and lastly combine anything remaining from PuppetDB if available.

Melissa Stone (JIRA)

unread,
Nov 17, 2017, 12:15:05 PM11/17/17
to puppe...@googlegroups.com
Melissa Stone commented on Bug PUP-7507

This has passed CI

Eric Delaney (JIRA)

unread,
Nov 20, 2017, 12:10:04 PM11/20/17
to puppe...@googlegroups.com
Eric Delaney updated an issue
 
Change By: Eric Delaney
QA Risk Assessment Reason: We should make a test for this in puppet and facter
QA Risk Assessment: Needs Assessment Automate

Eric Delaney (JIRA)

unread,
Nov 27, 2017, 1:57:02 PM11/27/17
to puppe...@googlegroups.com
Eric Delaney assigned an issue to Eric Delaney
Change By: Eric Delaney
Assignee: Eric Delaney

Eric Delaney (JIRA)

unread,
Nov 28, 2017, 2:49:04 PM11/28/17
to puppe...@googlegroups.com
Eric Delaney commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

Tested on SHA=5db2a7ea45c377429a62761b401ca066b55f15b6 SUITE_VERSION=1.10.8.116.g5db2a7e

Using the reproduce above:

[root@xltdnqmuemyejo7 ~]# puppet lookup ntp::servers --environment production --facts ./facts.yaml
 
--- 1.2.3.4
 
...
 
[root@xltdnqmuemyejo7 ~]#

Gerhardus Geldenhuis (JIRA)

unread,
Nov 28, 2017, 2:59:03 PM11/28/17
to puppe...@googlegroups.com

Eric Delaney does this also resolve the issue for trusted facts?

Thomas Hallgren (JIRA)

unread,
Nov 28, 2017, 4:37:04 PM11/28/17
to puppe...@googlegroups.com

The PR does not address trusted facts (it was created before Gerhardus Geldenhuis replied to my question).

Henrik Lindberg, I didn't perceive it as we reach a conclusion. Some of the trusted facts are derived from other facts and it's not trivial to just allow them to be overridden. It also raises a couple of questions. What would it meant to provide both

{ 'clientcert' => 'some.domain.name' }

and

{'trusted.domain' => 'other.name'}

?
Also, should be allowed to override 'trusted.authenticated'?

A middle ground could be to allow 'trusted.extensions' since they don't seem to be derived from anything.

Eric Delaney (JIRA)

unread,
Nov 28, 2017, 5:56:04 PM11/28/17
to puppe...@googlegroups.com
Eric Delaney commented on Bug PUP-7507

Gerhardus Geldenhuis Could I get you to create a separate ticket for the 'trusted' case and what you see as the problem. That way we won't lose the issue and we can work on it there.

Henrik Lindberg (JIRA)

unread,
Nov 29, 2017, 5:59:02 AM11/29/17
to puppe...@googlegroups.com

Thomas Hallgren Does it actually support dotted keys to provide values inside hashes? As I imagine this working is that you would have to provide the entire hash for trusted (as a fact) - the same way as it gets loaded from PDB when a node's "parameters" are read.
It is ok to allow manipulation of trusted information since user must be on the master to run this and need to have a way to do "what-if" including trusted information.

Thomas Hallgren (JIRA)

unread,
Nov 29, 2017, 7:26:02 AM11/29/17
to puppe...@googlegroups.com

No, my example is wrong. Should read:

{ 'trusted => { domain' => 'other.name'} }

The real problem is that much of the trusted information is derived from other facts and, for obvious reasons, not designed to be modified once it's set. We need to specify how ambiguities are resolved when explicitly given values conflict with derived ones. Which take precedence? Is it an error?

Henrik Lindberg (JIRA)

unread,
Nov 29, 2017, 11:10:02 AM11/29/17
to puppe...@googlegroups.com

You need to specify all of trusted for it to work - don't imagine that we will start supporting deep merge in this context.

Gerhardus Geldenhuis (JIRA)

unread,
Nov 29, 2017, 12:17:04 PM11/29/17
to puppe...@googlegroups.com

Thomas Hallgren my view would be that if you specify a fact via puppet lookup it should supersede everything else. I view --facts as primarily a debugging tool. It should override anything else that is already set and as long as the behaviour clearly documented then it is not a problem. Henrik Lindberg if you need to specify all of trusted to work then that needs to be very clearly documented. My assumption with specifying additional facts with --facts is that it would override and merge. I was not aware that it was an all or nothing approach.

My view with this has been that it is a debugging aide to be able to specify facts with --facts if you have to build a large amount of "scaffolding" to get it to work properly/correctly then its usefulness diminishes. It might very well be that the intent of the tool is entirely different but then that should be made really clear in the documentation.

Eric Delaney (JIRA)

unread,
Nov 29, 2017, 1:17:05 PM11/29/17
to puppe...@googlegroups.com
Eric Delaney assigned an issue to Henrik Lindberg
 
Change By: Eric Delaney
Assignee: Eric Delaney Henrik Lindberg

Henrik Lindberg (JIRA)

unread,
Nov 29, 2017, 1:28:03 PM11/29/17
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

We could use --facts to simply set the facts (including trusted). That naturally requires specifying all the values you want.
We could also add a --merge_facts to use deep merge to merge values into facts obtained for the node. That could be useful in debugging a small change that has not yet taken effect on the node since it allows you to specify only the fact that you need to override).
(Alternatively always specify facts with --facts and let --merge_facts be a boolean flag).

Henrik Lindberg (JIRA)

unread,
Nov 29, 2017, 1:34:03 PM11/29/17
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Unassigned
Change By: Henrik Lindberg
Assignee: Henrik Lindberg

Thomas Hallgren (JIRA)

unread,
Nov 29, 2017, 3:58:03 PM11/29/17
to puppe...@googlegroups.com
Thomas Hallgren commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

Gerhardus Geldenhuis, I agree that specifying a --facts file would supersede anything else. My concern is that some of the trusted facts are derived from other facts. In other words, what's provided in the --facts file might contradict itself. I'd like to know how that should be handled.

Henrik Lindberg, specifying all of trusted seems weird. Isn't the normal use-case that you would rely on the derived trusted facts and specify the extensions?

Henrik Lindberg (JIRA)

unread,
Nov 30, 2017, 4:24:03 AM11/30/17
to puppe...@googlegroups.com

I think we should open a separate ticket regarding the handling of trusted facts as this ticket is about a different specific problem that is now solved, and we need to move that along.

Henrik Lindberg (JIRA)

unread,
Nov 30, 2017, 4:27:02 AM11/30/17
to puppe...@googlegroups.com

Created PUP-8220 for the trusted facts discussion.

Henrik Lindberg (JIRA)

unread,
Nov 30, 2017, 4:28:03 AM11/30/17
to puppe...@googlegroups.com

Assigning to Eric Delaney (I think this ticket is now ready to be resolved, but delegating to let you decide).

Henrik Lindberg (JIRA)

unread,
Nov 30, 2017, 4:28:04 AM11/30/17
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Eric Delaney
 
Change By: Henrik Lindberg
Assignee: Eric Delaney

Branan Riley (JIRA)

unread,
Jan 23, 2018, 6:52:03 PM1/23/18
to puppe...@googlegroups.com
Branan Riley updated an issue
Change By: Branan Riley
Fix Version/s: PUP 4.10.z
Fix Version/s: PUP 5.3.4
Fix Version/s: PUP 4.10.10

Josh Cooper (Jira)

unread,
Apr 22, 2020, 7:29:03 PM4/22/20
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-7507
 
Re: puppet lookup --facts <json file> can't modify $facts hash

One problem with the way this was implemented is that the supplied facts are added to the node after classification. So you can reference the facts in your hiera config and manifests, but you can't override facts used for classification (trusted facts or otherwise). I'll file a separate ticket about that.
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages