Jira (FACT-2958) Fact names with dots shouldn't be converted to structured facts

46 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Mar 4, 2021, 11:10:01 PM3/4/21
to puppe...@googlegroups.com
Josh Cooper created an issue
 
Facter / Bug FACT-2958
Fact names with dots shouldn't be converted to structured facts
Issue Type: Bug Bug
Affects Versions: FACT 4.0.51
Assignee: Unassigned
Created: 2021/03/04 8:09 PM
Priority: Normal Normal
Reporter: Josh Cooper

In Facter 3, dots are legal characters in fact names, so "a.b" is a valid fact name. The fact can then be used in a puppet manifest using $facts['a.b'].

In Facter 4, dots in custom and external facts cause the fact to be converted to a structured fact. This breaks any puppet code that tries to resolve the fact:

For example, given custom and executable external facts:

mkdir -p /etc/facter/facts.d/
cat <<END > /etc/facter/facts.d/dot.sh
#!/bin/sh
echo exe.bar=baz
END
chmod u+x /etc/facter/facts.d/dot.sh
mkdir -p /opt/puppetlabs/puppet/cache/lib/facter
cat <<END > /opt/puppetlabs/puppet/cache/lib/facter/dot.rb
Facter.add('foo.bar') do
  setcode { 'baz' }
end
END

In 6.x, we can lookup the fact using its dotted name:

[root@beige-dread ~]# puppet apply -e 'notice($facts["exe.bar"])'
Notice: Scope(Class[main]): baz
[root@beige-dread ~]# puppet apply -e 'notice($facts["foo.bar"])'
Notice: Scope(Class[main]): baz

In 7.x, the facts cannot be resolved:

root@debatable-swing ~]# puppet apply -e 'notice($facts["foo.bar"])'
Notice: Scope(Class[main]):
[root@debatable-swing ~]# puppet apply -e 'notice($facts["exe.bar"])'
Notice: Scope(Class[main]):

Instead you have to access it as a structured fact explicitly or dig for the value:

[root@debatable-swing ~]# puppet apply -e 'notice($facts["exe"]["bar"])'
Notice: Scope(Class[main]): baz
[root@debatable-swing ~]# puppet apply -e 'notice($facts["foo"]["bar"])'
Notice: Scope(Class[main]): baz

I am thinking Facter 4's Facter.add method shouldn't convert facts to structured by default when the name contains dots. Perhaps allow an option to specify it's defining a structured fact, such as Facter.add("a.b", structured: true) do .. end

About external facts, it seems like we already have ways of providing structured facts as YAML or JSON so I'm not sure there is much benefit to converting key-value pairs into structured facts, such as my_org.my_group.my_fact1 = fact1_value

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Bogdan Irimie (Jira)

unread,
Mar 8, 2021, 4:36:03 AM3/8/21
to puppe...@googlegroups.com
Bogdan Irimie commented on Bug FACT-2958
 
Re: Fact names with dots shouldn't be converted to structured facts

➜ facter git:(main) ✗ cat /Users/bogdan.irimie/projects/facter/custom_facts/my_custom_fact.rb
# frozen_string_literal: true
 
Facter.add('foo.bar') do
  setcode { 'baz' }
end
 
2.6.3 :001 > require 'facter'
 => true
2.6.3 :002 > Facter.version
 => "4.0.52"
2.6.3 :003 > Facter.search('/Users/bogdan.irimie/projects/facter/custom_facts')
 => nil
2.6.3 :004 > Facter.value('foo.bar')
 => "baz"
2.6.3 :005 > exit

Josh Cooper (Jira)

unread,
Mar 8, 2021, 8:27:02 PM3/8/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
In Facter 3, dots are legal characters in fact names, so "a.b" is a valid fact name. The fact can then be used in a puppet manifest using {{$facts['a.b']}}.

In Facter 4, dots in custom and external facts cause the fact to be converted to a structured fact. This breaks any puppet code that tries to resolve the fact:

For example, given custom and executable external facts:

{noformat}

mkdir -p /etc/facter/facts.d/
cat <<END > /etc/facter/facts.d/dot.sh
#!/bin/sh
echo exe.bar=baz
END
chmod u+x /etc/facter/facts.d/dot.sh
mkdir -p /opt/puppetlabs/puppet/cache/lib/facter
cat <<END > /opt/puppetlabs/puppet/cache/lib/facter/dot.rb
Facter.add('foo.bar') do
  setcode { 'baz' }
end
END
{noformat}


In 6.x, we can lookup the fact using its dotted name:

{noformat}

[root@beige-dread ~]# puppet apply -e 'notice($facts["exe.bar"])'
Notice: Scope(Class[main]): baz
[root@beige-dread ~]# puppet apply -e 'notice($facts["foo.bar"])'
Notice: Scope(Class[main]): baz
{noformat}


In 7.x, the facts cannot be resolved:

{noformat}

root@debatable-swing ~]# puppet apply -e 'notice($facts["foo.bar"])'
Notice: Scope(Class[main]):
[root@debatable-swing ~]# puppet apply -e 'notice($facts["exe.bar"])'
Notice: Scope(Class[main]):
{noformat}


Instead you have to access it as a structured fact explicitly or dig for the value:

{noformat}

[root@debatable-swing ~]# puppet apply -e 'notice($facts["exe"]["bar"])'
Notice: Scope(Class[main]): baz
[root@debatable-swing ~]# puppet apply -e 'notice($facts["foo"]["bar"])'
Notice: Scope(Class[main]): baz
{noformat}

I am thinking Facter 4's {{Facter.add}} method shouldn't convert facts to structured by default when the name contains dots.
- Perhaps allow an option to specify it's defining a structured fact, such as {{Facter.add("a.b", structured: true) do .. end}} -

About external facts, it seems like we already have ways of providing structured facts as YAML or JSON so I'm not sure there is much benefit to converting key-value pairs into structured facts, such as {{my_org.my_group.my_fact1 = fact1_value}}

Josh Cooper (Jira)

unread,
Mar 8, 2021, 8:57:03 PM3/8/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-2958
 
Re: Fact names with dots shouldn't be converted to structured facts

As with many things in puppet, the issue of dotted fact names has a long, long history...

The VRA plugin makes use of dots in fact names, see https://github.com/puppetlabs/puppetlabs-stdlib/blob/a26d0c59cee2b946893d436c74c35b83cd47ee3c/lib/puppet/functions/fact.rb#L17. Confirmed this with Reid Vandewiele.

Facts are added as variables to puppet's topscope, eg:

$ puppet apply -e 'notice($osfamily)'
Notice: Scope(Class[main]): RedHat
$ puppet apply -e 'notice($os)'
Notice: Scope(Class[main]): {architecture => x86_64, family => RedHat, hardware => x86_64, name => RedHat, release => {full => 7.2, major => 7, minor => 2}, selinux => {enabled => false}}

However, puppet variables may not contain dots https://puppet.com/docs/puppet/7.4/lang_reserved.html#variable-names, so trying to access $os.family won't work. This is because in the puppet language, dot is a method call, see https://puppet.com/docs/puppet/7.4/lang_facts_accessing.html

Because of ambiguity with function invocation, the dot-separated access syntax that is available in Facter commands is not available with the $facts hash access syntax.

To access structured facts in 6.x, you can use the builtin get function (which supersedes the fact function in stdlib). Since "foo.bar" is not structured in 6, the result is nil

[root@arabic-ideology ~]# puppet apply -e 'notice($facts.get("os.family"))'
Notice: Scope(Class[main]): RedHat
[root@arabic-ideology ~]# puppet apply -e 'notice($facts.get("foo.bar"))'
Notice: Scope(Class[main]):

In puppet7, foo.bar is structured:

[root@greater-fantasy ~]# puppet apply -e 'notice($facts.get("os.family"))'
Notice: Scope(Class[main]): RedHat
[root@greater-fantasy ~]# puppet apply -e 'notice($facts.get("foo.bar"))'
Notice: Scope(Class[main]): baz

The issue of dotted facts has a large downstream impact, see https://tickets.puppetlabs.com/browse/FACT-932 for details.

As such I don't think facter should "auto promote" facts names with dots into structured facts. If someone wants to easily create a structured fact or add to an existing one, then there should be a different mechanism for opting into that behavior.

Josh Cooper (Jira)

unread,
Mar 10, 2021, 2:42:01 PM3/10/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Mar 17, 2021, 5:21:04 AM3/17/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Mar 17, 2021, 5:21:04 AM3/17/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-24.03.2021
Team: Ghost

Bogdan Irimie (Jira)

unread,
Mar 17, 2021, 10:58:03 AM3/17/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 18, 2021, 12:02:04 AM3/18/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 18, 2021, 12:05:03 AM3/18/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 18, 2021, 12:05:03 AM3/18/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 18, 2021, 12:06:03 AM3/18/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 18, 2021, 12:35:04 AM3/18/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-2958
 
Re: Fact names with dots shouldn't be converted to structured facts

Couple of follow up comments. Assume we have a custom fact:

Facter.add('b.dot') do
  setcode { 'dot' }
end

1. The console displays the fact from a puppet6/facter3 agent as:

The same fact from a puppet6/facter4 agent is displayed as a structured fact:

2. It's possible to query puppetdb for a node's fact using a dot in the URL for a puppet6 agent:

# curl -s -X POST http://localhost:8080/pdb/query/v4/nodes/agent6.platform9.puppet.net/facts/b.dot -H 'Content-Type:application/json' | jq -r .[0]
{
  "certname": "agent6.platform9.puppet.net",
  "environment": "production",
  "name": "b.dot",
  "value": "dot"
}

The same query for a puppet7 agent returns no results, and the query must be changed (the dot and everything after must be dropped):

# curl -s -X POST http://localhost:8080/pdb/query/v4/nodes/pe2021.platform9.puppet.net/facts/b.dot -H 'Content-Type:application/json' | jq -r .
[]
# curl -s -X POST http://localhost:8080/pdb/query/v4/nodes/pe2021.platform9.puppet.net/facts/b -H 'Content-Type:application/json' | jq -r .[0]
{
  "certname": "pe2021.platform9.puppet.net",
  "environment": "production",
  "name": "b",
  "value": {
    "dot": "dot"
  }
}

3. The PE console does NOT work with dotted facts that are not structured. It's possible to add a classification rule and quote the fact name, but as soon as you refresh the page, the quotes are lost and the rule no longer matches. So in practice if you want to use dots in fact names, then it must be a structured fact, the top-level component can't contain dots, and special characters within the path subcomponents must be quoted. For example, acme."product.version" would be a structured fact of the form:

{
  "acme" => { 
    "product.version" => "1.2.3" 
  }
}

4. The goal of Facter 4's Facter.add method was to provide an easy way to create structured facts. But that's already possible using the existing API:

Facter.add("acme") do
  setcode do
    { "product.version" => "1.2.3" }
  end
end

I suppose we could allow:

Facter.add("d.e", type: :structured) do
  setcode { "f" }
end

And that would represent the structured fact:

{
  "d" => {
    "e" => "f"
  }
}

But it gets confusing if the subpath component contains a dot, so then we'd need multiple quoting levels:

Facter.add("acme.'product.version'", :type => :structured) do
  setcode { "1.2.3" }
end

And at that point we have two ways to do the same thing.

So I'm inclined to revert the behavior and not add :type => :structured.

Gheorghe Popescu (Jira)

unread,
Mar 18, 2021, 8:33:04 AM3/18/21
to puppe...@googlegroups.com

Josh Cooper Bogdan Irimie
If I understand this correctly, the goal would be for Facter 4 to let the dotted keys from Facter.add intact(eg. Facter.add("a.b") will consider "a.b" as the fact key) and consider as structured facts the ones that return a Hash in setcode. Generally we won't consider . as e delimiter for structured facts.

# a normal fact
# Facter.value('a') => nil
# Facter.value('a.b') => 'c'
Facter.add('a.b') do
  setcode do
    'c'
  end
end
 
# a structured fact
# Facter.value('a') => { 'b' => 'c' }
# Facter.value('a.b') => 'c' (this will work only on Facter 4)
Facter.add('a') do
  setcode do
    { 'b' => 'c'}
  end
end

Then we would apply the same to the external facts, to be structured(json/yaml) but we won't consider . as a delimiter in fact names.
So the only thing that needs to happen in Facter 4 is not to split fact keys by . when creating the fact hierarchy.

Bogdan Irimie (Jira)

unread,
Mar 18, 2021, 1:26:04 PM3/18/21
to puppe...@googlegroups.com

Josh Cooper Gheorghe Popescu

I would like to point out some advantages Facter 4 structured custom/external facts have:

  • they can override core facts at any level of the fact hierarchy

e.g.

Facter.add('os.name', type: :structured) do
  has_weight(10)
  setcode do
    'my_custom_name'
  end
end

will override the core fact `os.name`.

os => {
  architecture => "x86_64",
  family => "Darwin",
  hardware => "x86_64",
  macosx => {
    build => "20D74",
    product => "macOS",
    version => {
      full => "11.2.1",
      major => "11.2",
      minor => "1"
    }
  },
  name => "my_custom_name",
  release => {
    full => "20.3.0",
    major => "20",
    minor => "3"
  }
}

  • they can augment any existing core/external/custom fact

e.g.

os => {
  architecture => "x86_64",
  family => "Darwin",
  hardware => "x86_64",
  macosx => {
    build => "20D74",
    product => "macOS",
    version => {
      full => "11.2.1",
      major => "11.2",
      minor => "1"
    }
  },
  name => "Darwin",
  release => {
    full => "20.3.0",
    major => "20",
    minor => "3"
  }
}

can be augmented with

Facter.add('os.popularity', type: :structured) do
  has_weight(10)
  setcode do
    '3'
  end
end

 
and the result will be:

os => {
  architecture => "x86_64",
  family => "Darwin",
  hardware => "x86_64",
  macosx => {
    build => "20D74",
    product => "macOS",
    version => {
      full => "11.2.1",
      major => "11.2",
      minor => "1"
    }
  },
  name => "Darwin",
  popularity => "3",
  release => {
    full => "20.3.0",
    major => "20",
    minor => "3"
  }
}

 

  • they can be blocked/cached individually

e.g. if we have

Facter.add('my_org.fact1', type: :structured) do
  has_weight(10)
  setcode do
    '1111111111'
  end
end
 
Facter.add('my_org.fact2', type: :structured) do
  has_weight(10)
  setcode do
    '2222222222'
  end
end
 
Facter.add('my_org.fact3', type: :structured) do
  has_weight(10)
  setcode do
    '3333333333'
  end
end

we can configure facter to block/cache each fact

facts : {
  blocklist : [ "my_org.fact1" ],
  ttls : [
    { "my_org.fact2": 30 days }
  ]
}

  • they provide a way to break your code in multiple units that are independent. If one fact fails, it will not affect the others

Facter.add('my_org.fact1', type: :structured) do
  has_weight(10)
  setcode do
    '1111111111'
  end
end
 
Facter.add('my_org.fact2', type: :structured) do
  has_weight(10)
  setcode do
    nil.size?
  end
end
 
Facter.add('my_org.fact3', type: :structured) do
  has_weight(10)
  setcode do
    '3333333333'
  end
end

the result will be

my_org => {
  fact1 => "1111111111",
  fact3 => "3333333333"
}

in contrast if we have

Facter.add('my_org') do
  has_weight(10)
  setcode do
    {
      "fact1" => "1111111111",
      "fact3" => nil.size,
      "fact3" =>  "3333333333"
    }
  end
end

we will get no `my_org` fact, although `my_org.fact1` and `my_org.fact3` can be resolved. Of course if we want this behaviour we can still structure facts like this and then they all get resolved or, if one fails, none get resolved.

Bogdan Irimie (Jira)

unread,
Mar 18, 2021, 1:34:03 PM3/18/21
to puppe...@googlegroups.com

The implementation proposed in https://github.com/puppetlabs/facter/pull/2308 allows all the old use cases (custom and external facts are not structured by default), but opens new possibilities, by making available all the use cases described above.

Josh Cooper (Jira)

unread,
Mar 18, 2021, 1:40:03 PM3/18/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-2958

Thanks Bogdan Irimie! The ability to define structured fact "units" and merge them together seems useful. So I'm  on keeping the feature, disabled by default like Facter 3, but can be opted into via type: :structured or a facter setting for external facts.

Bogdan Irimie (Jira)

unread,
Mar 24, 2021, 10:09:03 AM3/24/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
 
Change By: Bogdan Irimie
Sprint: ghost-24.03.2021 , ready for triage 3

Bogdan Irimie (Jira)

unread,
Mar 30, 2021, 3:01:04 AM3/30/21
to puppe...@googlegroups.com
Bogdan Irimie assigned an issue to Gheorghe Popescu
Change By: Bogdan Irimie
Assignee: Bogdan Irimie Gheorghe Popescu

Josh Cooper (Jira)

unread,
Apr 6, 2021, 1:48:03 PM4/6/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Flagged: Impediment
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Mihai Buzgau (Jira)

unread,
Apr 7, 2021, 3:23:03 AM4/7/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Apr 7, 2021, 9:04:03 AM4/7/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-24.03.2021, ghost-7.04.2021 , HAHA/Grooming 2

Bogdan Irimie (Jira)

unread,
Apr 7, 2021, 11:10:03 AM4/7/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-24.03.2021, ghost-7.04.2021 , ghost-21.04.2021

Josh Cooper (Jira)

unread,
Sep 15, 2022, 2:52:02 PM9/15/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-2958
 
Re: Fact names with dots shouldn't be converted to structured facts

The breaking behavior described in this ticket was reverted in Facter 4.x, so by default it handles dotted fact names like it did in Facter 3. So I'm going to close this ticket.

There is a related ticket FACT-3000 describing how dotted fact names should behave.

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages