Jira (FACT-2956) Facter should ensure core facts are resolved before loading custom facts

[Josh Cooper (Jira)]

Josh Cooper created an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts Issue Type: Bug Assignee: Unassigned Created: 2021/03/03 11:21 AM Priority: Normal Reporter: Josh Cooper Facter 4 calls Kernel.load on all custom ruby facts before core facts have been resolved. If the custom ruby facts has code outside a setcode block, then it's possible for core facts like Facter.value(:operatingsystem) to return nil, like FACT-2880. It is true that custom facts shouldn't misbehave that way, but a single misbehaving custom fact can prevent PE from managing itself. We also don't have tooling/validation to warn or prevent users from doing this, so there are many examples of it in practice. We should be more defensive about this. One way would be to ensure core facts are resolved before custom facts are loaded. It would still be possible for a custom fact with a higher weight to override the core fact, but at least the core fact would never have a nil value. Facter 3 didn't have this issue because core facts were implemented in C++ and evaluated before loading custom ruby facts. Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Mihai Buzgau (Jira)]

Mihai Buzgau updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts

Change By: Mihai Buzgau Story Points: 5

Add Comment

[Mihai Buzgau (Jira)]

Mihai Buzgau updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts

Change By: Mihai Buzgau Sprint: NW - 2021-03-17

Add Comment

[Gheorghe Popescu (Jira)]

Gheorghe Popescu assigned an issue to Gheorghe Popescu

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts

Change By: Gheorghe Popescu Assignee: Gheorghe Popescu

Add Comment

[Gheorghe Popescu (Jira)]

Gheorghe Popescu commented on FACT-2956

Re: Facter should ensure core facts are resolved before loading custom facts Hello, We compared Facter 3 and Facter 4 and found an issue in the way `Facter.value` is implemented. On Facter 3, search is done in multiple steps, and the next step is executed only if the previous one was not able to resolve the fact: search in the internal collection load the `fact_name.rb` from the configured custom directories load all the core facts, external facts and env facts load all custom facts Calling `Facter.value(:os)` will: search for `os` in its internal collection load `custom_dir/os.rb` load core, external and env facts Calling `Facter.value(:custom_fact)` will: search for `custom_fact` in its internal collection load `custom_dir/custom_fact.rb` load core, external and env facts load all custom facts On Facter 4, when `Facter.value` is called, Facter will load all the core facts and all the custom facts, then will resolve only the facts that match the requested query. We are working on porting the same searching mechanism form Facter 3 to Facter 4 which is preventing the load of all custom facts when a core fact is requested.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-2956

Facter should ensure core facts are resolved before loading custom facts

Change By: Josh Cooper Fix Version/s: FACT 4.0.52

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts Change By: Josh Cooper

Facter 4 calls {{Kernel.load}} on all custom ruby facts before core facts have been resolved. If the custom ruby facts has code outside a {{setcode}} block, then it's possible for core facts like {{Facter.value(:operatingsystem)}} to return nil, like FACT-2880. It is true that custom facts shouldn't misbehave that way, but a single misbehaving custom fact can prevent PE from managing itself. We also don't have tooling/validation to warn or prevent users from doing this, so there are many examples of it in practice. We should be more defensive about this. One way would be to ensure core facts are resolved before custom facts are loaded. It would still be possible for a custom fact with a higher weight to override the core fact, but at least the core fact would never have a nil value. Facter 3 didn't have this issue because core facts were implemented in C++ and evaluated before loading custom ruby facts.

Due to this issue, puppet may not be able to manage services, because puppet selects {{base}} as the default service provider instead of {{systemd}}: {noformat} change from 'stopped' to 'running' failed: Services must specify a start command or a binary (corrective) {noformat}

Add Comment

[Josh Cooper (Jira)]

I believe this issue is only triggered when running {{facter --puppet}}. It can happen when running {{puppet agent -t}} if there is a custom or external fact that shells out to {{facter --puppet}}

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts Change By: Josh Cooper

Facter 4 calls {{Kernel.load}} on all custom ruby facts before core facts have been resolved. If the custom ruby facts has code outside a {{setcode}} block, then it's possible for core facts like {{Facter.value(:operatingsystem)}} to return nil, like FACT-2880. It is true that custom facts shouldn't misbehave that way, but a single misbehaving custom fact can prevent PE from managing itself. We also don't have tooling/validation to warn or prevent users from doing this, so there are many examples of it in practice. We should be more defensive about this. One way would be to ensure core facts are resolved before custom facts are loaded. It would still be possible for a custom fact with a higher weight to override the core fact, but at least the core fact would never have a nil value. Facter 3 didn't have this issue because core facts were implemented in C++ and evaluated before loading custom ruby facts. Due to this issue, puppet may not be able to manage services, because puppet selects {{base}} as the default service provider instead of {{systemd}}: {noformat} change from 'stopped' to 'running' failed: Services must specify a start command or a binary (corrective) {noformat}

I believe this This issue is only can be triggered when running {{ puppet facts show}}: {noformat} # mkdir -p /etc/puppetlabs/code/environments/production/modules/badfact/lib/ facter # cat <<END > /etc/puppetlabs/code/environments/production/modules/badfact/lib/facter/service.rb require 'puppet' Puppet::Type.type(:service).new(:name => 'something') END # rm - - rf /opt/puppetlabs/ puppet /cache/lib/ # puppet facts show Error: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass Error: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass Error: Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass Error: Facter: error while resolving custom facts in /etc/puppetlabs/code/environments/production/modules/badfact/lib/facter/service.rb Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass {noformat } If the agent has pluginsynced, then {{facter -p } } will also trigger the issue: {noformat} # facter -p [2021-03-05 20:03:49 . It can happen when running 595880 ] ERROR Facter - error while resolving custom facts in /opt/puppetlabs/puppet/cache/lib/facter/service.rb Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass { noformat} However, { puppet agent {facter - t p <name> }} if there is a custom or external for an unrelated fact that shells out to won't trigger it. { { noformat} # facter - -puppet p os {   architecture => "x86_64",   family => "RedHat",   hardware => "x86_64",   name => "RedHat",   release => {     full => "8.0",     major => "8",     minor => "0" } ,   selinux => {     enabled => false } } {noformat}

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts

Change By: Josh Cooper Attachment: these-are-not-the-droids-youre-looking-for.png

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on FACT-2956

Re: Facter should ensure core facts are resolved before loading custom facts I think the issue Kevin and Charlie are concerned about is actually fixed in FACT-2937. The problem isn't that core facts aren't available when the custom fact is evaluated. It's that the init provider uses a legacy fact to confine its suitability and facter was filtering out the custom fact, even when requested via Facter.value. This is true for both facter -p and puppet facts show: [root@jerky-luxury ~]# puppet --version 7.4.1 [root@jerky-luxury ~]# cat /opt/puppetlabs/puppet/cache/lib/facter/boom.rb

require 'puppet' Puppet::Type.type(:service).new(:name => 'something')

[root@jerky-luxury ~]# facter -p | head -1 [2021-03-06 00:19:30.893599 ] ERROR Facter - error while resolving custom facts in /opt/puppetlabs/puppet/cache/lib/facter/boom.rb Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass aio_agent_version => 7.4.1 # puppet facts show | head -1

Error: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass Error: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass Error: Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass

Error: Facter: error while resolving custom facts in /opt/puppetlabs/puppet/cache/lib/facter/boom.rb Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass I can eliminate the issue doing any of the following: explicitly asking for legacy facts [root@jerky-luxury ~]# facter -p --show-legacy | head -1 aio_agent_version => 7.4.1 using facter#main $ bx facter -p | head -1 [2021-03-05 16:21:29.017042 ] ERROR Facter - error while resolving custom facts in /Users/josh/.puppetlabs/opt/puppet/cache/lib/facter/bad.rb Could not autoload puppet/type/service: Could not autoload puppet/provider/service/launchd: undefined method `downcase' for nil:NilClass aio_agent_version => 5.5.21 $ FACTER_LOCATION=file:///Users/josh/work/facter bundle exec facter -p | head -1 aio_agent_version => 5.5.21 changing the init confine logic to use a structured fact instead os = Facter.value('os.name').downcase family = Facter.value('os.family').downcase So I think this ticket is an issue, but it's not the issue we were thinking. Let's chat Monday to discuss

Add Comment

[Gheorghe Popescu (Jira)]

Gheorghe Popescu updated an issue

Facter / FACT-2956

Facter should ensure core facts are resolved before loading custom facts

Change By: Gheorghe Popescu   Facter 4 calls {{Kernel.load}} on all custom ruby facts before core facts have been resolved. If the custom ruby facts has code outside a {{setcode}} block, then it's possible for core facts like {{ implements ` Facter.value (:operatingsystem)}} to return nil, like FACT-2880 ` differently than Facter 3 .

It is true that custom facts shouldn't misbehave that way, but a single misbehaving custom fact can prevent PE from managing itself. We also don't have tooling/validation to warn or prevent users from doing this, so there are many examples of it in practice.   We should be more defensive about this. One way would be to ensure core facts are resolved before custom facts are loaded. It would still be possible for a custom fact with a higher weight to override the core fact, but at least the core fact would never have a nil value.   Facter 3 didn't have this issue because core facts were implemented in C++ and evaluated before loading custom ruby facts. Due to this issue, puppet may not be able to manage services, because puppet selects {{base}} as the default service provider instead of {{systemd}}: {noformat} change from 'stopped' to 'running' failed: Services must specify a start command or a binary (corrective) {noformat}

This issue can be triggered when running {{puppet facts show}}: {noformat}

# mkdir -p /etc/puppetlabs/code/environments/production/modules/badfact/lib/facter # cat <<END > /etc/puppetlabs/code/environments/production/modules/badfact/lib/facter/service.rb

require 'puppet' Puppet::Type.type(:service).new(:name => 'something')

END # rm -rf /opt/puppetlabs/puppet/cache/lib/ # puppet facts show

Error: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass Error: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass Error: Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass

Error: Facter: error while resolving custom facts in /etc/puppetlabs/code/environments/production/modules/badfact/lib/facter/service.rb Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass {noformat}

If the agent has pluginsynced, then {{facter -p}} will also trigger the issue: {noformat}

# facter -p [2021-03-05 20:03:49.595880 ] ERROR Facter - error while resolving custom facts in /opt/puppetlabs/puppet/cache/lib/facter/service.rb Could not autoload puppet/type/service: Could not autoload puppet/provider/service/bsd: Could not autoload puppet/provider/service/init: undefined method `downcase' for nil:NilClass {noformat} However, {{facter -p <name>}} for an unrelated fact won't trigger it. {noformat} # facter -p os

{   architecture => "x86_64",   family => "RedHat",   hardware => "x86_64",   name => "RedHat",   release => {     full => "8.0",     major => "8",     minor => "0"   },   selinux => {     enabled => false   } } {noformat}

Add Comment

[Gheorghe Popescu (Jira)]

Gheorghe Popescu updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts Change By: Gheorghe Popescu

Facter 4 implements `Facter.value` differently than Facter 3.

On Facter 3, search is done in multiple steps, and the next step is executed only if the previous one was not able to resolve the fact:

# search in the internal collection # load the `fact_name.rb` from the configured custom directories # load all the core facts, external facts and env facts # load all custom facts Calling `Facter.value(:os)` will: # search for `os` in its internal collection # load `custom_dir/os.rb` # load core, external and env facts Calling `Facter.value(:custom_fact)` will: # search for `custom_fact` in its internal collection # load `custom_dir/custom_fact.rb` # load core, external and env facts # load all custom facts

On Facter 4, when `Facter.value` is called, Facter will load all the core facts and all the custom facts, then will resolve only the facts that match the requested query.

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts

Change By: Josh Cooper Fix Version/s: FACT 4.0.52 Fix Version/s: FACT 4.0.53

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts Change By: Josh Cooper

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts

Change By: Josh Cooper Attachment: these-are-not-the-droids-youre-looking-for.png

Add Comment

[Claire Cadman (Jira)]

Claire Cadman updated an issue

Facter / FACT-2956 Facter should ensure core facts are resolved before loading custom facts

Change By: Claire Cadman Labels: doc_reviewed

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)