Who found the bug? Open Source Puppet user Where was the bug found?
- System: KVM virtual machines not running in AWS
- Version: N/A
- Operating system(s): CentOS 7.9, Rocky Linux 8.4
- Puppet version: puppet-agent-6.25.0-1.el8.x86_64
What is malfunctioning? Puppet agent 6.24.0 uses facter 3.14.19, Puppet agent 6.25.0 uses facter 3.14.20 Diff between facter versions: https://github.com/puppetlabs/facter/compare/3.14.19...3.14.20 Facter added the ability to use a IMDSv2 token when querying ec2 metadata. However, this implementation does not include an explicit timeout and non-AWS KVM virtual machines it will hang for minutes. For comparison, running "time puppet facts --debug" using puppet-agent 6.24.0 takes 4.6 seconds, and the same command on puppet-agent 6.25.0 takes 131.7 seconds - over 2 minutes to gather facts!
# puppet-agent 6.24.0 |
$ puppet facts --debug |
|
Debug: Facter: executing command: /opt/puppetlabs/puppet/bin/virt-what |
Debug: Facter: kvm |
Debug: Facter: completed processing output: closing child pipes. |
Debug: Facter: process exited with status code 0. |
Debug: Facter: fact "is_virtual" has resolved to true. |
Debug: Facter: fact "virtual" has resolved to "kvm". |
Debug: Facter: not running under a Azure instance. |
Debug: Facter: resolving EC2 facts. |
Debug: Facter: querying EC2 instance metadata at http://169.254.169.254/latest/meta-data/. |
Debug: Facter: requesting http://169.254.169.254/latest/meta-data/. |
Debug: Facter: Trying 169.254.169.254:80... |
Debug: Facter: Connection timed out after 600 milliseconds |
Debug: Facter: Closing connection 0 |
Debug: Facter: EC2 facts are unavailable: not running under an EC2 instance or EC2 is not responding in a timely manner. |
Debug: Facter: resolving cloud facts. |
Debug: Facter: resolving cloud fact |
Debug: Facter: resolving GCE facts. |
Debug: Facter: not running under a GCE instance. |
# puppet-agent 6.25.0 |
$ puppet facts --debug |
|
Debug: Facter: executing command: /opt/puppetlabs/puppet/bin/virt-what |
Debug: Facter: kvm |
Debug: Facter: completed processing output: closing child pipes. |
Debug: Facter: process exited with status code 0. |
Debug: Facter: fact "is_virtual" has resolved to true. |
Debug: Facter: fact "virtual" has resolved to "kvm". |
Debug: Facter: not running under a Azure instance. |
Debug: Facter: resolving EC2 facts. |
Debug: Facter: requesting IMDSv2 token at http://169.254.169.254/latest/api/token. |
Debug: Facter: requesting http://169.254.169.254/latest/api/token. |
Debug: Facter: Trying 169.254.169.254:80... |
Debug: Facter: connect to 169.254.169.254 port 80 failed: Connection timed out |
Debug: Facter: Failed to connect to 169.254.169.254 port 80: Connection timed out |
Debug: Facter: Closing connection 0 |
Debug: Facter: EC2 IMDSv2 endpoint is unavailable |
Debug: Facter: querying EC2 instance metadata at http://169.254.169.254/latest/meta-data/. |
Debug: Facter: requesting http://169.254.169.254/latest/meta-data/. |
Debug: Facter: Trying 169.254.169.254:80... |
Debug: Facter: Connection timed out after 600 milliseconds |
Debug: Facter: Closing connection 1 |
Debug: Facter: EC2 facts are unavailable: not running under an EC2 instance or EC2 is not responding in a timely manner. |
Debug: Facter: resolving cloud facts. |
Debug: Facter: resolving cloud fact |
Debug: Facter: resolving GCE facts. |
Debug: Facter: not running under a GCE instance. |
What does success look like? Fix the regression introduced in facter 3.14.20 by using the same 600ms timeout when requesting a token. How will success be validated? On a KVM virtual machine outside of AWS running puppet-agent 6.25, executing the command "puppet facts" should take less than 5 seconds. Should anyone be contacted after this is fixed? Open Source Puppet 6.x needs to be updated with this fix. |
