Jira (FACT-2988) Facter takes 20+ seconds on Windows due to azure metadata query

26 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Mar 16, 2021, 8:55:03 PM3/16/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Facter / Bug FACT-2988
Facter takes 20+ seconds on Windows due to azure metadata query
Change By: Josh Cooper
The azure metadata fact takes 20+ seconds before timing out on Windows.

{noformat}
$ cmd /c facter --version
4.0.52

$ time cmd /c facter > /dev/null
real 0m22.431s
user 0m0.000s
sys 0m0.000s
{noformat}

When running with debug you can see:

{noformat}
[2021-03-17 01:49:16.036357 ] DEBUG Facter::Resolvers::Az - Querying Az metadata
[2021-03-17 01:49:37.076303 ] DEBUG Facter::Util::Resolvers::Http - Trying to connect to http://169.254.169.254/metadata/instance?api-version=2020-09-01 but got: execution expired
{noformat}

The EC2 fact uses the same IP address, so I'm not sure if the EC2 fact is blocked by default, or is excluded based on other criteria, or because the EC2 fact uses a very small connect timeout: [ https://github.com/puppetlabs/facter/blob/f169f54ddae91787b2f36a25d6e7cbc2330c5bbc/lib/facter/util/resolvers/http.rb#L10 ]

Redhat8 doesn't have this issue:

{noformat}
[2021-03-17 00:53:25.649111 ] DEBUG Facter::Resolvers::Az - Querying Az metadata
[2021-03-17 00:53:25.670289 ] DEBUG Facter::Util::Resolvers::Http - Trying to connect to http://169.254.169.254/metadata/instance?api-version=2020-09-01 but got: Failed to open TCP connection to 169.254.169.254:80 (Network is unreachable - connect(2) for "169.254.169.254" port 80)
{noformat}
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Mar 16, 2021, 8:55:04 PM3/16/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Summary: Facter is slow takes 20+ seconds on windows Windows due to azure metadata query

Gheorghe Popescu (Jira)

unread,
Mar 17, 2021, 3:40:03 AM3/17/21
to puppe...@googlegroups.com
Gheorghe Popescu commented on Bug FACT-2988
 
Re: Facter takes 20+ seconds on Windows due to azure metadata query

The EC2 fact is resolved only if hypervisor is AWS.

About the AZ metadata, i think something is funky with the http client on Windows, and it does not respect the set timeouts.
The following snippet can be used to reproduce the issue:

require 'net/http'
require 'uri'
require 'benchmark'
 
url = URI.parse("http://169.254.169.254/metadata/instance?api-version=2020-09-01")
http = Net::HTTP.new(url.host, url.port)
http.read_timeout = 1
http.open_timeout = 1
p Benchmark.realtime { 
  begin
    resp = http.start() { |http| http.get(url.path) }
  rescue Net::OpenTimeout => e
    puts e.message
  end 
}

Another thing, the errors are different, `Net::OpenTimeout` on Windows after 20 seconds, even though `open_timeout` is set to 5, `Errno::ENETUNREACH` on linux and the connection is closed immediately .

Mihai Buzgau (Jira)

unread,
Mar 17, 2021, 6:12:03 AM3/17/21
to puppe...@googlegroups.com

Mihai Buzgau (Jira)

unread,
Mar 17, 2021, 6:12:04 AM3/17/21
to puppe...@googlegroups.com

Mihai Buzgau (Jira)

unread,
Mar 17, 2021, 6:13:03 AM3/17/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 18, 2021, 12:55:04 AM3/18/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-2988
 
Re: Facter takes 20+ seconds on Windows due to azure metadata query

Is there a way to detect the hypervisor and skip the AZ metadata like we do on EC2?

It sounds like EC2 would have the same issue if it wasn't confined to only run on AWS?

Maybe we need to create a thread to do the Net::HTTP.start, then have the main thread join with a small wait time?

Gabriel Nagy (Jira)

unread,
Mar 18, 2021, 9:51:03 AM3/18/21
to puppe...@googlegroups.com
Gabriel Nagy commented on Bug FACT-2988

Yeah, for the az_metadata fact we only resolve it when we're on hyperv. However, for the cloud fact I mistakenly used the az resolver instead of the fact (which handled this), so it should be possible to do something similar here.

Gabriel Nagy (Jira)

unread,
Mar 18, 2021, 9:51:03 AM3/18/21
to puppe...@googlegroups.com

Gabriel Nagy (Jira)

unread,
Mar 18, 2021, 10:33:04 AM3/18/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Mar 30, 2021, 11:47:02 AM3/30/21
to puppe...@googlegroups.com

Claire Cadman (Jira)

unread,
Apr 13, 2021, 9:24:02 AM4/13/21
to puppe...@googlegroups.com
Claire Cadman updated an issue
Change By: Claire Cadman
Labels: doc_reviewed
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages