Jira (FACT-1919) blank/negative facter cache due to timed out EC2 metadata

3 views
Skip to first unread message

Gustavo Randich (JIRA)

unread,
May 3, 2019, 11:54:04 AM5/3/19
to puppe...@googlegroups.com
Gustavo Randich created an issue
 
Facter / Bug FACT-1919
blank/negative facter cache due to timed out EC2 metadata
Issue Type: Bug Bug
Affects Versions: FACT 3.y
Assignee: Unassigned
Created: 2019/05/03 8:53 AM
Priority: Normal Normal
Reporter: Gustavo Randich

We have a situation where our cloud metadata service (it's not AWS) takes longer than 5 seconds which puppet agent uses as metadata HTTP session timeout.

In our case, this almost always produces:

  • In the EC2 cache-group JSON file, ec2_userdata populated, but ec2_metadata not populated
  • Blank results for all ec2_metadata facts (not so for ec2_userdata), until TTL reached and fact resolution retried

Maybe this 'negative / partial-group' caching is by design, but as we cannot control the built-in EC2 facts timeout for the metadata queries, we cannot reliably populate the cache in a first/slow facter or agent run, thus turning the facter cache feature useless.

 

~# puppet agent --test  --noop
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Facter: EC2 metadata request failed: Timeout was reached

Notice: Caught INT; exiting

~# time facter ec2_metadata.ami-id

real 0m0.093s
user 0m0.082s
sys 0m0.009s

 

 
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Mihai Buzgau (JIRA)

unread,
May 14, 2019, 9:38:02 AM5/14/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - Triage

Mihai Buzgau (JIRA)

unread,
May 14, 2019, 9:52:04 AM5/14/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Team: Puppet Romania

Mihai Buzgau (JIRA)

unread,
Jul 10, 2019, 5:18:02 AM7/10/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Story Points: 3

Mihai Buzgau (JIRA)

unread,
Jul 10, 2019, 5:18:02 AM7/10/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - Triage 2019-07-23

Mihai Buzgau (JIRA)

unread,
Jul 24, 2019, 4:31:09 AM7/24/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-07-23 , NW - 2019-08-07

Mihai Buzgau (JIRA)

unread,
Aug 7, 2019, 4:38:07 AM8/7/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-07-23, NW - 2019-08-07 , NW - 2019-08-21

Gabriel Nagy (JIRA)

unread,
Aug 19, 2019, 8:00:03 AM8/19/19
to puppe...@googlegroups.com
Gabriel Nagy assigned an issue to Gabriel Nagy
Change By: Gabriel Nagy
Assignee: Gabriel Nagy

Gabriel Nagy (JIRA)

unread,
Aug 19, 2019, 8:00:03 AM8/19/19
to puppe...@googlegroups.com
Gabriel Nagy commented on Bug FACT-1919
 
Re: blank/negative facter cache due to timed out EC2 metadata

Hi Gustavo Randich,

It's not clear from the ticket description what's the desired fix here.

My understanding is that the EC2 metadata request works, but takes longer than 5 seconds and execution times out.

Would you prefer to have the timeout parameter configurable, or have it increased? How long does the lookup take?

Gustavo Randich (JIRA)

unread,
Aug 19, 2019, 10:10:03 AM8/19/19
to puppe...@googlegroups.com

Hi Gabriel Nagy,

Being able to configure the timeout would be useful, yes; but I find it sort of a workaround.

The 5 seconds is fine for the majority of facts, but we have one specific slow fact which when timed out on the first puppet agent run invalidates the complete facter caching group (EC2). To clarify: only one slow EC2 metadata URL (public-ipv4 in our OpenStack cloud) invalidated the cache of other metadata URLs which were not slow (i.e. ami-id), producing failed and erroneous puppet catalog compilations.

So, my real concern is about the "granularity" of cache invalidation: to avoid this:

  1. facter ec2_metadata.local-ipv4

  1. Error: Facter: EC2 metadata request failed: Timeout was reached
  1. facter ec2_metadata.ami-id
    <blank>

Thinking again about Increasing the timeout parameter: doing this because of only one "slow" fact may hide real performance problems in other facts.

Mihai Buzgau (JIRA)

unread,
Aug 21, 2019, 5:17:08 AM8/21/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-07-23, NW - 2019-08-07, NW - 2019-08-21 , NW - 2019-09-03

Gabriel Nagy (JIRA)

unread,
Aug 21, 2019, 8:23:02 AM8/21/19
to puppe...@googlegroups.com
Gabriel Nagy commented on Bug FACT-1919
 
Re: blank/negative facter cache due to timed out EC2 metadata

Due to the way resolvers are handled in facter, we can't have caching granularity for single facts, just cache groups (which in this case is EC2, containing both user and metadata). So if either the user-data or meta-data fact does not resolve, neither will be cached.

We can provide a configurable parameter just for the HTTP request timeout (which is only used inside the ec2_resolver so it wouldn't affect other facts).

Gustavo Randich (JIRA)

unread,
Aug 21, 2019, 9:15:03 AM8/21/19
to puppe...@googlegroups.com

That would be fine then.
Thank you

Gabriel Nagy (JIRA)

unread,
Aug 27, 2019, 8:14:03 AM8/27/19
to puppe...@googlegroups.com
Gabriel Nagy updated an issue
 
Change By: Gabriel Nagy
Release Notes Summary: Make the EC2 session timeout configurable via the {{EC2_SESSION_TIMEOUT}} environment variable (in milliseconds).

If the environment variable does not exist or is set to an invalid value, Facter defaults it to 5000 (5 seconds).
Release Notes: Enhancement

Mihai Buzgau (JIRA)

unread,
Sep 4, 2019, 5:17:07 AM9/4/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR - 2019-07-23, NW - 2019-08-07, NW - 2019-08-21, NW - 2019-09-03 , NW - 2019-09-18

Mihai Buzgau (JIRA)

unread,
Sep 12, 2019, 3:14:03 AM9/12/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Fix Version/s: FACT 3.14.4

Jean Bond (JIRA)

unread,
Sep 13, 2019, 3:28:03 PM9/13/19
to puppe...@googlegroups.com
Jean Bond updated an issue
Change By: Jean Bond
Labels: resolved-issue-added

Mihai Buzgau (JIRA)

unread,
Oct 11, 2019, 4:15:02 AM10/11/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Fix Version/s: FACT 3.13.5
Reply all
Reply to author
Forward
0 new messages