Jira (PUP-7629) Hieradata should be loaded and cached in accordance with the environment_timeout

31 views
Skip to first unread message

Nick Walker (JIRA)

unread,
Jun 7, 2017, 2:23:03 PM6/7/17
to puppe...@googlegroups.com
Nick Walker created an issue
 
Puppet / Bug PUP-7629
Hieradata should be loaded and cached in accordance with the environment_timeout
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2017/06/07 11:22 AM
Priority: Normal Normal
Reporter: Nick Walker

The Problem

When setting environment_timeout to unlimited there is an expectation that all code and hieradata be loaded into memory and not re-read from disk until the environment cache is flushed.

If you change a puppet manifest on disk after it has been loaded into the puppet master the puppet master will not read that manifest.

However, if you change a hiera data file on disk after it has been loaded by the puppet master it will still be reread on disk and immediately affect catalog compilation.

Suggested Solution

These two things are not congruent. Either all files in the $codedir should be cached until the environment cache is flush or none of the files should be cached.

Performance Benefit

There is presumably a performance benefit to be had from loading hieradata once and then never having to recheck files on disk until the cache is flushed.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Nick Walker (JIRA)

unread,
Jun 7, 2017, 2:24:02 PM6/7/17
to puppe...@googlegroups.com

Nick Walker (JIRA)

unread,
Jun 7, 2017, 2:25:02 PM6/7/17
to puppe...@googlegroups.com
Nick Walker updated an issue
 
Change By: Nick Walker
h1. The Problem


When setting environment_timeout to unlimited there is an expectation that all code and hieradata be loaded into memory and not re-read from disk until the environment cache is flushed. 

If you change a puppet manifest on disk after it has been loaded into the puppet master the puppet master will not read that manifest.  

However, if you change a hiera data file on disk after it has been loaded by the puppet master it will still be  reread on  re-read from  disk and immediately affect catalog compilation.  

h1.  Suggested Solution 


These two things are not congruent.  Either all files in the $codedir should be cached until the environment cache is flush or none of the files should be cached.  

h1.  Performance Benefit


There is presumably a performance benefit to be had from loading hieradata once and then never having to recheck files on disk until the cache is flushed.  

Thomas Hallgren (JIRA)

unread,
Jun 7, 2017, 5:45:03 PM6/7/17
to puppe...@googlegroups.com
Thomas Hallgren commented on Bug PUP-7629
 
Re: Hieradata should be loaded and cached in accordance with the environment_timeout

The proposed change make sense and would remove the need for the stat that occurs on each file, once for every compile run.

Nick Walker (JIRA)

unread,
Jun 7, 2017, 6:47:03 PM6/7/17
to puppe...@googlegroups.com
Nick Walker updated an issue
Change By: Nick Walker
Method Found: Needs Assessment Manual Test

Henrik Lindberg (JIRA)

unread,
Jun 8, 2017, 2:05:03 AM6/8/17
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Jun 8, 2017, 2:10:02 AM6/8/17
to puppe...@googlegroups.com
Henrik Lindberg commented on Bug PUP-7629
 
Re: Hieradata should be loaded and cached in accordance with the environment_timeout

Nick Walker and other watchers - how important is it to fix this? Should we try to get this into Puppet 5.0.0 ?

Nick Walker (JIRA)

unread,
Jun 9, 2017, 11:00:05 AM6/9/17
to puppe...@googlegroups.com
Nick Walker commented on Bug PUP-7629

Henrik Lindberg Without knowing most of the variables around when Puppet 5 is expected and how many other things need to get in, I'll just say that I'd rather us take our time and not rush a solution just to get it into Puppet 5.

So, I guess I don't think it has to be in Puppet 5.

Gary Larizza (JIRA)

unread,
Jun 9, 2017, 12:43:03 PM6/9/17
to puppe...@googlegroups.com
Gary Larizza commented on Bug PUP-7629

I'm not sure how much work you think it would be, Henrik Lindberg, but if it's anything more than a trivial change, then probably not for 5. Would this be considered a backwards-incompatible change? Would it need to be in a major revision?

Henrik Lindberg (JIRA)

unread,
Jun 12, 2017, 5:29:02 AM6/12/17
to puppe...@googlegroups.com

I don't think we can just change this on users as it will affect their workflow. While not strictly a backwards compatibility concern it will be very irritating for those that rely on puppet/hiera working the way it is (while most would not notice at all).

Henrik Lindberg (JIRA)

unread,
Oct 25, 2017, 7:30:03 AM10/25/17
to puppe...@googlegroups.com

Another issue here is that non file based backends have no way (in general) of "snapshotting" data - thus data in a DB backend may change at any time.

David McTavish (Jira)

unread,
Dec 6, 2021, 1:49:01 PM12/6/21
to puppe...@googlegroups.com
David McTavish updated an issue
 
Change By: David McTavish
Labels: final_triage
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

David McTavish (Jira)

unread,
Dec 6, 2021, 1:49:01 PM12/6/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Dec 6, 2021, 8:11:02 PM12/6/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-7629
 
Re: Hieradata should be loaded and cached in accordance with the environment_timeout

Now that environment caching works as expected, it would be good to fix this issue, as the current system loads environments and their configs through different code paths, which greatly complicates the puppet code base.

Josh Cooper (Jira)

unread,
Dec 6, 2021, 8:15:01 PM12/6/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Comment:
Now that environment caching works as expected, it would be good to fix this issue, as the current system loads environments and their configs through different code paths, which greatly complicates the puppet code base.

Molly Waggett (Jira)

unread,
Feb 22, 2022, 1:44:02 PM2/22/22
to puppe...@googlegroups.com
Molly Waggett updated an issue
Change By: Molly Waggett
Team: Froyo Phoenix
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Molly Waggett (Jira)

unread,
Feb 22, 2022, 1:44:02 PM2/22/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Apr 27, 2022, 12:25:01 AM4/27/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Apr 27, 2022, 12:26:02 AM4/27/22
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages