Jira (PUP-9323) Resolve Deferred values on demand instead of at catalog read time

29 views
Skip to first unread message

Henrik Lindberg (JIRA)

unread,
Nov 19, 2018, 6:52:03 AM11/19/18
to puppe...@googlegroups.com
Henrik Lindberg created an issue
 
Puppet / New Feature PUP-9323
Resolve Deferred values on demand instead of at catalog read time
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 2018/11/19 3:51 AM
Priority: Normal Normal
Reporter: Henrik Lindberg

Instead of (as in Puppet 6.0) resolve all Deferred values at catalog read time it would be much better if Deferred values were resolved as they are needed by resources. The big benefit is that this would enable additional use cases - where it is possible to let changes earlier in the apply cycle affect the values for later resources.

From the perspective of a resource this change would not matter, it simply would not see the Deferred instance (just as when it is deferred up front).

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
Nov 19, 2018, 6:53:02 AM11/19/18
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Nov 28, 2018, 5:49:01 AM11/28/18
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Unassigned

Henrik Lindberg (JIRA)

unread,
Nov 28, 2018, 5:49:02 AM11/28/18
to puppe...@googlegroups.com

Kenn Hussey (JIRA)

unread,
Dec 10, 2018, 10:26:03 AM12/10/18
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Dec 10, 2018, 1:42:03 PM12/10/18
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Fix Version/s: PUP 6.1.0
Fix Version/s: PUP 6.2.0

Kenn Hussey (JIRA)

unread,
Dec 12, 2018, 4:06:03 PM12/12/18
to puppe...@googlegroups.com

Kenn Hussey (JIRA)

unread,
Jan 14, 2019, 3:36:03 PM1/14/19
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Jan 14, 2019, 5:29:03 PM1/14/19
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Fix Version/s: PUP 6.2.0
Fix Version/s: PUP 6.y

Henrik Lindberg (JIRA)

unread,
Jan 14, 2019, 5:30:05 PM1/14/19
to puppe...@googlegroups.com
Henrik Lindberg commented on New Feature PUP-9323
 
Re: Resolve Deferred values on demand instead of at catalog read time

Change this to 6.y - not there yet - don't know if the feature will actually work well enough to be included.

Kenn Hussey (JIRA)

unread,
Jan 14, 2019, 5:46:03 PM1/14/19
to puppe...@googlegroups.com

Michael Smith (JIRA)

unread,
Jun 4, 2019, 5:41:04 PM6/4/19
to puppe...@googlegroups.com
Michael Smith commented on New Feature PUP-9323
 
Re: Resolve Deferred values on demand instead of at catalog read time

Alexander Fisher and I are chatting about another use case for this: writing modules, you may want to have a package installed, but are ok with a minimum version as supplied by the OS. But then you need to know the version that was installed to figure out how to configure it. This change to deferred would enable looking up the package version that was ensured when generating a template.

Henrik Lindberg (JIRA)

unread,
Jun 5, 2019, 8:16:03 AM6/5/19
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
Jun 5, 2019, 8:49:03 AM6/5/19
to puppe...@googlegroups.com

Hm, the actual difficulties with delaying Deferred resolution was not shown in this ticket.
There are many resource types that visit all of their parameters early and thus force the resolution too early. The resource type may need the value early, and such a resource can thus never get late binding deferreds. This complicates everything in a way that isn't great for users, and could possibly be fixed for resources for parameters that are now just read without actually being required to be read early.

The PR that is up is just delaying the deferred until the parameter is read. It is worth trying it out, but it won't work with some resources - notably File as it reads its parameters early. Very disappointing because it is the typical use case (File with template produced contents).

Jorie Tappa (JIRA)

unread,
Oct 16, 2019, 5:28:03 PM10/16/19
to puppe...@googlegroups.com
Jorie Tappa commented on New Feature PUP-9323

Reworking File has been a longtime wishlist item for us. Is this PR and implementation blocked by changes to File? Or should we implement for everything but the resources it won't currently work for as-is?

I'm going to change this to "Needs Information" until we know what steps we're taking, since it seems it's not ready for merge based on some open questions.

Henrik Lindberg (JIRA)

unread,
Oct 17, 2019, 6:07:03 AM10/17/19
to puppe...@googlegroups.com

Jorie Tappa The implementation is not blocked per se but when resources read all of their parameters (long) before they are applied then it defeats the purpose of trying to delay the resolution of the deferred value as they will then not be resolved in the order resources are applied (instead from a user perspective it would seem to be somewhat random).

If File is fixed so it does not read all of its parameters up front then it is worth considering adding the delayed resolution in the PR. There would still be quirks for other resource types that read early. I am not sure File is fixable because of what it needs to do - adding resources for children to update the graph for example - can all of that be delayed until it is the File's time to actually apply?

Josh Cooper (Jira)

unread,
Aug 26, 2020, 6:25:02 PM8/26/20
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-9323

Another example is if the deferred function needs a gem to be present (eg to make an HTTP REST call), then you can't ensure the gem is installed prior to the deferred function being called.

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Jun 14, 2021, 3:55:02 PM6/14/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Epic Link: PUP-9827
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Paul Were (Jira)

unread,
Jul 20, 2021, 11:40:04 AM7/20/21
to puppe...@googlegroups.com
Paul Were commented on New Feature PUP-9323
 
Re: Resolve Deferred values on demand instead of at catalog read time

We also have T-Mobile raise this issue on Deferred Action. T-Mobile is engaged in leveraging Deferred Action as described below and at this point, they don't have any alternative options.

Here is what they have described. Please review and advise.

I am getting unexpected behavior from a deferred function, specifically, it seems to be running earlier in the catalog apply process than I expect.
I have a repo of user_scripts that I sync out to a host with a vcsrepo resource. These scripts need to get uploaded into our MaaS controller only if they don't exist yet or if the script gets updated in its repo (and synced to the host).
I am using a deferred function to calculate the sha256 of the file on disk so that my maas_node_script provider can compare the current file on disk with the one in its DB to determine if it needs to be updated or not.
I have a requirement on the vcsrepo but the function runs before the repo has cloned, causing failures on a fresh run, or requiring multiple runs to apply the script changes if they already exist.
I expect the function to run at the time the resource is evaluated in the catalog on the agent but it seems to be running before it should.
 
OS: rhel 8.3
puppet agent version: 6.21.1
puppet pe version: 2019.8.5
 
 
sample code:

$h =

Unknown macro: {  'scripts_dir'         => '/root/user_scripts',   'user_scripts_branch' => 'master',   'user_scripts'        => [     Unknown macro}

,
   
Unknown macro: { 'name' => '00-prework',       'file' => '00-prework.sh' }
,
  ]
}
 

  1. clone scripts repo
    vcsrepo { $h['scripts_dir']:
      ensure            => 'latest',
      provider          => git,
      source            => "https://gitlab:${token}@gitlab.com/user_scripts.git",
      revision          => $h['user_scripts_branch'],
      trust_server_cert => true,
    }
     
    $h['user_scripts'].each |$i,$k,$script_path="${h['scripts_dir']}/${k['file']}"|
    Unknown macro: {   maas_node_script Unknown macro}

    }
     

  1. lib/puppet/functions/tmo_maas_controller/file_sha256.rb
    Puppet::Functions.create_function(:'tmo_maas_controller::file_sha256') do
      dispatch :get_sha do
        param 'String', :file_and_path
      end
     
      def get_sha(file_and_path)
        # file should exist
        if File.exist?(file_and_path)
          script_contents = File.read(file_and_path)&.strip
        else
          raise(Puppet::Error, "file=>#{file_and_path}< does not exist") # <-- I always get this error and a failed catalog apply if the repo hasnt been cloned yet (so far requires a manual clone)
        end
     
        # file should containe some text
        if script_contents.nil? || script_contents.empty?
          raise(Puppet::Error, "file=>#{file_and_path}< does not have any contents")
        end
     
        # get the sha256 using digest module
        require 'digest'
        Digest::SHA256.hexdigest(script_contents)
      endend

Josh Cooper (Jira)

unread,
Feb 8, 2022, 2:02:02 AM2/8/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Team: Froyo
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Josh Cooper (Jira)

unread,
Apr 11, 2022, 7:05:01 PM4/11/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Apr 21, 2022, 11:06:02 AM4/21/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Apr 26, 2022, 11:59:02 PM4/26/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Apr 27, 2022, 12:01:01 AM4/27/22
to puppe...@googlegroups.com

Nirupama Mantha (Jira)

unread,
Apr 27, 2022, 11:40:01 AM4/27/22
to puppe...@googlegroups.com
Nirupama Mantha updated an issue
Change By: Nirupama Mantha
Acceptance Criteria: Install the dependencies needed for a deferred function and call the deferred function in the same run.

The current behavior should be preserved by default, the new behavior can be turned on with a configuration.

Josh Cooper (Jira)

unread,
May 5, 2022, 12:22:02 PM5/5/22
to puppe...@googlegroups.com

Tony Vu (Jira)

unread,
May 11, 2022, 11:10:02 AM5/11/22
to puppe...@googlegroups.com
Tony Vu updated an issue
Change By: Tony Vu
Sprint: Phoenix 2022-05-11 , Phoenix 2022-05-25

Josh Cooper (Jira)

unread,
May 18, 2022, 1:13:02 AM5/18/22
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
May 18, 2022, 6:21:03 PM5/18/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes: Enhancement
Release Notes Summary: It's now possible for deferred functions to be called on-demand instead of being preprocessed. This way other resources in the catalog can serve as inputs to the deferred function. If the deferred function fails, then only that resource will fail, while unrelated resources will be applied. To enable this behavior set "Puppet[:preprocess_deferred] = false" or use "--no-preprocess_deferred" on the command line.

Parker Leach (Jira)

unread,
May 23, 2022, 1:13:03 PM5/23/22
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages