Jira (PUP-9389) Add new parameter to File for tempfile staging location

2 views
Skip to first unread message

Brett Kochendorfer (JIRA)

unread,
Dec 21, 2018, 2:14:03 PM12/21/18
to puppe...@googlegroups.com
Brett Kochendorfer created an issue
 
Puppet / New Feature PUP-9389
Add new parameter to File for tempfile staging location
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 2018/12/21 11:13 AM
Priority: Normal Normal
Reporter: Brett Kochendorfer

When Puppet creates a new file it creates it in a temporary location. This location is currently the location the file will exist in plus a unique name, eg /etc/foo123fsd for /etc/foo . In cases where a user wants to run a validate_cmd against the file it should be possible to render this file in a different location, run the validation, and then if the validation passes move the file to it's desired location.

This avoids issues where, for example, an application reads all the files in a directory when performing a task (think sudo) or an application reloads configs automatically from a directory.

The changes to make this possible seems doable and I'd be happy to submit a PR but wanted to post a feature here first to ensure this is something that would be acceptable to the Puppet community.

The change would look something like the following, diffed against Puppet 4.5.2. The replace_file function is used in many other places so would require other modifications, just giving a brief idea of what this might look like (open to other param names)

diff --git a/vendor/puppet-4.5.2/lib/puppet/type/file.rb b/vendor/puppet-4.5.2/lib/puppet/type/file.rb
index 8a29286dce..131e6549b8 100644
--- a/vendor/puppet-4.5.2/lib/puppet/type/file.rb
+++ b/vendor/puppet-4.5.2/lib/puppet/type/file.rb
@@ -324,6 +324,29 @@ Puppet::Type.newtype(:file) do
     defaultto '%'
   end
 
+  newparam(:staging_location) do
+    desc "For use with validate_cmd parameter. When running validate_cmd first
+    render to this staging location. This prevents rendering a temporary file in a location
+    where it may break a configuration before running a test against the file to ensure that
+    it is valid"
+
+    validate do |value|
+      unless Puppet::Util.absolute_path?(value)
+        fail Puppet::Error, "File paths must be fully qualified, not '#{value}'"
+      end
+    end
+
+    munge do |value|
+      if value.start_with?('//') and ::File.basename(value) == "/"
+        # This is a UNC path pointing to a share, so don't add a trailing slash
+        ::File.expand_path(value)
+      else
+        ::File.join(::File.split(::File.expand_path(value)))
+      end
+    end
+
+  end
+
   # Autorequire the nearest ancestor directory found in the catalog.
   autorequire(:file) do
     req = []
@@ -849,7 +872,7 @@ Puppet::Type.newtype(:file) do
     mode_int = mode ? symbolic_mode_to_int(mode, Puppet::Util::DEFAULT_POSIX_MODE) : nil
 
 
     if write_temporary_file?
-      Puppet::Util.replace_file(self[:path], mode_int) do |file|
+      Puppet::Util.replace_file(self[:path], mode_int, self[:staging_location]) do |file|
diff --git a/vendor/puppet-4.5.2/lib/puppet/util.rb b/vendor/puppet-4.5.2/lib/puppet/util.rb
index 011f2108e8..e8818e6906 100644
--- a/vendor/puppet-4.5.2/lib/puppet/util.rb
+++ b/vendor/puppet-4.5.2/lib/puppet/util.rb
@@ -419,7 +419,7 @@ module Util
   # preserved.  This works hard to avoid loss of any metadata, but will result
   # in an inode change for the file.
   #
-  # Arguments: `filename`, `default_mode`
+  # Arguments: `filename`, `default_mode`, `staging_location`
   #
   # The filename is the file we are going to replace.
   #
@@ -431,7 +431,7 @@ module Util
   DEFAULT_POSIX_MODE = 0644
   DEFAULT_WINDOWS_MODE = nil
 
-  def replace_file(file, default_mode, &block)
+  def replace_file(file, default_mode, staging_location, &block)
     raise Puppet::DevError, "replace_file requires a block" unless block_given?
 
     if default_mode
@@ -449,8 +449,12 @@ module Util
     end
 
     begin
-      file     = Puppet::FileSystem.pathname(file)
-      tempfile = Puppet::FileSystem::Uniquefile.new(Puppet::FileSystem.basename_string(file), Puppet::FileSystem.dir_string(file))
+      file = Puppet::FileSystem.pathname(file)
+      if staging_location
+        tempfile = Puppet::FileSystem::Uniquefile.new(Puppet::FileSystem.basename_string(file), staging_location)
+      else
+        tempfile = Puppet::FileSystem::Uniquefile.new(Puppet::FileSystem.basename_string(file), Puppet::FileSystem.dir_string(file))
+      end
 
       # Set properties of the temporary file before we write the content, because
       # Tempfile doesn't promise to be safe from reading by other people, just

 

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (JIRA)

unread,
Jan 2, 2019, 12:10:03 PM1/2/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Jan 2, 2019, 12:10:03 PM1/2/19
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-9389
 
Re: Add new parameter to File for tempfile staging location

Brett Kochendorfer thanks for the suggestion. We create the tempfile in the same directory as it ensures the temp and destination files are on the same volume, so the rename will be atomic. We lose that guarantee if we allow arbitrary staging directories, but I don't see another way to solve the issue you mention. /cc Jacob Helwig

Jacob Helwig (JIRA)

unread,
Jan 2, 2019, 1:24:02 PM1/2/19
to puppe...@googlegroups.com
Jacob Helwig commented on New Feature PUP-9389

Yeah, in the case where an application (like sudo) is doing "live" parsing of files instead of waiting for a reload command, I don't see another way around making sure that the application always has either the old config, or the new one. For something like a daemon that waits for a reload command before parsing all of the files, the validation command could be a script that sets up a new directory before running the validation on that temp directory it set up, but as the comment in the patch alludes to, that could leave the "real" daemon in a broken state if we're unable to clean up the temp file for some reason (For example: power loss). I'd say that this is even more general of a problem than when someone is concerned with using validate_cmd.

I'd be fine with adding a "stage this file here" parameter, and documenting it that it defaults to the same directory as the final destination with a big warning that file replacement is only guaranteed to be atomic if the staging location is on the same filesystem as the final location. I probably wouldn't have the comments about being intended only for use with validate_cmd as this seems more generally useful than that, though mentioning that it might be useful when using validate_cmd seems fine.

Jacob Helwig (JIRA)

unread,
Apr 22, 2019, 12:47:03 PM4/22/19
to puppe...@googlegroups.com

Jacob Helwig (JIRA)

unread,
Apr 22, 2019, 12:47:04 PM4/22/19
to puppe...@googlegroups.com
Jacob Helwig updated an issue
Change By: Jacob Helwig
Sprint: Platform Core KANBAN

Jacob Helwig (JIRA)

unread,
Apr 29, 2019, 11:24:03 AM4/29/19
to puppe...@googlegroups.com
Jacob Helwig updated an issue
Change By: Jacob Helwig
Release Notes Summary: The temporary location used when creating the new content for a File resource can now be customized using the `staging_location` parameter.
Release Notes: Enhancement

Jacob Helwig (JIRA)

unread,
Apr 29, 2019, 11:30:01 AM4/29/19
to puppe...@googlegroups.com
Jacob Helwig commented on New Feature PUP-9389
 
Re: Add new parameter to File for tempfile staging location

This has been merged to the master branch in 2834f16cef.

Jacob Helwig (JIRA)

unread,
Apr 29, 2019, 12:33:03 PM4/29/19
to puppe...@googlegroups.com

Heston Hoffman (JIRA)

unread,
Jun 11, 2019, 6:16:03 PM6/11/19
to puppe...@googlegroups.com
Heston Hoffman updated an issue
Change By: Heston Hoffman
Labels: resolved-issue-added
Reply all
Reply to author
Forward
0 new messages