Jira (PUP-11189) Add functions to iterate over files/templates in the current module

2 views
Skip to first unread message

Dirk Heinrichs (Jira)

unread,
Jul 27, 2021, 4:17:04 AM7/27/21
to puppe...@googlegroups.com
Dirk Heinrichs created an issue
 
Puppet / New Feature PUP-11189
Add functions to iterate over files/templates in the current module
Issue Type: New Feature New Feature
Assignee: Henrik Lindberg
Components: Functions, Language
Created: 2021/07/27 1:16 AM
Priority: Normal Normal
Reporter: Dirk Heinrichs

Puppet Version: 6.x, 7.x
Puppet Server Version: 6.x, 7.x
OS Name/Version: all

Sometimes, one needs to put several files in place on an agent node that come from the same module. As it is now, one needs to explicitly name each file in an array to iterate over. This has the drawback that one needs to adapt the code whenever a new file is added to <module>/files (or <module>/templates)

Desired Behavior:

Be able to write something like that:

files('puppet:///modules/foo/subdir').each |$file| {
  file {
    ...
  }
 }

and similar for templates (you get the idea).

Actual Behavior:

This is currently not possible (at least to my knowledge).

 

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Dirk Heinrichs (Jira)

unread,
Jul 27, 2021, 4:24:03 AM7/27/21
to puppe...@googlegroups.com
Dirk Heinrichs commented on New Feature PUP-11189
 
Re: Add functions to iterate over files/templates in the current module

Would also be great if the new functions had an optional "recurse" parameter (default: false).

Josh Cooper (Jira)

unread,
Jul 27, 2021, 9:12:04 PM7/27/21
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-11189

There is also the find_file function. It accepts a list of paths, but it only returns the first match:

❯ tree  ~/.puppetlabs/etc/code/environments/production/modules/test/files/   
/home/josh/.puppetlabs/etc/code/environments/production/modules/test/files/
└── a
    ├── b
    └── c
 
1 directory, 2 files
❯ bx puppet apply -e 'notice(find_file("test/a/b", "test/a/c"))'                 
Notice: Scope(Class[main]): /home/josh/.puppetlabs/etc/code/environments/production/modules/test/files/a/b
Notice: Compiled catalog for localhost in environment production in 0.02 seconds

If we allow recurse, then there's the issue of recurse limit and making sure users don't accidentally DoS puppetserver dynamically adding file resources to the catalog (like PUP-10946). I have a preference for glob syntax, e.g. files('foo/subdir/*/.ini')?

If we modify find_file then we should keep find_template in sync (as proposed in PUP-1214).

Also, thoughts Charlie Sharpsteen and Reid Vandewiele?

Dirk Heinrichs (Jira)

unread,
Jul 29, 2021, 5:18:03 AM7/29/21
to puppe...@googlegroups.com

I'm fine with globs, if extended glob syntax can be used, like "foo/sub/*/.ini".

Ciprian Badescu (Jira)

unread,
Aug 2, 2021, 10:41:04 AM8/2/21
to puppe...@googlegroups.com

Ben Ford, is the required case also a common use case in the modules?

Reid Vandewiele (Jira)

unread,
Aug 2, 2021, 11:28:04 AM8/2/21
to puppe...@googlegroups.com

I'm curious to understand the use case better. From my first reading of the desired goal, it seems like you should be able to do this today as follows:

file { 'target-directory':
  ensure  => directory,  
  path    => '/path/to/dir',
  source  => 'puppet:///foo/subdir',
  recurse => remote,
}

Dirk Heinrichs could you describe in a little bit more detail what you are trying to do specifically? Curious to understand better. Less the technical how-to of what you've tried; more the use case why/what-for. Hoping that a better understanding will shed some light on whether there's already a recommended way to do what you're looking for today, or if not, exactly what benefits the suggested improvement could provide, and how it looks when solving the target problem(s). Especially if introducing it would change the recommended way of solving the problem from an existing option today.

Dirk Heinrichs (Jira)

unread,
Aug 3, 2021, 2:13:02 AM8/3/21
to puppe...@googlegroups.com

Reid Vandewiele, my use case is to manage files on the agent based on content of the "files" or "templates" directory of a module (or a sub directory thereof), so that I don't need to change the class' code every time new files are added.

Anyway, I've read the documentation for the "file" resource again, and it looks like I can indeed achieve that with "recurse" (at least partly). Up to now, I've only used recurse together with "ensure => absent". It didn't come to my mind that it would also work this way.

What's missing is the "templates" part, though.

I also like the idea of using glob patterns here, which is currently not possible.

Reid Vandewiele (Jira)

unread,
Aug 3, 2021, 4:32:02 PM8/3/21
to puppe...@googlegroups.com

Thanks for that additional context, Dirk Heinrichs. That's helpful.

I'm hearing that the simple case, copy files as-is from module to target, is achievable today with the File resource and recurse => remote.

I'm also hearing that for a more complex, probably less common use case involving templates, it would be useful to have a general capability of inspecting module content dynamically from Puppet DSL code.

Given the complexity and non-generality of the second use case, I don't think it would make sense to implement a series of content-type-specific functions to enable this. That is, I don't think extending or modifying find_template, find_file, etc. would be generally beneficial.

It could make sense to extend a capability similar to the existing module_directory() function. For example, pairing it with a new module_files() function. A function like that could take a list of module-relative glob paths, and return an array of file paths. E.g.

call: module_files('module1/files/*', 'module2/templates/**/*.epp', 'module3/metadata.json')
return: [
  "/etc/puppetlabs/code/environments/production/modules/module1/files/file1",
  "/etc/puppetlabs/code/environments/production/modules/module1/files/file2",
  "/etc/puppetlabs/code/environments/production/modules/module2/templates/tmp1.txt.epp",
  "/etc/puppetlabs/code/environments/production/modules/module2/templates/tmp2.txt.epp",
  "/etc/puppetlabs/code/environments/production/modules/module2/templates/subdir/tmp3.txt.epp",
  "/etc/puppetlabs/code/environments/production/modules/module3/metadata.json",
]

 

If we provide additional functionality for this kind of use case out-of-box, the template-specific use case could be built on top of it using something like:

$module_directory = module_directory('xampl')
 
module_files('xampl/templates/**/*.epp').each |$template| {
  $relative_path = regsubst($template, "${module_directory}/templates/(.*).epp", '\\1')
  file { "/target/path/${relative_path}":
    ensure  => file,
    content => epp($template, {
      template_param_1 => 'value1',
      template_param_2 => 'value2',
    }),
  }
}

 

The technical complexity of a function like this is very low. If this doesn't make it into product for a time, it's probably something you could write yourself. For example:

 

Puppet::Functions.create_function(:'custom::module_files') do
  dispatch :module_files do
    repeated_param 'String', :patterns
  end
 
  def module_files(*patterns)
    patterns.reduce([]) do |memo, pattern|
      module_name = pattern.split('/').first
      glob = pattern.split('/').drop(1).join('/')
      module_directory = call_function('module_directory', module_name)
      memo + Dir.glob("#{module_directory}/#{glob}").select { |path| File.file?(path) }
    end
  end
end 

Dirk Heinrichs (Jira)

unread,
Aug 5, 2021, 2:19:01 AM8/5/21
to puppe...@googlegroups.com

Reid Vandewiele, thanks a lot for the detailed explanation. Yes, the request was to add (a) new function(s), not to modify any existing ones.

I will definitely try to implement "module_files()" myself (using your example above) to get a quick result. However, I also think this might be useful for others, too, so it would be really great to have it shipped as part of Puppet itself.

Ben Ford (Jira)

unread,
Aug 30, 2021, 6:34:02 PM8/30/21
to puppe...@googlegroups.com
Ben Ford commented on New Feature PUP-11189

Dirk Heinrichs you might consider building and maintaining this yourself as a module on the Forge.

Ciprian Badescu (Jira)

unread,
Sep 21, 2021, 6:00:04 AM9/21/21
to puppe...@googlegroups.com

We agree it is likely an improvement for the core functionality, but due to other issues demanding precedence, we don’t anticipate being able to address this any time soon. As such we are closing this as “Won’t Fix.” We may revisit it at a later time, and if so will re-open this ticket

Reply all
Reply to author
Forward
0 new messages