Jira (PUP-7051) Static catalog compiler uses Puppet master hostname for %H in fileserver config

9 views
Skip to first unread message

Matthias Hörmann (JIRA)

unread,
Dec 29, 2016, 7:15:02 AM12/29/16
to puppe...@googlegroups.com
Matthias Hörmann created an issue
 
Puppet / Bug PUP-7051
Static catalog compiler uses Puppet master hostname for %H in fileserver config
Issue Type: Bug Bug
Affects Versions: PUP 4.8.1
Assignee: Unassigned
Created: 2016/12/29 4:14 AM
Environment:

Puppet Server 2.7.2 on Debian wheezy

Priority: Normal Normal
Reporter: Matthias Hörmann

We have fileserver mount points for files that should not be checked into version control along with the environment and module manifests, either because they are too large or shared between environments (e.g. a host should still use the same SSL certificates if it is moved from production to development environment). One of these mount points is defined like this

[nodespecific]
  path /etc/puppet/files/nodespecific/%H
  allow *

It is meant to store data specific to the node (e.g. the above mentioned SSL certificates).

This has worked fine for quite a while now but it breaks when using the static catalog compiler, presumably due to the behaviour logged in puppetserver.log with this message

[puppetserver] Puppet No client; expanding '/etc/puppet/files/nodespecific/%H' with local host

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Moses Mendoza (JIRA)

unread,
Dec 30, 2016, 2:55:02 PM12/30/16
to puppe...@googlegroups.com
Moses Mendoza commented on Bug PUP-7051
 
Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

Thanks for filing Matthias Hörmann. I wasn't able to reproduce locally, though I was using puppet 4.8.1 with puppet server 2.6.0 so maybe something changed between 2.6.0 and 2.7.2. I was able to use %H in a mountpoint and expand without issue to retrieve a file under a directory for the agent's fqdn. This worked both with and without static catalogs enabled.

That message in your log comes from:https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/mount/file.rb#L85-L95 - which seems to imply the request was received without node information (it will default to just using the server's hostname, as you noted).

cc Jeremy Barlow can you repro or think of anything that changed recently that might have caused this?

Matthias Hörmann (JIRA)

unread,
Jan 2, 2017, 7:35:02 AM1/2/17
to puppe...@googlegroups.com

What could cause the request to be received without node information? Is there any sort of setting that might differ between our setup and your test setup to control this?

Moses Mendoza (JIRA)

unread,
Jan 3, 2017, 1:12:02 PM1/3/17
to puppe...@googlegroups.com
Moses Mendoza commented on Bug PUP-7051

That's a good question Matthias Hörmann. I'm not entirely sure. I also realized I don't have a full repro set up because I haven't configured `code_id` and the `static_file` endpoing as described here: https://docs.puppet.com/puppet/latest/static_catalogs.html, nor is my file served from within the `modules` mount point - it was just a custom mountpoint (apparently to inline file metadata, the file must be served via the `modules` mountpoint, according to that page). I apologize - I'm not especially familiar with Puppet Server, I work mostly on the agent side of things.

Can you confirm - this behavior does not occur when the `static_catalogs` setting is not enabled on your agent? Enabling that is all that is required for your repro?

Jeremy Barlow (JIRA)

unread,
Jan 3, 2017, 8:35:11 PM1/3/17
to puppe...@googlegroups.com
Jeremy Barlow commented on Bug PUP-7051

Moses Mendoza, I'm able to reproduce this just by enabling static catalogs from the Puppet Server perspective. I added this content to my puppetserver.conf file:

versioned-code: {
  code-id-command: /root/code-id-command_script.sh
  code-content-command: /root/code-content-command_script.sh
}

I just had some simple content in the two shell scripts.

For code-id-command_script.sh, I had:

#!/bin/sh
echo "1234"

For code-content-command_script.sh, I had:

#!/bin/sh
cat /etc/puppetlabs/code/environments/$1/$3

I don't think the content of the code-content-command_script.sh file even matters since the failure happens during the catalog compilation, before any subsequent requests that the agent might make to the static_file_content endpoint (where the code-content-command_script.sh file is used.

I don't think this problem has anything to do specifically with Puppet Server. With static catalogs enabled, I traced the request map which is passed around through the HTTP and indirector layers up to the catalog code. The request map definitely has the node in it as the catalog code is first entered, as evidenced from this line in the puppetserver.log file:

Compiled static catalog for mynode in environment production

Between that code and where the file_server terminus is navigated to retrieve file content, though, the node is dropped from the request options map. It is not present here. If I hardcode values into the option map for :node and :ip at this point, though, the request succeeds:

options = {
          :environment        => catalog.environment_instance,
          :links              => resource[:links] ? resource[:links].to_sym : :manage,
          :checksum_type      => resource[:checksum] ? resource[:checksum].to_sym : checksum_type.to_sym,
          :source_permissions => resource[:source_permissions] ? resource[:source_permissions].to_sym : :ignore,
          :node => 'mynode',
          :ip=>"127.0.0.1"
        }

The :ip is needed in order to avoid having the fileserver.conf authorization check fail.

I'm not sure what is expected to happen in this case. Should the :node and :ip be passed up through the calculation of the inline metadata or should the metadata code somehow be more tolerant of a failure to inline the node-specific metadata without causing the catalog request to fail with the HTTP 500 error, "Could not get metadata for puppet:///my_module/myfile.txt"?

Josh Cooper, what do you think about this one?

Josh Cooper (JIRA)

unread,
Jan 5, 2017, 1:30:19 PM1/5/17
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-7051

Due to custom mount points, etc it is possible for the static compiler to inline most metadata, but exclude metadata for individual file resources. We should only attempt in inline file metadata if the following are true: 1) the file source URI uses the puppet scheme, 2) the source URI path starts with the modules (using the modules mount point, 2) the files being served are in a files directory as they would be for a module, etc. See https://github.com/puppetlabs/puppet/blob/4.8.1/lib/puppet/indirector/catalog/compiler.rb#L114-L125
https://github.com/puppetlabs/puppet/blob/4.8.1/lib/puppet/indirector/catalog/compiler.rb#L130-L136.

That said it seems like the static compiler is trying to inline something when it shouldn't? It would be great to get a backtrace to see what code path we took to get to: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/mount/file.rb#L70

It would also be useful to get fileserver.conf and any mount related configuration.

Jeremy Barlow (JIRA)

unread,
Jan 5, 2017, 2:28:04 PM1/5/17
to puppe...@googlegroups.com
Jeremy Barlow commented on Bug PUP-7051

It would be great to get a backtrace to see what code path we took to get to: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/mount/file.rb#L70

Here's what I get when running Puppet from source at the 4.8.1 tag:

/puppet/lib/puppet/file_serving/configuration/parser.rb:119:in `validate'
/puppet/lib/puppet/file_serving/configuration/parser.rb:119:in `each'
/puppet/lib/puppet/file_serving/configuration/parser.rb:119:in `validate'
/puppet/lib/puppet/file_serving/configuration/parser.rb:48:in `parse'
/puppet/lib/puppet/file_serving/configuration.rb:99:in `readconfig'
/puppet/lib/puppet/file_serving/configuration.rb:38:in `initialize'
/puppet/lib/puppet/file_serving/configuration.rb:13:in `configuration'
/puppet/lib/puppet/indirector/file_server.rb:49:in `configuration'
/puppet/lib/puppet/indirector/file_server.rb:14:in `authorized?'
/puppet/lib/puppet/indirector/indirection.rb:305:in `check_authorization'
/puppet/lib/puppet/indirector/indirection.rb:324:in `prepare'
/puppet/lib/puppet/indirector/indirection.rb:267:in `search'
/puppet/lib/puppet/network/http/api/indirected_routes.rb:151:in `do_search'
/puppet/lib/puppet/network/http/api/indirected_routes.rb:48:in `call'
/puppet/lib/puppet/context.rb:65:in `override'
/puppet/lib/puppet.rb:241:in `override'
/puppet/lib/puppet/network/http/api/indirected_routes.rb:47:in `call'
/puppet/lib/puppet/network/http/route.rb:82:in `process'
/puppet/lib/puppet/network/http/route.rb:81:in `each'
/puppet/lib/puppet/network/http/route.rb:81:in `process'
/puppet/lib/puppet/network/http/route.rb:87:in `process'
/puppet/lib/puppet/network/http/route.rb:87:in `process'
/puppet/lib/puppet/network/http/handler.rb:60:in `process'
/puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile'
/puppet/lib/puppet/util/profiler.rb:51:in `profile'
/puppet/lib/puppet/network/http/handler.rb:58:in `process'

Here's the stack trace up to the point of the "No client" error message in the log, mentioned in the ticket description:

/puppet/lib/puppet/file_serving/mount/file.rb:41:in `path'
/puppet/lib/puppet/file_serving/mount/file.rb:16:in `complete_path'
/puppet/lib/puppet/file_serving/mount/file.rb:35:in `find'
/puppet/lib/puppet/indirector/file_server.rb:29:in `find'
/puppet/lib/puppet/indirector/indirection.rb:194:in `find'
/puppet/lib/puppet/indirector/catalog/compiler.rb:226:in `inline_metadata'
/puppet/lib/puppet/indirector/catalog/compiler.rb:223:in `each'
/puppet/lib/puppet/indirector/catalog/compiler.rb:223:in `inline_metadata'
/puppet/lib/puppet/indirector/catalog/compiler.rb:159:in `each'
/puppet/lib/puppet/indirector/catalog/compiler.rb:159:in `inline_metadata'
/puppet/lib/puppet/indirector/catalog/compiler.rb:279:in `compile'
/puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile'
/puppet/lib/puppet/util/profiler.rb:51:in `profile'
/puppet/lib/puppet/indirector/catalog/compiler.rb:278:in `compile'
/puppet/lib/puppet/util.rb:223:in `benchmark'
.m2/repository/org/jruby/jruby-stdlib/1.7.26/jruby-stdlib-1.7.26.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime'
/puppet/lib/puppet/util.rb:222:in `benchmark'
/puppet/lib/puppet/indirector/catalog/compiler.rb:277:in `compile'
/puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile'
/puppet/lib/puppet/util/profiler.rb:51:in `profile'
/puppet/lib/puppet/indirector/catalog/compiler.rb:264:in `compile'
/puppet/lib/puppet/util.rb:223:in `benchmark'
.m2/repository/org/jruby/jruby-stdlib/1.7.26/jruby-stdlib-1.7.26.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime'
/puppet/lib/puppet/util.rb:222:in `benchmark'
/puppet/lib/puppet/indirector/catalog/compiler.rb:262:in `compile'
/puppet/lib/puppet/indirector/catalog/compiler.rb:53:in `find'
/puppet/lib/puppet/indirector/indirection.rb:194:in `find'
/puppet/lib/puppet/network/http/api/indirected_routes.rb:121:in `do_find'
/puppet/lib/puppet/network/http/api/indirected_routes.rb:48:in `call'
/puppet/lib/puppet/context.rb:65:in `override'
/puppet/lib/puppet.rb:241:in `override'
/puppet/lib/puppet/network/http/api/indirected_routes.rb:47:in `call'
/puppet/lib/puppet/network/http/route.rb:82:in `process'
/puppet/lib/puppet/network/http/route.rb:81:in `each'
/puppet/lib/puppet/network/http/route.rb:81:in `process'
/puppet/lib/puppet/network/http/route.rb:87:in `process'
/puppet/lib/puppet/network/http/route.rb:87:in `process'
/puppet/lib/puppet/network/http/handler.rb:60:in `process'
/puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile'
/puppet/lib/puppet/util/profiler.rb:51:in `profile'
/puppet/lib/puppet/network/http/handler.rb:58:in `process'

My fileserver.conf file looked like this:

[amodule]
path /root/amodule/%H
allow *

My file resource from the manifest looked like this:

file { '/root/myfile.txt':
  source => 'puppet:///amodule/somefile.txt'
}

We should only attempt in inline file metadata if the following are true:
1) the file source URI uses the puppet scheme

This appears to be true.

2) the source URI path starts with the modules (using the modules mount point

Does having the source URI path start with the name of a module mount which matches one from the fileserver.conf file satisfy this?

3) the files being served are in a files directory as they would be for a module, etc.

When going through a file mount, should the file exist under a "files" directory under the mount? Seems like it wouldn't work that way even when static catalogs is disabled. Puppet seems to want to find it right immediately under the path from the fileserver.conf file. In my case, I get this error just after the "No client" error when static catalogs is enabled (no reference to a "files" subdirectory):

2017-01-05 11:20:34,298 INFO  [qtp633274311-38] [puppetserver] Puppet File does not exist or is not accessible: /root/amodule/localhost/somefile.txt

Note that "localhost" is what Puppet subbed in as the node name since the node name from my agent run was not passed through.

The agent run works fine if I remove the "%H" from the fileserver.conf file and have the file at the root of the mount path.

Per the rules you mentioned, should static catalog compilation be inlining any files accessed via a fileserver.conf mount?

Matthias Hörmann (JIRA)

unread,
Jan 6, 2017, 4:48:03 AM1/6/17
to puppe...@googlegroups.com

I don't quite see how static compilation would be any different in terms of knowing the node name. Isn't it still specifically compiled for each node?

I also don't see why static compilation should only work for files in modules. Is there a technical reason for this?

Josh Cooper (JIRA)

unread,
May 16, 2017, 5:16:04 PM5/16/17
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
May 16, 2017, 5:16:05 PM5/16/17
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-7051
 
Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

It looks like we're not passing the node object to the Puppet::FileServing::Metadata.indirection.find calls. Should be a simple fix to pass that through.

Josh Cooper (JIRA)

unread,
May 16, 2017, 5:16:06 PM5/16/17
to puppe...@googlegroups.com

Matthias Hörmann (JIRA)

unread,
Apr 23, 2018, 9:59:02 AM4/23/18
to puppe...@googlegroups.com
Matthias Hörmann commented on Bug PUP-7051
 
Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

Is there any progress on this? As long as this bug exists the static catalog compiler is basically useless to us.

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Matthias Hörmann (JIRA)

unread,
Apr 24, 2018, 9:07:04 AM4/24/18
to puppe...@googlegroups.com

I tried it again sicne the documentation claimed static metadata was only used for files in the modules' files directory.

Additionally, Puppet Server only inlines metadata for file resources if all of the following conditions are true:
 
    It contains a source parameter with a Puppet URI, such as source => 'puppet:///path/to/file'.
    It contains a source parameter that uses the built-in modules mount point.
    The file it sources is within the following glob relative to the environment’s root directory: */*/files/**. For example, Puppet Server will inline metadata into static catalogs for file resources sourcing module files located by default in /etc/puppetlabs/code/environments/<ENVIRONMENT>/modules/<MODULE NAME>/files/**.

which turned out to be false. As did the bit about the modules mountpoint as it clearly tries to compile metadata (and fails with an exception) for other mountpoints.

 

Josh Cooper (Jira)

unread,
May 22, 2020, 3:38:03 PM5/22/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Epic Link: PUP-10533
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages