Jira (PUP-7051) Static catalog compiler uses Puppet master hostname for %H in fileserver config

[Matthias Hörmann (JIRA)]

Matthias Hörmann created an issue

Puppet / PUP-7051 Static catalog compiler uses Puppet master hostname for %H in fileserver config Issue Type: Bug Affects Versions: PUP 4.8.1 Assignee: Unassigned Created: 2016/12/29 4:14 AM Environment: Puppet Server 2.7.2 on Debian wheezy Priority: Normal Reporter: Matthias Hörmann We have fileserver mount points for files that should not be checked into version control along with the environment and module manifests, either because they are too large or shared between environments (e.g. a host should still use the same SSL certificates if it is moved from production to development environment). One of these mount points is defined like this [nodespecific] path /etc/puppet/files/nodespecific/%H allow * It is meant to store data specific to the node (e.g. the above mentioned SSL certificates). This has worked fine for quite a while now but it breaks when using the static catalog compiler, presumably due to the behaviour logged in puppetserver.log with this message [puppetserver] Puppet No client; expanding '/etc/puppet/files/nodespecific/%H' with local host Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Moses Mendoza (JIRA)]

Moses Mendoza commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config Thanks for filing Matthias Hörmann. I wasn't able to reproduce locally, though I was using puppet 4.8.1 with puppet server 2.6.0 so maybe something changed between 2.6.0 and 2.7.2. I was able to use %H in a mountpoint and expand without issue to retrieve a file under a directory for the agent's fqdn. This worked both with and without static catalogs enabled. That message in your log comes from:https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/mount/file.rb#L85-L95 - which seems to imply the request was received without node information (it will default to just using the server's hostname, as you noted). cc Jeremy Barlow can you repro or think of anything that changed recently that might have caused this?

Add Comment

[Matthias Hörmann (JIRA)]

Matthias Hörmann commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

What could cause the request to be received without node information? Is there any sort of setting that might differ between our setup and your test setup to control this?

Add Comment

[Moses Mendoza (JIRA)]

Moses Mendoza commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

That's a good question Matthias Hörmann. I'm not entirely sure. I also realized I don't have a full repro set up because I haven't configured `code_id` and the `static_file` endpoing as described here: https://docs.puppet.com/puppet/latest/static_catalogs.html, nor is my file served from within the `modules` mount point - it was just a custom mountpoint (apparently to inline file metadata, the file must be served via the `modules` mountpoint, according to that page). I apologize - I'm not especially familiar with Puppet Server, I work mostly on the agent side of things. Can you confirm - this behavior does not occur when the `static_catalogs` setting is not enabled on your agent? Enabling that is all that is required for your repro?

Add Comment

[Jeremy Barlow (JIRA)]

Jeremy Barlow commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

Moses Mendoza, I'm able to reproduce this just by enabling static catalogs from the Puppet Server perspective. I added this content to my puppetserver.conf file: versioned-code: { code-id-command: /root/code-id-command_script.sh code-content-command: /root/code-content-command_script.sh } I just had some simple content in the two shell scripts. For code-id-command_script.sh, I had: #!/bin/sh echo "1234" For code-content-command_script.sh, I had: #!/bin/sh cat /etc/puppetlabs/code/environments/$1/$3 I don't think the content of the code-content-command_script.sh file even matters since the failure happens during the catalog compilation, before any subsequent requests that the agent might make to the static_file_content endpoint (where the code-content-command_script.sh file is used. I don't think this problem has anything to do specifically with Puppet Server. With static catalogs enabled, I traced the request map which is passed around through the HTTP and indirector layers up to the catalog code. The request map definitely has the node in it as the catalog code is first entered, as evidenced from this line in the puppetserver.log file: Compiled static catalog for mynode in environment production Between that code and where the file_server terminus is navigated to retrieve file content, though, the node is dropped from the request options map. It is not present here. If I hardcode values into the option map for :node and :ip at this point, though, the request succeeds: options = { :environment => catalog.environment_instance, :links => resource[:links] ? resource[:links].to_sym : :manage, :checksum_type => resource[:checksum] ? resource[:checksum].to_sym : checksum_type.to_sym, :source_permissions => resource[:source_permissions] ? resource[:source_permissions].to_sym : :ignore, :node => 'mynode', :ip=>"127.0.0.1" } The :ip is needed in order to avoid having the fileserver.conf authorization check fail. I'm not sure what is expected to happen in this case. Should the :node and :ip be passed up through the calculation of the inline metadata or should the metadata code somehow be more tolerant of a failure to inline the node-specific metadata without causing the catalog request to fail with the HTTP 500 error, "Could not get metadata for puppet:///my_module/myfile.txt"? Josh Cooper, what do you think about this one?

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

Due to custom mount points, etc it is possible for the static compiler to inline most metadata, but exclude metadata for individual file resources. We should only attempt in inline file metadata if the following are true: 1) the file source URI uses the puppet scheme, 2) the source URI path starts with the modules (using the modules mount point, 2) the files being served are in a files directory as they would be for a module, etc. See https://github.com/puppetlabs/puppet/blob/4.8.1/lib/puppet/indirector/catalog/compiler.rb#L114-L125 https://github.com/puppetlabs/puppet/blob/4.8.1/lib/puppet/indirector/catalog/compiler.rb#L130-L136. That said it seems like the static compiler is trying to inline something when it shouldn't? It would be great to get a backtrace to see what code path we took to get to: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/mount/file.rb#L70 It would also be useful to get fileserver.conf and any mount related configuration.

Add Comment

[Jeremy Barlow (JIRA)]

Jeremy Barlow commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

It would be great to get a backtrace to see what code path we took to get to: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_serving/mount/file.rb#L70 Here's what I get when running Puppet from source at the 4.8.1 tag: /puppet/lib/puppet/file_serving/configuration/parser.rb:119:in `validate' /puppet/lib/puppet/file_serving/configuration/parser.rb:119:in `each' /puppet/lib/puppet/file_serving/configuration/parser.rb:119:in `validate' /puppet/lib/puppet/file_serving/configuration/parser.rb:48:in `parse' /puppet/lib/puppet/file_serving/configuration.rb:99:in `readconfig' /puppet/lib/puppet/file_serving/configuration.rb:38:in `initialize' /puppet/lib/puppet/file_serving/configuration.rb:13:in `configuration' /puppet/lib/puppet/indirector/file_server.rb:49:in `configuration' /puppet/lib/puppet/indirector/file_server.rb:14:in `authorized?' /puppet/lib/puppet/indirector/indirection.rb:305:in `check_authorization' /puppet/lib/puppet/indirector/indirection.rb:324:in `prepare' /puppet/lib/puppet/indirector/indirection.rb:267:in `search' /puppet/lib/puppet/network/http/api/indirected_routes.rb:151:in `do_search' /puppet/lib/puppet/network/http/api/indirected_routes.rb:48:in `call' /puppet/lib/puppet/context.rb:65:in `override' /puppet/lib/puppet.rb:241:in `override' /puppet/lib/puppet/network/http/api/indirected_routes.rb:47:in `call' /puppet/lib/puppet/network/http/route.rb:82:in `process' /puppet/lib/puppet/network/http/route.rb:81:in `each' /puppet/lib/puppet/network/http/route.rb:81:in `process' /puppet/lib/puppet/network/http/route.rb:87:in `process' /puppet/lib/puppet/network/http/route.rb:87:in `process' /puppet/lib/puppet/network/http/handler.rb:60:in `process' /puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile' /puppet/lib/puppet/util/profiler.rb:51:in `profile' /puppet/lib/puppet/network/http/handler.rb:58:in `process' Here's the stack trace up to the point of the "No client" error message in the log, mentioned in the ticket description: /puppet/lib/puppet/file_serving/mount/file.rb:41:in `path' /puppet/lib/puppet/file_serving/mount/file.rb:16:in `complete_path' /puppet/lib/puppet/file_serving/mount/file.rb:35:in `find' /puppet/lib/puppet/indirector/file_server.rb:29:in `find' /puppet/lib/puppet/indirector/indirection.rb:194:in `find' /puppet/lib/puppet/indirector/catalog/compiler.rb:226:in `inline_metadata' /puppet/lib/puppet/indirector/catalog/compiler.rb:223:in `each' /puppet/lib/puppet/indirector/catalog/compiler.rb:223:in `inline_metadata' /puppet/lib/puppet/indirector/catalog/compiler.rb:159:in `each' /puppet/lib/puppet/indirector/catalog/compiler.rb:159:in `inline_metadata' /puppet/lib/puppet/indirector/catalog/compiler.rb:279:in `compile' /puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile' /puppet/lib/puppet/util/profiler.rb:51:in `profile' /puppet/lib/puppet/indirector/catalog/compiler.rb:278:in `compile' /puppet/lib/puppet/util.rb:223:in `benchmark' .m2/repository/org/jruby/jruby-stdlib/1.7.26/jruby-stdlib-1.7.26.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime' /puppet/lib/puppet/util.rb:222:in `benchmark' /puppet/lib/puppet/indirector/catalog/compiler.rb:277:in `compile' /puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile' /puppet/lib/puppet/util/profiler.rb:51:in `profile' /puppet/lib/puppet/indirector/catalog/compiler.rb:264:in `compile' /puppet/lib/puppet/util.rb:223:in `benchmark' .m2/repository/org/jruby/jruby-stdlib/1.7.26/jruby-stdlib-1.7.26.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime' /puppet/lib/puppet/util.rb:222:in `benchmark' /puppet/lib/puppet/indirector/catalog/compiler.rb:262:in `compile' /puppet/lib/puppet/indirector/catalog/compiler.rb:53:in `find' /puppet/lib/puppet/indirector/indirection.rb:194:in `find' /puppet/lib/puppet/network/http/api/indirected_routes.rb:121:in `do_find' /puppet/lib/puppet/network/http/api/indirected_routes.rb:48:in `call' /puppet/lib/puppet/context.rb:65:in `override' /puppet/lib/puppet.rb:241:in `override' /puppet/lib/puppet/network/http/api/indirected_routes.rb:47:in `call' /puppet/lib/puppet/network/http/route.rb:82:in `process' /puppet/lib/puppet/network/http/route.rb:81:in `each' /puppet/lib/puppet/network/http/route.rb:81:in `process' /puppet/lib/puppet/network/http/route.rb:87:in `process' /puppet/lib/puppet/network/http/route.rb:87:in `process' /puppet/lib/puppet/network/http/handler.rb:60:in `process' /puppet/lib/puppet/util/profiler/around_profiler.rb:58:in `profile' /puppet/lib/puppet/util/profiler.rb:51:in `profile' /puppet/lib/puppet/network/http/handler.rb:58:in `process' My fileserver.conf file looked like this: [amodule] path /root/amodule/%H allow * My file resource from the manifest looked like this: file { '/root/myfile.txt': source => 'puppet:///amodule/somefile.txt' }

We should only attempt in inline file metadata if the following are true: 1) the file source URI uses the puppet scheme

This appears to be true.

2) the source URI path starts with the modules (using the modules mount point

Does having the source URI path start with the name of a module mount which matches one from the fileserver.conf file satisfy this? 3) the files being served are in a files directory as they would be for a module, etc. When going through a file mount, should the file exist under a "files" directory under the mount? Seems like it wouldn't work that way even when static catalogs is disabled. Puppet seems to want to find it right immediately under the path from the fileserver.conf file. In my case, I get this error just after the "No client" error when static catalogs is enabled (no reference to a "files" subdirectory): 2017-01-05 11:20:34,298 INFO [qtp633274311-38] [puppetserver] Puppet File does not exist or is not accessible: /root/amodule/localhost/somefile.txt Note that "localhost" is what Puppet subbed in as the node name since the node name from my agent run was not passed through. The agent run works fine if I remove the "%H" from the fileserver.conf file and have the file at the root of the mount path. Per the rules you mentioned, should static catalog compilation be inlining any files accessed via a fileserver.conf mount?

Add Comment

[Matthias Hörmann (JIRA)]

Matthias Hörmann commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

I don't quite see how static compilation would be any different in terms of knowing the node name. Isn't it still specifically compiled for each node? I also don't see why static compilation should only work for files in modules. Is there a technical reason for this?

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-7051

Static catalog compiler uses Puppet master hostname for %H in fileserver config

Change By: Josh Cooper Team: Agent

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config It looks like we're not passing the node object to the Puppet::FileServing::Metadata.indirection.find calls. Should be a simple fix to pass that through.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-7051

Static catalog compiler uses Puppet master hostname for %H in fileserver config

Change By: Josh Cooper Labels: triaged

Add Comment

[Matthias Hörmann (JIRA)]

Matthias Hörmann commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config Is there any progress on this? As long as this bug exists the static catalog compiler is basically useless to us.

Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Matthias Hörmann (JIRA)]

Matthias Hörmann commented on PUP-7051

Re: Static catalog compiler uses Puppet master hostname for %H in fileserver config

I tried it again sicne the documentation claimed static metadata was only used for files in the modules' files directory. Additionally, Puppet Server only inlines metadata for file resources if all of the following conditions are true:   It contains a source parameter with a Puppet URI, such as source => 'puppet:///path/to/file'. It contains a source parameter that uses the built-in modules mount point. The file it sources is within the following glob relative to the environment’s root directory: */*/files/**. For example, Puppet Server will inline metadata into static catalogs for file resources sourcing module files located by default in /etc/puppetlabs/code/environments/<ENVIRONMENT>/modules/<MODULE NAME>/files/**. which turned out to be false. As did the bit about the modules mountpoint as it clearly tries to compile metadata (and fails with an exception) for other mountpoints.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7051

Static catalog compiler uses Puppet master hostname for %H in fileserver config

Change By: Josh Cooper Epic Link: PUP-10533

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)