Jira (PUP-10855) Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog

[Nacho Barrientos (Jira)]

Nacho Barrientos created an issue

Puppet / PUP-10855 Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog Issue Type: Bug Assignee: Unassigned Created: 2021/01/15 1:45 AM Priority: Normal Reporter: Nacho Barrientos Puppet Version: 6.15.0 Puppet Server Version: Does not apply OS Name/Version: CentOS8 Hi, Opening this for discussion mainly, it might not be a bug. We have an in-house debugging tool that downloads a catalog from a master and persists it to disk (serialised as JSON) to further inspection later on by another tool. Inspired by this, the code is something like this: require 'puppet' require 'puppet/face' Puppet.initialize_settings Puppet.initialize_facts Puppet.settings.preferred_run_mode = :agent Puppet::Resource::Catalog.indirection.terminus_class = :rest Puppet::Resource::Catalog.indirection.cache_class = nil Puppet::Node::Facts.indirection.terminus_class = :facter Puppet[:server] = "foo" Puppet[:masterport] = 1234 Puppet[:client_datadir] = "/somewhere" facts = Puppet::Node::Facts.indirection.find(Puppet[:node_name_value]).render(:pson) query_options = {:facts_format => :pson, :facts => CGI.escape(facts)} environment = Puppet::Node::Environment.remote(Puppet[:environment]) Puppet.override(:current_environment => environment) do begin catalog = Puppet::Resource::Catalog.indirection.find(Puppet[:certname], query_options) rescue Exception => e puts e exit 1 end exit 2 if catalog.nil? Puppet::Resource::Catalog.indirection.terminus_class = :json Puppet::Face[:catalog, '0.0.1'].save(catalog) end puts "Saved catalog to #{Puppet::Resource::Catalog.indirection.terminus.path(Puppet[:certname])}" exit 0 This was working code on Puppet5, that only needed some trivial modifications to make it work with Puppet6 (6.15.0): -facts = Puppet::Node::Facts.indirection.find(Puppet[:node_name_value]).render(:pson) -query_options = {:facts_format => :pson, :facts => CGI.escape(facts)} +facts = Puppet::Node::Facts.indirection.find(Puppet[:node_name_value]) +query_options = {:facts_for_catalog => facts} However, we noticed that it was failing when handling some catalogs as follows: Failed to deserialize Puppet::Resource::Catalog from rich_data_json: Internal Error: Puppet Context ':loaders' missing Digging into the bugtracker, we found PUP-10572. Initially, we tried backporting the patch that fixed the issue but no luck, so we patched our code to inject the loaders ourselves into the context: -Puppet.override(:current_environment => environment) do +Puppet.override(:current_environment => environment, loaders: Puppet::Pops::Loaders.new(environment, true)) do And the problem disappeared. Is preparing and injecting the loaders something we're supposed to do now on our side or perhaps it's a bug similar to PUP-10572? Thanks. (CERN internal ref: AI-6051) Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10855

Re: Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog This behavior is expected. In puppet 6, agents default to requesting a so called "rich data" catalog, which allows puppet to serialize data types that aren't natively supported in JSON, such as sensitive, binary, deferred, etc. Note the Accept: application/vnd.puppet.rich+json, ... header: bx puppet agent -t --http_debug > /dev/null 2>&1 | grep -e 'puppet/v3/catalog' <- "POST /puppet/v3/catalog/localhost?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.20.0\r\nUser-Agent: Puppet/6.20.0 Ruby/2.5.8-p224 (x86_64-darwin18)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: silly-devotion.delivery.puppetlabs.net:8140\r\nContent-Length: 24765\r\n\r\n" Depending on what your application needs to do, you could instead ask for a "json" catalog, for example if you're doing a plain diff of the catalog. bx puppet agent -t --http_debug --no-rich_data > /dev/null 2>&1 | grep -e 'puppet/v3/catalog' <- "POST /puppet/v3/catalog/localhost?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.20.0\r\nUser-Agent: Puppet/6.20.0 Ruby/2.5.8-p224 (x86_64-darwin18)\r\nAccept: application/json, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: silly-devotion.delivery.puppetlabs.net:8140\r\nContent-Length: 24765\r\n\r\n" If you need to preserve the puppet data types, then you will need to setup the loaders in the client application. Note in puppet 7 (and in later versions of puppet 6.x), it's not necessary to use the indirector to retrieve catalogs from the server. You might want to take a look at the new http client, something like: require 'puppet' require 'pp'   Puppet.initialize_settings # Puppet[:server] = "foo" # Puppet[:masterport] = 1234 # Puppet[:client_datadir] = "/somewhere" # Puppet[:http_debug] = true

Puppet::Node::Facts.indirection.terminus_class = :facter

facts = Puppet::Node::Facts.indirection.find(Puppet[:node_name_value])   environment = Puppet::Node::Environment.remote('production')   http = Puppet.runtime[:http] session = http.create_session route = session.route_to(:puppet) response, catalog = route.post_catalog(Puppet[:certname], facts: facts, environment: environment.name) pp catalog.to_data_hash

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10855

Re: Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog

Since this is expected behavior, I'm going to close this. Feel free to reach out in #puppet-dev in slack if you have questions.

Add Comment

[Nacho Barrientos (Jira)]

Nacho Barrientos commented on PUP-10855

Re: Need to inject loaders when retrieving a catalog via Puppet::Resource::Catalog

Thanks for confirming the loaders bit and for the hint about the catalog retrieval.

Add Comment