Jira (PUP-10775) Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

[Tim Meusel (Jira)]

Tim Meusel created an issue

Puppet / PUP-10775 Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog` Issue Type: Bug Assignee: Unassigned Created: 2020/11/05 9:17 AM Priority: Normal Reporter: Tim Meusel Hey Josh Cooper, I did further debugging for https://tickets.puppetlabs.com/browse/PUP-10772. while doing so I discovered that I've 5 Ubuntu 16.04 machines, all of them currently show the `Failed to deserialize Puppet::Resource::Catalog` error message that I show later my client: root~ # facter -p os pup{ architecture => "amd64", distro => { codename => "xenial", description => "Ubuntu 16.04.7 LTS", id => "Ubuntu", release => { full => "16.04", major => "16.04" } }, family => "Debian", hardware => "x86_64", name => "Ubuntu", release => { full => "16.04", major => "16.04" }, selinux => { enabled => false } } root~ # puppet --version 6.19.1 root ~ # preferred_serialization_format is not set: root ~ # cat /etc/puppetlabs/puppet/puppet.conf   [main] certname = * codedir = /etc/puppetlabs/code hostprivkey = $privatekeydir/$certname.pem { mode = 640 } logdir = /var/log/puppetlabs/puppet pluginfactsource = puppet:///pluginfacts pluginsource = puppet:///plugins privatekeydir = $ssldir/private_keys { group = service } rundir = /var/run/puppetlabs server = * show_diff = true ssldir = /etc/puppetlabs/puppet/ssl vardir = /opt/puppetlabs/puppet/cache   [agent] classfile = $statedir/classes.txt default_schedules = false environment = production localconfig = $vardir/localconfig masterport = 8140 noop = false report = true runinterval = 7200 splay = true splaylimit = 7200 usecacheonfailure = true root ~ # but msgpack is installed: root ~ # /opt/puppetlabs/puppet/bin/gem list msgpack   *** LOCAL GEMS ***   msgpack (1.3.3, 1.3.1) root ~ # during a puppetrun to a 6.14.1 puppetserver with msgpack installed I see the following messages: starting SSL for *:8140... SSL established <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40356\r\n\r\n" ... Info: Unable to serialize catalog to json, retrying with pson ... the agent run works fine. Now the same agent connects to a puppetserver without msgpack installed: ... starting SSL for *:8140... SSL established <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40357\r\n\r\n" ... -> "Content-Type: text/pson;charset=iso-8859-1\r\n" -> "Transfer-Encoding: chunked\r\n" -> "Connection: keep-alive\r\n" -> "X-Puppet-Version: 6.19.1\r\n" -> "Vary: Accept-Encoding, User-Agent\r\n" -> "Content-Encoding: gzip\r\n" ... Error: Could not retrieve catalog from remote server: Failed to deserialize Puppet::Resource::Catalog from pson: expected value in object at '"ELF\u0002\u0001\u0'! Wrapped exception: expected value in object at '"ELF\u0002\u0001\u0'! Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run now it fails? Sadly I don't know since when this is the case. I suspect someone added a new file resource with a binary that's read in with file(). But this is really just a guess. Our codebase is pretty huge and digging through all of the modules takes some time. I cannot reproduce this on other agents with a different operating system. Sadly the code on those Ubuntu 16.04 boxes does only work on this specific OS. I don't know if this Ubuntu specific or indeed some puppet code. Third run. msgpack installed again on the puppetserver. also enabled preferred_serialization_format = msgpack on the agent: starting SSL for *:8140... SSL established <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+msgpack, application/x-msgpack, application/vnd.puppet.rich+json, application/json, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40356\r\n\r\n" ... -> "Date: Thu, 05 Nov 2020 17:14:27 GMT\r\n" -> "Content-Type: application/vnd.puppet.rich+msgpack;charset=iso-8859-1\r\n" -> "Transfer-Encoding: chunked\r\n" -> "Connection: keep-alive\r\n" -> "X-Puppet-Version: 6.19.1\r\n" -> "Vary: Accept-Encoding, User-Agent\r\n" -> "Content-Encoding: gzip\r\n" ... Info: Unable to serialize catalog to json, retrying with pson .... the agent run works as well with this setup. but I think the info message is wrong. While it might try pson, afterwards msgpack is used to decode the payload? Or the msgpack serialized to msgpack? Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Tim Meusel (Jira)]

Tim Meusel commented on PUP-10775

Re: Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog` heyho, please let me know if I should test more things or if you need a specific output

Add Comment

[Tim Meusel (Jira)]

Tim Meusel commented on PUP-10775

Re: Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

A short update here. After digging alot through the thousands of nodes, I discovered at least a few CentOS 6 derivates with the exact same behaviour as above: root ~ # facter -p os { architecture => "x86_64", family => "RedHat", hardware => "x86_64", name => "VirtuozzoLinux", release => { full => "6.8", major => "6", minor => "8"

}, selinux => { enabled => false } } root ~ #

so I guess there is something in the catalog that's not compatible with PSON, but with msgpack?

Add Comment

[Tim Meusel (Jira)]

Tim Meusel commented on PUP-10775

Re: Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

wopwop, some progress \o/ I created a testenv and checked it for all binary files: ... root@puppetserver ~ # shopt -s globstar root@puppetserver ~ # file /etc/puppetlabs/code/environments/msgpack2/modules/*/files/**/* | grep ELF /etc/puppetlabs/code/environments/msgpack2/modules/mcollective_agent_dockerinfo/files/mcollective/agent/dockerinfo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped /etc/puppetlabs/code/environments/msgpack2/modules/mcollective_agent_user/files/mcollective/agent/user: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped ... root@puppetserver ~ # There were some more binary blobs. All not listed blobs are distributed like this: file {'bla': ensure => file, source => '///..' } the two blobs that fail use `file()`: file{'bla': ensure => 'file', content => file(..), } In case you want to verify this: https://forge.puppet.com/helio/mcollective_agent_dockerinfo - version 0.1.2 https://forge.puppet.com/exaldraen/mcollective_agent_user - version 0.4.0 I still have some nodes where this works fine with PSON, but a few get `Failed to deserialize Puppet::Resource::Catalog from pson: expected value in object at '"ELF\u0002\u0001\u0'!` on every run. All run the same Puppet version. Let me know if you need more information.

Add Comment

[Henrik Lindberg (Jira)]

Henrik Lindberg commented on PUP-10775

Re: Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

A guess here is that binary files that cause it to fail contain sequences that result in invalid UTF-8 encoding. There is a function binary_file() that works like the file() function but it returns a Binary data type instead of a String. That function should be used instead of file() for binary content. This is supported since puppet 6 where the "rich data" catalog format is the default format (it can serialize all of the data types in the Puppet type system in two different encodings; JSON or Msgpack, where JSON is the default). In addition to handling instance of the Binary data type, it also does the right thing if it gets a string in ASCII-8bit encoding as it will be treated the same way as Binary. In a JSON encoding the binary content will be Base64 encoded text, and in Msgpack it will be a Msgpack binary. Hope that helps.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10775

Re: Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

The case where msgpack is installed on the agent and preferred_serialization_format=msgpack, but you get:

Info: Unable to serialize catalog to json, retrying with pson

means the agent successfully retrieved and deserialized the catalog as richdata+msgpack, but the agent couldn't save the cached catalog as json. This is configured by default in https://github.com/puppetlabs/puppet/blob/1af4a1ad62ec73a1c58cc5f3d073d23a7f889e65/lib/puppet/application/agent.rb#L15. Try running as puppet agent -t --preferred_serialization_format msgpack --catalog_cache_terminus msgpack. Also note msgpack support is experimental. Is there a particular reason why you need/want to use it?

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-10775

Re: Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

This is another case where binary data gets into the catalog and then the error presents itself much later when it's too late. We're going to add a warning to puppetserver when this happens (PUP-11787), add a setting so folks can prevent puppetserver from falling back and in Puppet 8 PSON will be going away. I'm going to close this as a dup of PUP-10928.

Add Comment

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)