Puppet 5.0.1: JSON to PSON automatic downgrade doesn't account for binary facts

[Dominic Scheirlinck]

Thought I'd leave a note about an issue I ran into when upgrading to Puppet 5.0.1, in case someone else is wrestling with the same thing (also just to provide a result for some poor person Googling it after me). Turned out to be a whole bunch of factors:

- The default serialization format is now JSON (previously PSON), which doesn't support arbitrary binary data (only UTF-8 strings)

- PUP-7602 is supposed to automatically downgrade back to PSON if there's binary data in the catalog

- But this doesn't seem to account for binary facts - you get an error on apply: "Error: Failed to apply catalog: Could not render to json: source sequence is illegal/malformed utf-8" 

I surmise that a binary fact is at fault because of a debug message from Facter: "Debug: Facter: Received a log message with invalid encoding:"fact \"ec2_userdata\" has resolved to [...]" (escaped data follows) - and because I'm not shipping binary in my catalog otherwise. This is particularly annoying if you're using local VMs to test your puppet server upgrade, because you won't run into it until you run it on your production EC2 node :)

The EC2 user data is gzipped to work around a user_data size limitation. (i.e. https://www.terraform.io/docs/providers/template/d/cloudinit_config.html#gzip) - I guess I'm not close enough to the limit that I could pay the size penalty and base-64 encode the compressed user-data as well - but you can't change user data while the instance is running, so it's not nice as a workaround.

I've seen the Facter blocklists documentation, but it doesn't make it clear whether you can block a specific fact instead of the whole EC2 blockgroup - or more accurately, it appears I can't. (I am using ec2_metadata, to get ['placement']['availability-zone'] so I don't want to block the group - I'd only want to block the ec2_userdata fact). I guess I could try overwriting the value with a blank string (a la https://tickets.puppetlabs.com/browse/FACT-1354?focusedCommentId=410038&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-410038)?

For now, I've just reverted to PSON serialization. That's not deprecated, right? (Just the default was changed?)

(Also, I'd file a JIRA ticket, but I'm not sure whether support for binary fact values is desired or necessary, whether Facter should be giving up on passing a fact if it has a binary value, or whether a PUP-7602-style serialization fallback would be better, etc.)

Regards,

Dominic

[Eric Sorenson]

Hi Dominic, thanks for tracking this down and documenting it so thoroughly. Some responses inline:

On Tuesday, August 1, 2017 at 9:56:28 PM UTC-7, Dominic Scheirlinck wrote:

Thought I'd leave a note about an issue I ran into when upgrading to Puppet 5.0.1, in case someone else is wrestling with the same thing (also just to provide a result for some poor person Googling it after me). Turned out to be a whole bunch of factors:

- The default serialization format is now JSON (previously PSON), which doesn't support arbitrary binary data (only UTF-8 strings)

- PUP-7602 is supposed to automatically downgrade back to PSON if there's binary data in the catalog

- But this doesn't seem to account for binary facts - you get an error on apply: "Error: Failed to apply catalog: Could not render to json: source sequence is illegal/malformed utf-8" 

I surmise that a binary fact is at fault because of a debug message from Facter: "Debug: Facter: Received a log message with invalid encoding:"fact \"ec2_userdata\" has resolved to [...]" (escaped data follows) - and because I'm not shipping binary in my catalog otherwise. This is particularly annoying if you're using local VMs to test your puppet server upgrade, because you won't run into it until you run it on your production EC2 node :)

The EC2 user data is gzipped to work around a user_data size limitation. (i.e. https://www.terraform.io/docs/providers/template/d/cloudinit_config.html#gzip) - I guess I'm not close enough to the limit that I could pay the size penalty and base-64 encode the compressed user-data as well - but you can't change user data while the instance is running, so it's not nice as a workaround.

Yeah, this is one of the main shifts between pson and json. Since there's not type hinting for facter, we assume everything's a string, and while pson used to best-effort deal with binary encodings, json won't support it.  Seems like you could either un-gzip the user data or b64 encode it.

 

I've seen the Facter blocklists documentation, but it doesn't make it clear whether you can block a specific fact instead of the whole EC2 blockgroup - or more accurately, it appears I can't. (I am using ec2_metadata, to get ['placement']['availability-zone'] so I don't want to block the group - I'd only want to block the ec2_userdata fact). I guess I could try overwriting the value with a blank string (a la https://tickets.puppetlabs.com/browse/FACT-1354?focusedCommentId=410038&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-410038)?

Unfortunately the blocklist is currently at the level of the resolution group, as you've discovered. The overwrite would work OK and be pretty simple, assuming you don't actually *need* any of the userdata for Puppet to run. 

For now, I've just reverted to PSON serialization. That's not deprecated, right? (Just the default was changed?)

That's right. One thing to note though... json is way faster. If you can get around this, the performance gains probably make it worthwhile to shift to json.

 

(Also, I'd file a JIRA ticket, but I'm not sure whether support for binary fact values is desired or necessary, whether Facter should be giving up on passing a fact if it has a binary value, or whether a PUP-7602-style serialization fallback would be better, etc.)

It'd be great to have a bug on this to talk over the options.  Thanks again for the sleuthing!

--eric0