Jira (PUP-10105) puppet resource --to

puppet resource --to_yaml regression

Change By:	Josh Cooper
Team:	Coremunity

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

Josh Cooper (JIRA)

unread,

Oct 16, 2019, 11:16:03 AM10/16/19

to puppe...@googlegroups.com

Josh Cooper created an issue

Puppet /

puppet resource --to_yaml regression

Issue Type:	Bug
Assignee:	Unassigned
Created:	2019/10/16 8:15 AM
Priority:	Normal
Reporter:	Josh Cooper

From https://tickets.puppetlabs.com/browse/PUP-7808?focusedCommentId=693345&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-693345

It seems we're outputting symbols and ruby class tags

Josh Cooper (JIRA)

unread,

Oct 16, 2019, 11:39:03 AM10/16/19

to puppe...@googlegroups.com

Josh Cooper updated an issue

Puppet /

puppet resource --to_yaml regression

Change By:	Josh Cooper

From https://tickets.puppetlabs.com/browse/PUP-7808?focusedCommentId=693345&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-693345

It seems we This change caused a regression. The option --to_yaml no longer generates syntactically valid YAML.

Before pull request #7718:
{code:java}
mysql_database:
  PERCONA_SCHEMA:
    ensure : ' re outputting symbols and present'
    charset: 'utf8'
    collate: 'utf8_general_ci'
{code}
After pull request #7718:
{code:java}
mysql_database:
  PERCONA_SCHEMA:
    ensure: present
    charset: ! ruby class tags /string:Puppet::Util::Execution::ProcessOutput utf8
    collate: !ruby/string:Puppet::Util::Execution::ProcessOutput utf8_general_ci
{code}

Iain Buclaw (JIRA)

unread,

Oct 16, 2019, 11:45:02 AM10/16/19

to puppe...@googlegroups.com

Iain Buclaw commented on

Josh Cooper -

To reproduce: /opt/puppetlabs/bin/puppet resource --to_yaml mysql_database.

The mysql module installed is puppetlabs-mysql 10.2.0.

Josh Cooper (JIRA)

unread,

Oct 16, 2019, 11:52:04 AM10/16/19

to puppe...@googlegroups.com

Josh Cooper commented on

The output is syntactically valid YAML (as it's generated using YAML.dump). But the serialized ProcessOutput class is unexpected, and will cause failures when trying to parse the YAML from another language or in ruby if the puppet classes are not loaded:

# cat out.yaml

mysql_database:

  PERCONA_SCHEMA:

    ensure: present

    charset: !ruby/string:Puppet::Util::Execution::ProcessOutput utf8

    collate: !ruby/string:Puppet::Util::Execution::ProcessOutput utf8_general_ci

# /opt/puppetlabs/puppet/bin/ruby -ryaml -e "puts YAML.load_file('out.yaml')"

Traceback (most recent call last):

	43: from -e:1:in `<main>'

	42: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/psych.rb:497:in `load_file'

<snip>

	 1: from /opt/puppetlabs/puppet/lib/ruby/2.5.0/psych/class_loader.rb:54:in `resolve'

/opt/puppetlabs/puppet/lib/ruby/2.5.0/psych/class_loader.rb:54:in `path2class': undefined class/module Puppet:: (ArgumentError)

However this works:

# /opt/puppetlabs/puppet/bin/ruby -ryaml -rpuppet -e "puts YAML.load_file('out.yaml')"

{"mysql_database"=>{"PERCONA_SCHEMA"=>{"ensure"=>"present", "charset"=>"utf8", "collate"=>"utf8_general_ci"}}}

As does this code to strip the tags:

require 'yaml'

data = YAML::parse_file(ARGV[0])

data.root.each do |o|

  if o.respond_to?(:tag=) && o.tag != nil && o.tag.start_with?("!ruby")

    o.tag = nil

end

end

puts data.to_ruby

# /opt/puppetlabs/puppet/bin/ruby reader.rb out.yaml

{"mysql_database"=>{"PERCONA_SCHEMA"=>{"ensure"=>"present", "charset"=>"utf8", "collate"=>"utf8_general_ci"}}}

The mysql module should call .to_s on the process output, and puppet pops serialization should not emit ProcessOutput objects.

Iain Buclaw (JIRA)

unread,

Oct 16, 2019, 11:53:03 AM10/16/19

to puppe...@googlegroups.com

Iain Buclaw commented on

In the function `to_data_hash`

    params = {}

    self.to_hash.each_pair do |param, value|

      # Don't duplicate the title as the namevar

      unless param == namevar && value == title

        params[param.to_s] = Puppet::Resource.value_to_json_data(value)

end

end

Replacing the params setting with the following gets us closer to what we'd expect.

params[param.to_s] = Puppet::Resource.value_to_json_data(Puppet::Parameter.format_value_for_display(value))

Output:

mysql_database:

  PERCONA_SCHEMA:

    ensure: "'present'"

    charset: "'utf8'"

    collate: "'utf8_general_ci'"

So there's something in Puppet::Parameter.format_value_for_display that gets us to the desired output.

Iain Buclaw (JIRA)

unread,

Oct 16, 2019, 11:55:03 AM10/16/19

to puppe...@googlegroups.com

Iain Buclaw commented on

Ah, I did not see that you responded in parallel.

Josh Cooper (JIRA)

unread,

Oct 16, 2019, 12:04:03 PM10/16/19

to puppe...@googlegroups.com

Josh Cooper commented on

Henrik Lindberg we recently changed puppet resource --to_yaml to use Puppet::Resource#to_data_hash, which uses pops serialization, and then we call YAML.dump on that. It appears the mysql module generates Puppet::Resource instances whose attributes contain the result of running Puppet::Util::Execution::ProcessOutput objects. Note those objects extend ruby String, which is a hack we added so that we didn't have to change the Puppet::Util::Execution API. My question is can pops serialization detect that class and emit the to_s version, and possibly warn?

Henrik Lindberg (JIRA)

unread,

Oct 16, 2019, 12:26:04 PM10/16/19

to puppe...@googlegroups.com

Henrik Lindberg commented on

Josh Cooper I suppose so, it is in the pops/serialization/to_data_converter.rb - but I don't know the best place to make the change without digging in. Problem if class is derived from Ruby String since is_a?(String) would return true. Thomas Hallgren may know more precisely...