Jira (PUP-7808) puppet resource <...> --to_yaml mishandles structured resource values

1 view
Skip to first unread message

Wyatt Alt (JIRA)

unread,
Aug 1, 2017, 12:48:16 PM8/1/17
to puppe...@googlegroups.com
Wyatt Alt created an issue
 
Puppet / Bug PUP-7808
puppet resource <...> --to_yaml mishandles structured resource values
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2017/08/01 9:47 AM
Priority: Normal Normal
Reporter: Wyatt Alt

I'm specifically experiencing this with

puppet resource ec2_instance --to_yaml

using the puppetlabs-aws module from the forge. The module itself doesn't seem to be touching any yaml related stuff. Without the yaml flag I get this data: https://gist.github.com/wkalt/a97a42b84b1afc3cd23236ea32eed3fa and using the yaml flag I get https://gist.github.com/wkalt/a97a42b84b1afc3cd23236ea32eed3fa . Note that the former has ruby maps embedded, which is the bug.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Josh Cooper (JIRA)

unread,
Aug 1, 2017, 12:50:02 PM8/1/17
to puppe...@googlegroups.com

Edwin Wiles (JIRA)

unread,
Mar 27, 2018, 5:27:03 PM3/27/18
to puppe...@googlegroups.com
Edwin Wiles commented on Bug PUP-7808
 
Re: puppet resource <...> --to_yaml mishandles structured resource values

The gist links are identical?

What I'm getting from a custom resource is the following, which is a reasonable translation into a .pp format.

dpw_wsendpointremoterewriterules { 'BAR:FOO_WEBSERVICE':
{{ ensure => 'present',}}
{{ domain => 'BAR',}}
{{ rules => [}}
{{ {}}
{{ 'ServicePortMatchRegexp' => '^{https://foo/webservice/}barGet$',}}
{{ 'RemoteEndpointProtocol' => 'https',}}
{{ 'RemoteEndpointHostname' => 'mock',}}
{{ 'RemoteEndpointPort' => 8443,}}
{{ 'RemoteEndpointURI' => '/webservice/bar.asmx',}}
{{ 'RemoteMQQM' => '',}}
{{ 'RemoteTibcoEMS' => '',}}
{{ 'RemoteWebSphereJMS' => ''}}
{{ }, ...}}

 

But when I specify --to_yaml, I'm getting this half/n/half translation.  The rules, which are an array of hashes are not being translated correctly.

dpw_wsendpointremoterewriterules:
  BAR:FOO_WEBSERVICE:
    ensure: 'present'
    domain: 'BAR'
    rules : [
  {
    'ServicePortMatchRegexp' => '^{https://foo/webservice/}barGet$',
    'RemoteEndpointProtocol' => 'https',
    'RemoteEndpointHostname' => 'mock',
    'RemoteEndpointPort' => 8443,
    'RemoteEndpointURI' => '/webservice/bar.asmx',
    'RemoteMQQM' => '',
    'RemoteTibcoEMS' => '',
    'RemoteWebSphereJMS' => ''
  }, ...

 

I would instead expect to see the following:

 

dpw_wsendpointremoterewriterules:
  BAR:FOO_WEBSERVICE:
    ensure: 'present'
    domain: 'BAR'
    rules:
      - 'ServicePortMatchRegexp' => '^{https://foo/webservice/}barGet$',
        'RemoteEndpointProtocol' => 'https',
        'RemoteEndpointHostname' => 'mock',
        'RemoteEndpointPort' => 8443,
        'RemoteEndpointURI' => '/webservice/bar.asmx',
        'RemoteMQQM' => '',
        'RemoteTibcoEMS' => '',
        'RemoteWebSphereJMS' => ''
      - ...

 

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Josh Cooper (JIRA)

unread,
Jul 6, 2018, 3:55:03 PM7/6/18
to puppe...@googlegroups.com

Craig Castle-Mead (JIRA)

unread,
Nov 11, 2018, 3:03:03 AM11/11/18
to puppe...@googlegroups.com
Craig Castle-Mead commented on Bug PUP-7808
 
Re: puppet resource <...> --to_yaml mishandles structured resource values

I've been working through converting a brownfield AWS environment to Puppet managed and there's a significant amount of nested json coming through when using the --to_yaml (-y) flag with puppet resource. Having the output correctly formatted at all layers in yaml would be a huge help.

Steve Marlow (JIRA)

unread,
Aug 1, 2019, 11:01:04 AM8/1/19
to puppe...@googlegroups.com
Steve Marlow commented on Bug PUP-7808

I have also been seeing this with a client. They're using `puppet resource pe_node_group` on PE which outputs:

 

pe_node_group:
 PE Certificate Authority:
 ensure : 'present'
 classes : {
 'puppet_enterprise::profile::certificate_authority' => {
 
  }
}

The classes parameter is a Hash internally.

YAML implementations can't parse the above output. As a workaround we found that a simple s/=>/:/g fixes the problem, but obviously that's not ideal.

The --to_yaml option in Puppet::Application::Resource ends up invoking Puppet::Resource.to_hierayaml. This function outputs the YAML with the following code:

 

# In Puppet::Resource.to_hierayaml
...
 attributes = attr.collect { |k|
 v = parameters[k]
 " %-#{attr_max}s: %s\n" % [k, Puppet::Parameter.format_value_for_display(v)]
 }.join
...

The Puppet::Parameter.format_value_for_display function formats the Hash in a Puppet style, which is why we get the mixed syntax. The format_value_for_display function doesn't take any additional parameters currently that would easily allow us to do YAML-specific formatting.

 

Puppet::Parameter.format_value_for_display does the following:

def self.format_value_for_display(value)
 Puppet::Pops::Types::StringConverter.convert(value, Puppet::Pops::Types::StringConverter::DEFAULT_PARAMETER_FORMAT)
end

Based on my reading, it seems that using DEFAULT_PARAMETER_FORMAT will cause the StringConvert to default to DEFAULT_HASH_FORMAT (shown below).

DEFAULT_HASH_FORMAT = Format.new('%h')
DEFAULT_HASH_FORMAT.separator = ', '.freeze
DEFAULT_HASH_FORMAT.separator2 = ' => '.freeze
DEFAULT_HASH_FORMAT.container_string_formats = DEFAULT_CONTAINER_FORMATS
DEFAULT_HASH_FORMAT.freeze

This format is appropriate for Puppet code but the ' => ' separator does not work for YAML output.

One possible fix would be to provide Puppet::Pops::Types::StringConverter.convert with a format hash that overrides the separator2 field. It would also require either altering Puppet::Parameter.format_value_for_display to accept the overrides or create a new function (e.g. format_value_for_yaml) as appropriate.

I think that would be sufficient to make this work. I don't know of any other spots where the manifest output would break YAML parsing, but I also haven't done a comprehensive check on that.

 

Josh Cooper (JIRA)

unread,
Sep 11, 2019, 5:42:04 PM9/11/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Sep 13, 2019, 5:35:04 PM9/13/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.4.z
Fix Version/s: PUP 6.10.0
Fix Version/s: PUP 6.4.4
Fix Version/s: PUP 5.5.17

Josh Cooper (JIRA)

unread,
Sep 13, 2019, 5:35:05 PM9/13/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Sep 13, 2019, 5:35:05 PM9/13/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Sep 13, 2019, 10:07:04 PM9/13/19
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Sep 17, 2019, 12:43:03 PM9/17/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Release Notes Summary: The "puppet resource --to_yaml" and "puppet device --to_yaml" did not generate valid YAML when the output contained special characters such as a single quote.
Release Notes: Bug Fix

Jean Bond (JIRA)

unread,
Sep 30, 2019, 1:21:03 PM9/30/19
to puppe...@googlegroups.com

Iain Buclaw (JIRA)

unread,
Oct 16, 2019, 10:36:04 AM10/16/19
to puppe...@googlegroups.com
Iain Buclaw commented on Bug PUP-7808
 
Re: puppet resource <...> --to_yaml mishandles structured resource values

This change caused a regression. The option --to_yaml no longer generates syntactically valid YAML.

Josh Cooper (JIRA)

unread,
Oct 16, 2019, 11:18:03 AM10/16/19
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-7808

Thanks Iain Buclaw. I filed a new ticket PUP-10105. Can you comment how you're are generating that output on the new ticket, including which version of the mysql module you're using?

Reply all
Reply to author
Forward
0 new messages