puppet resource <...> --to_yaml mishandles structured resource values
Description
Environment
Acceptance Criteria
is duplicated by
relates to
Activity
Josh CooperOctober 16, 2019 at 3:17 PM
Thanks . I filed a new ticket . Can you comment how you're are generating that output on the new ticket, including which version of the mysql module you're using?
Iain BuclawOctober 16, 2019 at 2:35 PMEdited
This change caused a regression. The option --to_yaml no longer generates syntactically valid YAML.
Before pull request #7718:
mysql_database:
PERCONA_SCHEMA:
ensure : 'present'
charset: 'utf8'
collate: 'utf8_general_ci'
After pull request #7718:
mysql_database:
PERCONA_SCHEMA:
ensure: present
charset: !ruby/string:Puppet::Util::Execution::ProcessOutput utf8
collate: !ruby/string:Puppet::Util::Execution::ProcessOutput utf8_general_ci
Josh CooperSeptember 14, 2019 at 2:06 AM
Stephen MarlowAugust 1, 2019 at 3:00 PMEdited
I have also been seeing this with a client. They're using `puppet resource pe_node_group` on PE which outputs:
pe_node_group:
PE Certificate Authority:
ensure : 'present'
classes : {
'puppet_enterprise::profile::certificate_authority' => {
}
}
The classes parameter is a Hash internally.
YAML implementations can't parse the above output. As a workaround we found that a simple s/=>/:/g fixes the problem, but obviously that's not ideal.
The --to_yaml option in Puppet::Application::Resource ends up invoking Puppet::Resource.to_hierayaml. This function outputs the YAML with the following code:
# In Puppet::Resource.to_hierayaml
...
attributes = attr.collect { |k|
v = parameters[k]
" %-#{attr_max}s: %s\n" % [k, Puppet::Parameter.format_value_for_display(v)]
}.join
...
The Puppet::Parameter.format_value_for_display function formats the Hash in a Puppet style, which is why we get the mixed syntax. The format_value_for_display function doesn't take any additional parameters currently that would easily allow us to do YAML-specific formatting.
Puppet::Parameter.format_value_for_display does the following:
# In Puppet::Parameter
def self.format_value_for_display(value)
Puppet::Pops::Types::StringConverter.convert(value, Puppet::Pops::Types::StringConverter::DEFAULT_PARAMETER_FORMAT)
end
Based on my reading, it seems that using DEFAULT_PARAMETER_FORMAT will cause the StringConvert to default to DEFAULT_HASH_FORMAT (shown below).
# In Puppet::Pops::Types::StringConverter
DEFAULT_HASH_FORMAT = Format.new('%h')
DEFAULT_HASH_FORMAT.separator = ', '.freeze
DEFAULT_HASH_FORMAT.separator2 = ' => '.freeze
DEFAULT_HASH_FORMAT.container_string_formats = DEFAULT_CONTAINER_FORMATS
DEFAULT_HASH_FORMAT.freeze
This format is appropriate for Puppet code but the ' => ' separator does not work for YAML output.
One possible fix would be to provide Puppet::Pops::Types::StringConverter.convert with a format hash that overrides the separator2 field. It would also require either altering Puppet::Parameter.format_value_for_display to accept the overrides or create a new function (e.g. format_value_for_yaml) as appropriate.
I think that would be sufficient to make this work. I don't know of any other spots where the manifest output would break YAML parsing, but I also haven't done a comprehensive check on that.
Craig Castle-MeadNovember 11, 2018 at 8:02 AM
I've been working through converting a brownfield AWS environment to Puppet managed and there's a significant amount of nested json coming through when using the --to_yaml (-y) flag with puppet resource. Having the output correctly formatted at all layers in yaml would be a huge help.
Edwin WilesMarch 27, 2018 at 9:26 PMEdited
The gist links are identical?
What I'm getting from a custom resource is the following, which is a reasonable translation into a .pp format.
dpw_wsendpointremoterewriterules { 'BAR:FOO_WEBSERVICE':
ensure => 'present',
domain => 'BAR',
rules => [
{
'ServicePortMatchRegexp' => '^{https://foo/webservice/}barGet$',
'RemoteEndpointProtocol' => 'https',
'RemoteEndpointHostname' => 'mock',
'RemoteEndpointPort' => 8443,
'RemoteEndpointURI' => '/webservice/bar.asmx',
'RemoteMQQM' => '',
'RemoteTibcoEMS' => '',
'RemoteWebSphereJMS' => ''
}, ...
But when I specify --to_yaml, I'm getting this half/n/half translation. The rules, which are an array of hashes are not being translated correctly.
dpw_wsendpointremoterewriterules:
BAR:FOO_WEBSERVICE:
ensure: 'present'
domain: 'BAR'
rules : [
{
'ServicePortMatchRegexp' => '^{[https://foo/webservice/]}barGet$',
'RemoteEndpointProtocol' => 'https',
'RemoteEndpointHostname' => 'mock',
'RemoteEndpointPort' => 8443,
'RemoteEndpointURI' => '/webservice/bar.asmx',
'RemoteMQQM' => '',
'RemoteTibcoEMS' => '',
'RemoteWebSphereJMS' => ''
}, ...
I would instead expect to see the following:
dpw_wsendpointremoterewriterules:
BAR:FOO_WEBSERVICE:
ensure: 'present'
domain: 'BAR'
rules:
- 'ServicePortMatchRegexp' => '^{[https://foo/webservice/]}barGet$',
'RemoteEndpointProtocol' => 'https',
'RemoteEndpointHostname' => 'mock',
'RemoteEndpointPort' => 8443,
'RemoteEndpointURI' => '/webservice/bar.asmx',
'RemoteMQQM' => '',
'RemoteTibcoEMS' => '',
'RemoteWebSphereJMS' => ''
- ...
I'm specifically experiencing this with
puppet resource ec2_instance --to_yaml
using the puppetlabs-aws module from the forge. The module itself doesn't seem to be touching any yaml related stuff. Without the yaml flag I get this data: https://gist.github.com/wkalt/a97a42b84b1afc3cd23236ea32eed3fa and using the yaml flag I get https://gist.github.com/wkalt/a97a42b84b1afc3cd23236ea32eed3fa . Note that the former has ruby maps embedded, which is the bug.