Jira (PUP-10654) Supplying data of mixed data types (Variant) via Hiera (YAML backend)

19 views
Skip to first unread message

Josh Cooper (Jira)

unread,
Sep 1, 2020, 12:44:03 PM9/1/20
to puppe...@googlegroups.com
Josh Cooper moved an issue
 
Puppet / Improvement PUP-10654
Supplying data of mixed data types (Variant) via Hiera (YAML backend)
Change By: Josh Cooper
Key: SERVER PUP - 2873 10654
Project: Puppet Server
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Sep 1, 2020, 2:14:04 PM9/1/20
to puppe...@googlegroups.com

Henrik Lindberg (Jira)

unread,
Sep 1, 2020, 6:56:03 PM9/1/20
to puppe...@googlegroups.com

You can achieve the same by using a hiera lookup option to convert a string to a Regexp. Then you alias the typecasted key. For example for a parameter foo::bar, if value should be a string, just bind it to a string, if you wanted to be a regexp bind it to

"%{alias("foo::bar::regexp")}"

and bind the regexp string to that key and associate a lookup_option that converts it to a Regexp.

You could also have your own backed function if you have lots of these.

Josh Cooper A cool feature for hiera would be to support pcore serialization, could be an option to json/yaml backends or a separate backend - it would read json/yaml, and then run that through pcore deserialize before returning the data hash. That way it would be possible to safely produce and return any pcore compatible data type.

Nacho Barrientos (Jira)

unread,
Sep 2, 2020, 4:24:03 AM9/2/20
to puppe...@googlegroups.com

Thanks for you input!

I moved to PUP project as all of Hiera5 is implemented in the puppet project.

Sorry about that, I was not sure

Can you confirm that adding Regexp to https://github.com/puppetlabs/puppet/blob/17b2735731f6468aef92a5540b35292e056726e3/lib/puppet/functions/yaml_data.rb#L25 solves the issue?

Yes, the catalog compiles and the catalog diff is empty

Henrik Lindberg (Jira)

unread,
Sep 2, 2020, 2:27:03 PM9/2/20
to puppe...@googlegroups.com

Josh Cooper IMHO it is a mistake to allow any non YAML/JSON supported data types into the data as that prohibits a standard reader from reading the data. So... no ruby specific extensions should be allowed.

Josh Cooper (Jira)

unread,
Sep 2, 2020, 4:56:03 PM9/2/20
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-10654

IMHO it is a mistake

Yeah I had a similar thought last night. Adding a builtin rich_data backend to complement yaml_data, etc makes sense to me. Better that than trying to shoehorn non-YAML/JSON data into the yaml_data, etc backends.

Nacho Barrientos (Jira)

unread,
Sep 3, 2020, 3:37:04 AM9/3/20
to puppe...@googlegroups.com

Thanks both for your input.

I agree it makes sense to respect the native data types of the data backend and use a richer backend for these cases. Technically speaking is the correct path to take, however in our environment introducing an extra backend would add complexity for our users, more training to do and more things to maintain so I think we're just going to split the parameter in this particular case. It's a pity given the existence of a type system but it's the simplest approach at the moment so we can carry on with the upgrade.

Henrik Lindberg (Jira)

unread,
Sep 3, 2020, 6:28:04 AM9/3/20
to puppe...@googlegroups.com

I also think a RichData backend could be difficult for people to adopt especially since you may want this support for both yaml, json, and eyaml. This is why I suggested adding deserialization as an option to existing backends. RichData is really an encoding on top of an existing data format, so this is a natural fit.

This could also be done at the lookup_option level similar to how convert_to works. Unfortunately it cannot be done with convert_to since IIRC there is no data type that accepts Pcore as input to its new() and that would return a deserialized value - one could be created though.

A general lookup_option feature would be to add a convert_with which has the name of a function as its value. It would be called before a convert_to if both are used. The convert_with would take RichData as an argument and produce Any (although in hiera it would be restricted to returning RichData. Then you can plug in a function like this one: https://github.com/hlindberg/tahu/blob/master/lib/puppet/functions/tahu/convert_from_rich_data.rb to do the deserialization. Or users can do their own simple conversion per key.

The same feature could be added as an option to backends. The hiera implementation would then call the given function for every value the backend returns. This way the backend implementations does not have to change.

Henrik Lindberg (Jira)

unread,
Sep 3, 2020, 6:33:04 AM9/3/20
to puppe...@googlegroups.com

Possibly hacky suggestion... The RichData datatype could be given a new function that takes two parameters, a value, and an extra argument "encode" or "decode" - then it would either "serialize" or "deserialize".

Josh Cooper (Jira)

unread,
Aug 18, 2021, 5:52:03 PM8/18/21
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-10654

Could we just add a lookup_option like supports_pcore: true. If set, then deserialize whatever the lookup returns to an instance of Puppet::LookupValue (currently defined as RichData). Then pass that through the normal convert_to logic in: https://github.com/puppetlabs/puppet/blob/d567575ba8c5b2c903044b80b0adaab176c8da5d/lib/puppet/pops/lookup/lookup_adapter.rb#L72

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Josh Cooper (Jira)

unread,
Jul 22, 2022, 2:46:03 PM7/22/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Component/s: Hiera & Lookup
This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages