Ruby, YAML & Hiera data

[Matthew Burgess]

Hi,

It's probably best for you to not ask why we're doing the following,
but here's what we're trying to achieve:

We'd like to programmatically edit our Hiera data, and we're using the
YAML backend. I can successfully load the correct YAML file and make
the changes we require. However, we've hit upon a bit of a stumbling
block if the YAML file contains an interpolation token, if gets
'corrupted' by the default Ruby YAML emitter like so:

ipaddress: "%{::ipaddress_eth0}"

becomes

ipaddress: ! '%{::ipaddress_eth0}'

When I inspect the data following the load of the YAML file, I can see
it's been loaded correctly, so it seems to be when I call YAML.dump()
that it gets 'corrupted'.

Just posting here to see if anyone else has done anything similar and
overcome the same problem; my google-fu led me down the path of some
custom Psych emitters, but the gist I re-used
(https://gist.github.com/mislav/2023978) exhibited the same issue.

The frustration here is that the data we need to change are guaranteed
to *not* have any interpolation tokens, but I can't see a way to
overwrite just the portion of the file that I've changed, hence why
I'm dumping out the full file again and hitting this issue.

Cheers,

Matt.

[jcbollinger]

On Tuesday, July 22, 2014 10:20:29 AM UTC-5, Matthew Burgess wrote:

Hi,

It's probably best for you to not ask why we're doing the following,

Cause if you told us, you'd have to kill us?

 

but here's what we're trying to achieve:

We'd like to programmatically edit our Hiera data, and we're using the
YAML backend.  I can successfully load the correct YAML file and make
the changes we require.  However, we've hit upon a bit of a stumbling
block if the YAML file contains an interpolation token, if gets
'corrupted' by the default Ruby YAML emitter like so:

ipaddress: "%{::ipaddress_eth0}"

becomes

ipaddress: ! '%{::ipaddress_eth0}'

When I inspect the data following the load of the YAML file, I can see
it's been loaded correctly, so it seems to be when I call YAML.dump()
that it gets 'corrupted'.

The '!' is the (correct) "non-specific tag" for the value, chosen by default when the YAML parser loads the untagged data.  I am uncertain why the emitter is outputting it, but I don't think it changes the meaning.  Can you successfully load the output back into Ruby?  If so, then would it be acceptable to just accept the current result?

John

[Matthew Burgess]

On 23 July 2014 15:20, jcbollinger <John.Bo...@stjude.org> wrote:
>
>
> On Tuesday, July 22, 2014 10:20:29 AM UTC-5, Matthew Burgess wrote:
>>
>> Hi,
>>
>> It's probably best for you to not ask why we're doing the following,
>
>
>
> Cause if you told us, you'd have to kill us?

Not at all :-) I just didn't think the details were particular
pertinent to the issue at hand. As it is, we're offering a
self-service tool to our development teams to deploy the RPMs they
build through their CI processes to environments further up the
deployment lifecycle. That tool has a UI that accepts an RPM +
version number as input, then goes and updates our Hiera data, then
triggers a Puppet run to deploy the RPM all without the developers
having to be given access to our hiera data directly or have to
understand it.

>
>
>>
>> but here's what we're trying to achieve:
>>
>> We'd like to programmatically edit our Hiera data, and we're using the
>> YAML backend. I can successfully load the correct YAML file and make
>> the changes we require. However, we've hit upon a bit of a stumbling
>> block if the YAML file contains an interpolation token, if gets
>> 'corrupted' by the default Ruby YAML emitter like so:
>>
>> ipaddress: "%{::ipaddress_eth0}"
>>
>> becomes
>>
>> ipaddress: ! '%{::ipaddress_eth0}'
>>
>> When I inspect the data following the load of the YAML file, I can see
>> it's been loaded correctly, so it seems to be when I call YAML.dump()
>> that it gets 'corrupted'.
>
>
>
> The '!' is the (correct) "non-specific tag" for the value, chosen by default
> when the YAML parser loads the untagged data. I am uncertain why the
> emitter is outputting it, but I don't think it changes the meaning. Can you
> successfully load the output back into Ruby? If so, then would it be
> acceptable to just accept the current result?

I must admit to not even testing whether I could either load it back
into Ruby or whether Hiera would understand it. Until I'd read a bit
more I'd thought it was some kind of 'error' marker denoting that the
parser couldn't understand it. Now I know through both that reading
and your response what the '!' means, I'd imagine it just works.
However, since then, a colleague who is working on a Hiera-tidying
tool for us found a Java library, SnakeYAML, that doesn't emit the '!'
character. (We stupidly haven't enforced any indentation/quoting
coding standards on the project yet, so there's a whole range of
different styles in play currently). The rest of the self-service
tool is being developed in Java, so moving the modification code to
the same library as the tidying tool seems like a much better fit.

Thanks,

Matt.