serializing a pillar with multi-line values into a yaml file

[tiadobatima]

Hello guys,

I've been trying to serialize a pillar with multi-line values into a yaml file and for whatever reason salt is adding an extra line break in to the serialized string. I tried with both file.serialize and file.managed with the same results. Basically, I just want to dump the whole pillar as-is to the file. Here's one of many things I attempted. Any idea what I'm doing wrong?

============================== PILLAR ==============================

my_cluster:

  my_app:

    certificates:

      1: '-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC\nQ04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g\nQ2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0\naW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa\n-----END CERTIFICATE-----'

      2: "-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC\nQ04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g\nQ2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0\naW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa\n-----END CERTIFICATE-----"

      3: |

        -----BEGIN CERTIFICATE-----

        MIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC

        Q04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g

        Q2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0

        aW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa

        -----END CERTIFICATE-----

===================================================================

============================== STATE ==============================

/etc/my_fake_config.yaml:

  file.serialize:

    - makedirs: True

    - dataset_pillar: my_cluster

    - formatter: yaml

===================================================================

============================== /etc/my_fake_config.yaml ==============================

my_app:

  certificates:

    1: '-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC\nQ04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g\nQ2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0\naW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa\n-----END

      CERTIFICATE-----'

    2: '-----BEGIN CERTIFICATE-----

      MIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC

      Q04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g

      Q2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0

      aW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa

      -----END CERTIFICATE-----'

    3: '-----BEGIN CERTIFICATE-----

      MIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC

      Q04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g

      Q2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0

      aW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa

      -----END CERTIFICATE-----

===================================================================

Thanks guys!

g.

[tiadobatima]

Interestingly enough, the json formatter seems to do the right thing, ie, no extra line breaks.

[Florian Ermisch]

Hi g.,

You can just point to the key of a pillar which contains the (multiline) content you want in your file using the "content_pillar" option of ``file.managed`` [0].
Works just fine for me [1].

Regards, Florian

[0] https://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html#salt.states.file.managed
[1] https://github.com/0xf10e/trustedCAs-formula/blob/master/trustedCAs/CAs.sls#L8

[tiadobatima]

Thanks Florian,

It looks like this is only working when the pillar is not a nested structure. Ie, in the CA example you provided it works fine if we have this pillar:

trustedCAs:

ca1: |

--- CERT ---

bla bla

bla bla

--- END CERT ---

ca2: |

--- CERT ---

another cert

another cert

--- END CERT ---

Your "file.managed" state is assigning only the value of "trustedCAs:ca1" and "trustedCAs:ca2" to "contents_pillar".

But if we try to assign the whole "trustedCAs" to "contents_pillar", the yaml for the multi-line certificate gets rendered with extra line breaks.

What I want to do is to dump the whole trustedCAs structure into a yaml file, not just the scalar representing the certificate. As of now there seems to be no way to dump whole yaml files from pillars. Does it sound like a serious enough bug?

Thanks!

g.

[Florian Ermisch]

Hi g.,

might not be a bug but providing a _list_ to the parameter
`contents_pillar` would be a neat feature. Then you could
just iterate over the sub-keys to list all the pillar to
include in your template.

/path/to/file:
file.managed:
- pillar_contents:
{% for sub in salt['pillar.get']('trustedCAs').keys() %}
- {{ salt['pillar.get']('trustedCAs:'+sub) }}
{% endfor %}

Regards, Florian

PS: Automatically expanding a nested structure had no
obvious correct behavior (I can think of).
Also a missing sub-key/mistyped pillar-key could
mess up the file quite heavily instead of just
letting the state fail.

>> <javascript:>>:

[Seth House]

On Tue, Oct 27, 2015 at 3:21 PM, tiadobatima <gbar...@gmail.com> wrote:
> But if we try to assign the whole "trustedCAs" to "contents_pillar", the
> yaml for the multi-line certificate gets rendered with extra line breaks.

The trick here is the multi-line values output into the sls file must
have the correct indentation, and they must _also_ be marked as
multi-line. Jinja is not YAML-aware and so you must instruct Jinja to
keep the same indentation level. E.g.,

/some/file.txt:
file.managed:
- contents: |
{{ salt.pillar.get('some:pillar:value1') | indent(8) }}

-- non-Jinja text content here ---

{{ salt.pillar.get('some:pillar:value2') | indent(8) }}

This is what the `contents_pillar` argument is attempting to avoid but
sometimes it's not flexible enough and you just have to do it
manually. This kind of example use to be in the docs but was pulled
out when that argument was added. I would be in favor of adding it
back. If you're interested in adding it, I'll pull it in.

[Gustavo Baratto]

Thanks Seth...

The example you provide reinforces the behaviour I described in my previous example: we can interpolate a single value, but not a nested structure. Ie, if "some:pillar:value1" is a string/integer/float, we're good, but if it's a dictionary (nested structure), an extra newline add to *any* existing newline present in a string in the pillar:

We end up with something like "2" and "3" in the example below:

============================== /etc/my_fake_config.yaml ==============================

my_app:

  certificates:

    1: '-----BEGIN CERTIFICATE-----\nMIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC\nQ04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g\nQ2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0\naW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa\n-----END

      CERTIFICATE-----'

    2: '-----BEGIN CERTIFICATE-----

      MIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC

      Q04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g

      Q2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0

      aW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa

      -----END CERTIFICATE-----'

    3: '-----BEGIN CERTIFICATE-----

      MIID9zCCAt+gAwIBAgIESJ8AATANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMC

      Q04xMjAwBgNVBAoMKUNoaW5hIEludGVybmV0IE5ldHdvcmsgSW5mb3JtYXRpb24g

      Q2VudGVyMUcwRQYDVQQDDD5DaGluYSBJbnRlcm5ldCBOZXR3b3JrIEluZm9ybWF0

      aW9uIENlbnRlciBFViBDZXJ0aWZpY2F0ZXMgUm9vdDAeFw0xMDA4MzEwNzExMjVa

      -----END CERTIFICATE-----

===================================================================

The use case is to write a yaml config file for a generic application directly from a pillar. Since lots of apps use yaml/json based config files, I think there should be a very easy way for people to do that without having to manually parse and reassemble a data structure that's already in its final format.

Cheers,

g.

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/SwnrmB2KlaQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Seth House]

There's three ways to think about non-string content:

`file.serialize` takes a data structure and blindly writes it to disk
_verbatim_ as a serialized version of that exact data structure. It
doesn't know anything about the nature of the content you're writing.

There was a new module type added to Salt recently called serializers.
It would be a good fit for what you want to accomplish, though It is
currently incomplete (still missing state and execution module
wrappers). The idea is these modules accept a given data structure and
they _do_ know how the content is intended to be used. So a
hypothetical "ca_certs" serializer module would know how to take a
dictionary and write it to disk in the correct format. You can read
more about that effort in the issue below.

https://github.com/saltstack/salt/issues/22257

Last, as you know `file.managed` only works with string content. But
you know how your data structure can be used so you can combine
Florian's suggestion with mine to easily make a state that knows how
to take a given data structure and extract/format the relevant data
from it in order to _produce_ string content. A custom, one-off
"serializer" if you will. E.g.:

/some/file.txt:
file.managed:
- contents: |
{% for ca_name, ca_cert in salt.pillar.get('trustedCAs').items() %}
{{ ca_cert | indent(8) }}
{% endfor %}

[Dani C]

Nice examples Seth/ Florian.

Seth, not sure if you are working for SaltStack like Colton but if you do would be awsome if you can pitch internally to have a Salt repo where we can add all this examples/ examples which a lot of folks are coming up here/ irc/ GitHub and don't fit into docs (not this are not formulas at all)

Will help tons of people, trust me ...

[Seth House]

I do work for SaltStack. These kinds of examples should definitely be
in the docs IMO. This particular example was in the `file.managed`
docs at one point (I put it there). Looks like it was pulled out when
the `contents_pillar` argument was added but it should be added back
since that argument is a nice convenience but doesn't work for complex
needs like in this thread.

If you'd care to add it again, ping me (@whiteinge) in the pull req
and I'll merge it.

> --
> You received this message because you are subscribed to the Google Groups
> "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an