Merging pillar data

880 views
Skip to first unread message

Alfredo Palhares

unread,
Mar 31, 2015, 1:22:58 PM3/31/15
to salt-users
Hello fellow salt users,


I have a Sensu formula that gets the sensu client configuration from pillar
data and serializes it into the configuration directly into json.

Now on the saltstack repository I have the following structure

```
sensu
├── groups
│   ├── group1.sls
│   ├── group2.sls
│   ├── group3.sls
│   ├── group4.sls
```

The files should contain something like this :
```yaml
sensu:
client:
subscriptions:
- mysubscription-group
```

After that a pillar/top.sls would look like:
```yaml
"mynode*":
- sensu/groups/group1
- sensu/groups/group2
```

The point it to have pillar data merged on the node like this.

```yaml
sensu:
client:
option1: value1
foo: bar
subscriptions:
- mysubscription-group
- mysubscription-group2
- mysubscription-group3
```

And its true on most configs, but since there is the subscrptions group the
array elements do not get merged, just overriden by the last value.

I would like to merge the subscriptions array so the the sensu client
subscribed the following all the groups I set them to.

How can I achieve the intended results ?

--
Regards,
Alfredo Palhares

Alfredo Palhares

unread,
Apr 1, 2015, 6:13:48 AM4/1/15
to salt-users
Hello,

Any ideas ?


--
Regards,
Alfredo Palhares

Florian Ermisch

unread,
Apr 2, 2015, 4:12:10 AM4/2/15
to salt-...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Alfredo,

I use templating in pillar when I have to do any form of data manipulation (like applying a default set and just changing a few values). You can include a template and pass it your groups so it can include and merge the appropriate files containing your data.
Gets a little more involved though.

Regards, Florian
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFTBAEBCAA9BQJVHPnQNhxGbG9yaWFuIEVybWlzY2ggPGZsb3JpYW4uZXJtaXNj
aEBhbHVtbmkudHUtYmVybGluLmRlPgAKCRAu8tzCHoBI/f+LB/9gfvjpSvb/UYFE
XKI4E7e5xsXM9IZJ0b6qkf6SEXTPhHdXh20g4vBlMgR5xv0oMJr1IQ+7oXYQBQ/g
cLzr1CevAJJoSv58nW++GoWRz3gpm4C/O92P8aPWIR283+Jw6hsOPlbjg5+ocNcv
xr5uFzSSWOLALape4lbBamagNP5Qukec7rWgGul2k421pvorJbfDaKHTGNCdn2Av
ti/afLP15HjvYdsiMXUDoivbvsu0eOp51MsMajnXkfFkxHxdwy3JiGDZtP1LnIiu
yqa+QzkOoe1gGFgxIUJZrI6jr7NsJrFhNl6QhZ3fHvmstcswHpTKUPd1NSCndFuk
fIOTQR8U
=AU1l
-----END PGP SIGNATURE-----

Alfredo Palhares

unread,
Apr 2, 2015, 7:04:17 PM4/2/15
to salt-users
Hello Florian,

> I use templating in pillar when I have to do any form of data manipulation (like applying a default set and just changing a few values). You can include a template and pass it your groups so it can include and merge the appropriate files containing your data.
> Gets a little more involved though.
Can you give me an example for this use or point me to documentation ?

I would be very thankful.

--
Alfredo Palhares
GPG/PGP Key Fingerprint
68FC B06A 6C22 8B9B F110
38D6 E8F7 4D1F 0763 CAAD
signature.asc

Daniel Jagszent

unread,
Apr 2, 2015, 7:31:00 PM4/2/15
to salt-...@googlegroups.com

Hello Alfredo,

I haven’t tried that myself but you should be able to use the !aggregate tag of the yamlex renderer
http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.yamlex.html

something along those lines:

group1.sls:

#!yamlex
sensu:
 client:
   subscriptions: !aggregate
   - mysubscription-group

group2.sls

#!yamlex
sensu:
 client:
   option1: value1
   foo: bar
    subscriptions: !aggregate
    - mysubscription-group2

You maybe also need to set the pillar source merging strategy to aggregate http://docs.saltstack.com/en/latest/ref/configuration/master.html#pillar-source-merging-strategy

31. März 2015 19:22

​

Alfredo Palhares

unread,
Apr 6, 2015, 8:56:44 AM4/6/15
to salt-users
Hello Daniel,

> I haven’t tried that myself but you should be able to use the
> |!aggregate| tag of the yamlex renderer
> http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.yamlex.html
>
> something along those lines:
>
> |group1.sls|:
>
> |#!yamlex
> sensu:
> client:
> subscriptions: !aggregate
> - mysubscription-group
> |
>
> |group2.sls|
>
> |#!yamlex
> sensu:
> client:
> option1: value1
> foo: bar
> subscriptions: !aggregate
> - mysubscription-group2
> |
>
> You maybe also need to set the pillar source merging strategy to
> aggregate
> http://docs.saltstack.com/en/latest/ref/configuration/master.html#pillar-source-merging-strategy

This is what I was looking for but is giving me a little greaf[1] not all the
values are aggegating and its messing with formula behauviour.

[1] https://github.com/saltstack/salt/issues/22241
signature.asc

Colton Myers

unread,
Apr 6, 2015, 6:02:17 PM4/6/15
to salt-...@googlegroups.com
Thanks for filing that issue, Alfredo, we'll follow up there.

--
Colton Myers
Platform Engineer, SaltStack
@basepi on Twitter/Github/IRC

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alfredo Palhares

unread,
Apr 7, 2015, 7:25:23 AM4/7/15
to salt-users
Hello Colton,

> Thanks for filing that issue, Alfredo, we'll follow up there.
Sure,

But with this I have a currently broken salt. I can reverse the formula. Can I
configure salt-master gitfs to checkout to a tag or temporary branch so I have a
working salt-setup ?
signature.asc

Alfredo Palhares

unread,
Apr 8, 2015, 10:59:58 AM4/8/15
to salt-users
Hello,

> But with this I have a currently broken salt. I can reverse the formula. Can I
> configure salt-master gitfs to checkout to a tag or temporary branch so I have a
> working salt-setup ?

Any ideas?

Regards,
signature.asc

Florian Ermisch

unread,
Apr 8, 2015, 11:07:06 AM4/8/15
to salt-...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Just clone the git-repo into the master's filesystem and use file_roots?

Regards,Florian
>--
>You received this message because you are subscribed to the Google
>Groups "Salt-users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to salt-users+...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFTBAEBCAA9BQJVJUQRNhxGbG9yaWFuIEVybWlzY2ggPGZsb3JpYW4uZXJtaXNj
aEBhbHVtbmkudHUtYmVybGluLmRlPgAKCRAu8tzCHoBI/eYDB/0bAwhK0KIdoUhm
mog1b9Qt2meFO4upNu1OatSxf1bIdd3V+NJBbhhwEm0VAmdtHJyKF7IRhN5CDwiu
QnvlPj/Cap9HXLN+7nhVkzceEk9JRlCYK9acmAXePENQ5cKtghnV9YllczxO3yUJ
R+Wfvq8+IVApIrGe6CDO5HZk0M79XQMsdMcTfkeyjB4QPxrL0WvwcuilgL8DSTy4
slCoYLYKWVAP9Bm3rbL9C0Y1rFpShMxzL/GR7fXI5tXfu1GGsLuYCqlGL1xTUvaS
5JLaSaIaZLMV5qMwmNT6ZAs/YBGkTTuTicqH+8ymBQyZVEocbnU0v9easlTAOg/V
RMxmR8HN
=PjJ4
-----END PGP SIGNATURE-----

Florian Ermisch

unread,
Apr 8, 2015, 3:15:36 PM4/8/15
to salt-...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi Alfredo,

sorry, took a few days. Here you go:

https://github.com/0xf10e/blog/blob/master/SaltStack_templating_pillar.r
st

Regards, Florian

Am 03.04.2015 um 01:04 schrieb Alfredo Palhares:
> Hello Florian,
>
>> I use templating in pillar when I have to do any form of data
>> manipulation (like applying a default set and just changing a few
>> values). You can include a template and pass it your groups so it
>> can include and merge the appropriate files containing your
>> data. Gets a little more involved though.
> Can you give me an example for this use or point me to
> documentation ?
>
> I would be very thankful.
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCgAGBQJVJX44AAoJEC7y3MIegEj9kuMH/RoTRN7+AHlZlGLthmME8uBp
Q3KBKId2v5Z1NI7T5UpQ3ZFLFcjTJpwhM872TqDbkDPIXF3lxiaZedBnjw74dFdB
A9z8MdqnSdjo0Njol/ZrHlEAOyTAzoMHPf8T29mFMntbbaS1jVuKfkPRzMGhjpRF
jeEtkPCVTX9TfDk7Ul39wINPsNRaraSzCMPmPzLHqQuRPhRODaZWQo2jXvlYDW1s
CaIvZ7YjpKSlvS2DfVIEoOwTbg7XAnZrJI6EL8zowlVcjnfSuq2IMMKixK2WQNIT
qKzc/jZ39JnvzqT0d9yDe195OJLBs8nYQlZXY3z2P1LDJ0nVZK3Mrf9mleFjM9E=
=s0Lb
-----END PGP SIGNATURE-----
Reply all
Reply to author
Forward
0 new messages