Etcd external pillar and keys with multiple values

[Robert Callicotte]

Hello,

I am experimenting with putting pillar data into etcd. Keys with individual values work great!  But I'm seeing problems with keys and multiple values in etcd. I have an existing pillar file that looks like the following:

osprep-installed6-atlassian:

  - firefox

  - git

  - tmux

  - htop

etc etc

My etcd info was ingested to the key via the urlencode function in etcd from a text file:

text file contents:

- firefox

- git

- tmux

- htop

etcd data ingestion:

curl -L http://127.0.0.1:2379/v2/keys/packages/osprep-installed6-atlassian -XPUT --data-urlencode va...@text.file

salt pillar items:

From etcd:

root@stewie osprep]# salt \* pillar.item osprep-installed6-atlassian

brian.example.com:

    ----------

    osprep-installed6-atlassian:

        - firefox

        - git

        - tmux

        - htop

From pillar file:

root@stewie osprep]# salt \* pillar.item osprep-installed6-atlassian

brian.example.com:

    ----------

    osprep-installed6-atlassian:

        - firefox

        - git

        - tmux

        - htop

When I run a salt state against this data I see the following from minion debug:

[INFO    ] Running state [osprepinstalled] at time 13:18:34.957863

[INFO    ] Executing state pkg.latest for osprepinstalled

[ERROR   ] Invalid input: [[None],

 None,

 'f',

 'i',

 'r',

 'e',

 'f',

 'o',

 'x',

 None,

 [None],

 None,

 'g',

 'i',

 't',

 None,

 [None],

 None,

 't',

 'm',

 'u',

 'x',

 None,

 [None],

 None,

 'h',

 't',

 'o',

 'p',

 None]

[ERROR   ] Input must be a list of strings/dicts

[ERROR   ] Invalidly formatted "pkgs" parameter. See minion log.

The state run fails with "Invalidly formatted "pkgs" parameter. See minion log." message.    Does anyone know how I can get this data formatted properly?  

Thanks!

-Robby

[Colton Myers]

Does it fail from both the pillar from a pillar file, and the pillar from etcd? If the failure is limited to etcd, I would recommend running pillar.item with --out=json so you can see the exact structure of the data and can figure out what's different between the two data sets. Sometimes the nested outputter (which is what outputs by default) can mask differences in data.

If both sets of pillar fail, then the problem is likely in the way you're ingesting the data in your state files, which you have not included, so I can't debug.

--

Colton Myers

Core Engineer, SaltStack

@basepi on Twitter/Github/IRC

Registration for SaltConf 2016 is open! http://saltconf.com/register/

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Callicotte]

Colton,

Thank you for your reply.  

This is the output of pillar data from etcd:

[root@salty ~]# salt \* pillar.item packages --out=json

{

    "kvm0.site.org": {

        "packages": "- tmux\n- screen\n- git\n"

    }

}

This is the output of pillar data from file:

[root@salty master.d]# salt \* pillar.item packages --out=json

{

    "dev1.site.org": {

        "packages": [

            "vim", 

            "tmux", 

            "git"

        ]

    }

}

The pillar data from a file works in my state files.  So it appears that I need to find a way to ingest the file into etcd so that it resembles the json output from the the second example.  

-Robby

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/bHEuIaiiRTM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

[Steve Hajducko]

The problem with trying to store multiple values within a single key in etcd is that etcd stores that as a string.  It literally is a 'key=value' store, so each key needs a corresponding value.  It looks like you're wanting a key that's pointing at a list of values and etcd doesn't support that.

The only way to do it would be some special formatting rule that would designate the difference between list elements, but then you'd run into issues where it may screw up someone else's etcd store that has that delimiter.  For instance, the consul ( which is also a k/v store ) pillar assumes that '\n' is the delimiter.  I suppose it could be a config option so people could tune it to their own needs.

For now though, I think the only way would be to store each individual value as it's own key with a bogus ( 'True' ) or none value, then use some jinja in your state file to get the pillar items and just use the 'keys' as your list.

[Florian Ermisch]

If you want to use etcd for this no matter what you can try to store a string of YAML or JSON under this specific key* in etcd. Then you can have the templating interpret this and load your list into a variable:

```
{% load_yaml as pkgs %}
{% pillar['packages'] %}
{% endload %}
list_of_pkgsrc_from_etcd:
pkg.installed:
- names: {{ pkgs }}
```

Not sure if jinja would evaluate this in the right order to work but may be worth a try.

To enforce the order of evaluation you could move the `load_yaml` into a different file you tell jinja to load the variable from but now the hacks are really getting ugly…

Regards, Florian

> >> <javascript:>> wrote:
> >>
> >>> Hello,
> >>>
> >>> I am experimenting with putting pillar data into etcd. Keys with
> >>> individual values work great! But I'm seeing problems with keys
> and
> >>> multiple values in etcd. I have an existing pillar file that looks
> like the
> >>> following:
> >>>
> >>>
> >>> osprep-installed6-atlassian:
> >>> - firefox
> >>> - git
> >>> - tmux
> >>> - htop
> >>>
> >>>
> >>> etc etc
> >>>
> >>>
> >>>
> >>> My etcd info was ingested to the key via the urlencode function in
> etcd
> >>> from a text file:
> >>>

> >>> *text file contents*:

> >>>
> >>> - firefox
> >>> - git
> >>> - tmux
> >>> - htop
> >>>
> >>>

> >>> *etcd data ingestion*:

> >>> curl -L
> >>> http://127.0.0.1:2379/v2/keys/packages/osprep-installed6-atlassian
>
> >>> -XPUT --data-urlencode va...@text.file
> >>>
> >>>

> >>> *salt pillar items*:
> >>>
> >>> *From etcd*:

> >>> root@stewie osprep]# salt \* pillar.item
> osprep-installed6-atlassian
> >>> brian.example.com:
> >>> ----------
> >>> osprep-installed6-atlassian:
> >>> - firefox
> >>> - git
> >>> - tmux
> >>> - htop
> >>>
> >>>

> >>> *From pillar file*:

[Robert Callicotte]

Thanks everyone for the assistance!  I ended up going with a hybrid approach.  For package list related items in pillar I am keeping things in plain old pillar root files.  For the key/value stuff I'm keeping that in etcd.  The end game of all of this is to isolate the use of certain records (i.e. application owner/contact info etc) within an external pillar so that different teams can update records without the need to access the salt master directly.  Etcd fulfills this requirement for specific records.  Our inventory management system is homegrown and the easiest -- and most resilient solution,  is to use the etcd provided external pillar.   

-Robby

[Robert Callicotte]

Hello all,

I wanted to follow up on this topic as it has been a while.  I switched back to a hybid model of using etcd for some things and flat files for others... I finally got it all to work based on Florian's advice.  Here's what I did:

Changed the yaml file to the following format:

[firefox, git, tmux, htop]

My package state:

{% load_yaml as pkgs %}

  packs: {{ pillar['osprep-installed6-atlassian'] }}

{% endload %}

default_packages:

  pkg.installed:

    - names: {{ pkgs.packs }}

Now my package lists can live in etcd and be consumed by salt.  Thanks once again for all of your help.

-Robby