For Loop to Iterate Through Files

[Jameson]

I'm trying to use a Jinja for loop to go through the files in a
particular directory, and I can't seem to get it right. Is there a way
to do something along the lines of: {% for file in "salt://sshkeys/*"
%}

I've tried several different things like salt['file.find
salt://sshkeys/'], and even salt['file.find /srv/salt/sshkeys'], but
I'm obviously confused somewhere.

Thanks for any help you can provide.
=-Jameson

[Jameson]

On Tue, Jan 6, 2015 at 3:16 PM, Jameson <imnt...@gmail.com> wrote:
> I'm trying to use a Jinja for loop to go through the files in a
> particular directory, and I can't seem to get it right. Is there a way
> to do something along the lines of: {% for file in "salt://sshkeys/*"
> %}

My ultimate goal is to come up with a state that looks like:

sshkeys:
{% for file in salt['file.find salt://sshkeys/'] %}
ssh_auth:
- present
- user: {{ file }}
- source: {{ file }}
- onlyif:
- user.present:
- name: {{ file }}
{% endfor %}

[Matthew Williams]

I don't think file.find works directly with salt:// urls, but to invoke a function from the salt dict you need to do this: salt['file.find'](...). With ... replaced by the args and/or kwargs.

> --
> You received this message because you are subscribed to the Google Groups "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ethan Erchinger]

Try using cp.list_master.

[Jameson]

This is definitely the right track. I'm still having an issue, though.
When I execute cp.list_master 'prefix=sshkeys/' from the command line,
I do get a list of files in that directory. Unfortunately, when I try
to call it from my state file, it's actually seeing the state as
invalid. This is where I'm at, so far, but it doesn't seem to like me
using cp.list_master in a for loop. It works fine if I manually set
the file variable to what it would return:

sshkeys:
{% for file in salt.cp.list_master('prefix=sshkeys/') %}
{% set user = file.split('/') %}
ssh_auth:
- present
- user: {{ user[1] }}
- source: salt://{{ file }}
- onlyif:
- user.present:
- name: {{ user[1] }}
{% endfor %}

On Wed, Jan 7, 2015 at 2:14 AM, Ethan Erchinger <eth...@erchinger.org> wrote:
> Try using cp.list_master.

[Ethan Erchinger]

The quotes around prefix=sshkeys are unnecessary at the shell, and misused in the state.  You'll want something like:

{% for file in salt['cp.list_master'](prefix='sshkeys') %}
  # {{ file }}
{% endfor %}

[Jameson]

Thanks. It's been challenging for me to wrap my head around salt
methodology and the syntax all at once, but this looks like it will do
what I want:

{% for file in salt['cp.list_master'](prefix='sshkeys') %}

{% set user = file.split('/')[1] %}
{{ user }}:

ssh_auth:
- present
- user: {{ user }}

- source: salt://{{ file }}
- onlyif:
- user.present:
- name: {{ user }}

{% endfor %}

[Jameson]

Well, I say that, but it looks like the onlyif isn't working
correctly. On my CentOS boxes, it is always returning false whether or
not the user exists, and on SLES it's always returning true.

[Ethan Erchinger]

Onlyif isn't a property that works in all states.

Two options:

  - If you are creating the users via salt also, you could use a "require: requisite to require the user to be created first.

  - Wrap the ssh_auth logic for each key with a check to ensure the user exists first, as such:

{% set users = salt['user.list_users']() %}

{% for file in salt['cp.list_master'](prefix='sshkeys') %}
{% set user = file.split('/')[1] %}

{% if user in users %}

{{ user }}:

  ssh_auth:
    - present
    - user: {{ user }}

    - source: salt://{{ file }}
    - onlyif:
      - user.present:
        - name: {{ user }}

{% endif %}{% endfor %}

[Ethan Erchinger]

Corrected version below, the onlyif portion shouldn't have been left in.

On Thursday, January 8, 2015 10:45:53 AM UTC-8, Ethan Erchinger wrote:

Onlyif isn't a property that works in all states.

Two options:

  - If you are creating the users via salt also, you could use a "require: requisite to require the user to be created first.

  - Wrap the ssh_auth logic for each key with a check to ensure the user exists first, as such:

{% set users = salt['user.list_users']() %}
{% for file in salt['cp.list_master'](prefix='sshkeys') %}
{% set user = file.split('/')[1] %}
{% if user in users %}
{{ user }}:

  ssh_auth:
    - present
    - user: {{ user }}
    - source: salt://{{ file }}

[Jameson]

I specifically wanted to use onlyif because the majority of the
accounts I plan on using this for are actually created at login by
winbind. I suppose if worse comes to worse, I can just live with the
failures where the accounts don't exist.

[Ethan Erchinger]

Not sure how that'll work considering ssh_auth.present will likely want/need to set permissions on that file to that of the user.

If that works properly, maybe you could look to see if the homedir exists or something instead of using user.list().

[Jameson]

I feel like I'm going around the world on this. Thanks for all your
help, btw. The issue I'm running into with home directories is
file.directory_exists and file.exists seem to be checking from /root
instead of the absolute path I'm giving it:

{% for file in salt['cp.list_master'](prefix='sshkeys') %}
{% set user = file.split('/')[1] %}

{{ user }}:

ssh_auth:
- present
- user: {{ user }}
- source: salt://{{ file }}

- onlyif:
{% set home = salt['user.info'](user)['home'] %}
- file.directory_exists: {{ home }}

{% endfor %}

I'm getting stuff like:
[INFO ] Running state [test] at time 15:38:02.595937
[INFO ] Executing state ssh_auth.present for test
[INFO ] Executing command OrderedDict([('file.directory_exists',
'/home/NGHS/test')]) in directory '/root'
[INFO ] onlyif execution failed
[INFO ] Completed state [test] at time 15:38:02.607251
[INFO ] Running state [user] at time 15:38:02.607968
[INFO ] Executing state ssh_auth.present for user
[INFO ] Executing command OrderedDict([('file.directory_exists',
'/home/user')]) in directory '/root'
[INFO ] onlyif execution failed
[INFO ] Completed state [user] at time 15:38:02.620963

Where user and test are obviously possible accounts.

[Ryan Lane]

I may be wrong, but I think onlyif will only run shell commands. It can't reference states.

[Jameson]

This is getting me much closer. I may try to switch the entire
conditional to jinja, though, for better compatibility.

{% for file in salt['cp.list_master'](prefix='sshkeys') %}
{% set user = file.split('/')[1] %}

{% set home = salt['user.info'](user)['home'] %}
{{ user }}:

ssh_auth:
- present
- user: {{ user }}
- source: salt://{{ file }}

- onlyif:

- if [[ -e {{ home }} ]]; then exit 0; else exit 1; fi

{% endfor %}

[Ryan Lane]

This is shorter/cleaner: test -e {{ home }}

[Jameson]

Thanks. I couldn't for the life of me remember the test command. This
is working for me everywhere I've tested it, except CentOS 6. I think
there may be an issue with python-2.6.6.

{% for file in salt['cp.list_master'](prefix='sshkeys') %}
{% set user = file.split('/')[1] %}
{% set home = salt['user.info'](user)["home"] %}

{{ user }}:
ssh_auth:
- present
- user: {{ user }}
- source: salt://{{ file }}

- onlyif:

- test -e {{ home }}

{% endfor %}

[Jameson]

It looks like the issue on the CentOS boxes is when the jinja is
rendered it doesn't seem to like user in {% set home =
salt['user.info'](user)["home"] %}. If I manually specify something
for user there it works. Is there some way I should be escaping it to
properly handle the variable substitution there? Thanks.

[Ethan Erchinger]

The syntax looks valid.  If you run that from the command-line is a value returned for "home"?

salt-call user.info <user>

[Jameson]

Yep. 'home' is there. In my testing I tried replacing user in that
line with both a valid user, and an invalid user, and it worked as
expected in both cases. For some reason it just doesn't like parsing
that correctly. I even tried manually building a newer version of
python-jinja2 on one of my CentOS 6 boxes, but that didn't help,
either.

[Jameson]

On Mon, Jan 12, 2015 at 9:28 AM, Jameson <imnt...@gmail.com> wrote:
> Yep. 'home' is there. In my testing I tried replacing user in that
> line with both a valid user, and an invalid user, and it worked as
> expected in both cases. For some reason it just doesn't like parsing
> that correctly. I even tried manually building a newer version of
> python-jinja2 on one of my CentOS 6 boxes, but that didn't help,
> either.

So, this is what I wound up with in case someone else runs into this:

{% for file in salt['cp.list_master'](prefix='sshkeys') %}
{% set user = file.split('/')[1] %}

{% if grains['os_family'] == 'RedHat' and grains['osmajorrelease'] == '6' %}
{% set home = salt['cmd.script']("salt://scripts/home.sh",
args=user).stdout %}
{% else %}

{% set home = salt['user.info'](user)['home'] %}

{% endif %}

{{ user }}:
ssh_auth:
- present
- user: {{ user }}
- source: salt://{{ file }}

- onlyif:
- test -e {{ home }}

{% endfor %}

home.sh is just:
#!/bin/bash
if [[ -e $(eval echo ~"$1") ]]; then
eval echo ~"$1"
exit 0
else
echo "No home"
exit 1
fi

I do still have some weirdness going on, but hopefully I can work around it.