using csvfile lookups inside a module

[Asil Carlin]

Hi,

I'm trying to use csvfile lookups to populate values in the groups module, but not having much luck. My csvfile:

# groups.csv
# name, gid [optional - leave blank], state [present|absent], system [yes|no]
 accounts,502,present,no
engineering,504,present,no

The playbook:

---

- hosts: localhost
  become: True
  become_user: root

  tasks:

  - name: get groups
    command: /usr/bin/awk -F',' '!/^#/ && !/^$/ { print $1 }' groups.csv
    register: groups_out

  - name: Process groups
    group: >
      name="{{ lookup('csvfile', 'item file=groups.csv col=0') }}"
      gid="{{ lookup('csvfile', 'item file=groups.csv col=1') }}"
      state="{{ lookup('csvfile', 'item file=groups.csv col=2') }}"
      system="{{ lookup('csvfile', 'item file=groups.csv col=3') }}"
    # with_lines: "/usr/bin/awk -F',' '!/^#/ && !/^$/ { print $1 }' groups.csv"
    # with_items: "{{ groups_out.stdout_lines }}"
    with_lines: "{{ groups_out.stdout_lines }}"

 Using with_lines and groups_out.stdout_lines gives me:

TASK [Process groups] **********************************************************
/bin/sh: accounts: command not found
fatal: [localhost]: FAILED! => {"failed": true, "msg": "lookup_plugin.lines(accounts) returned 127"}

Using with_items or with_lines and the awk command directly gives me:

TASK [Process groups] **********************************************************
failed: [localhost] (item=accounts) => {"failed": true, "item": "accounts", "msg": "argument system is of type <type 'list'> and we were unable to convert to bool"}
failed: [localhost] (item=engineering) => {"failed": true, "item": "engineering", "msg": "argument system is of type <type 'list'> and we were unable to convert to bool"}

Removing the loop and just specifying a key, in the following output "accounts" was used:

TASK [Process groups] **********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "argument system is of type <type 'list'> and we were unable to convert to bool"}

Ultimately I'd like to use csv files to define config although I'm not sure i'm going about this the right way.

Asil

[J Hawkesworth]

Presumably you've got csv files coming from somewhere else that you want to use to drive things?

If not, I'd suggest to you see if you can use group_vars http://docs.ansible.com/ansible/intro_inventory.html#group-variables to set variables for specific groups.  Bear in mind you can have multiple groups and you can have groups which are specific to just one inventory file or are shared, so they can be pretty flexible.

I think using group_vars would 'go with the grain' better than using the lookups, and give you more readable playbooks.

Probably not what you wanted to hear, but hope it helps,

Jon

[Asil Carlin]

Thanks for replying.

At the moment no, there aren't any csv files anywhere. Just trying out different approaches to config management using ansible. I'm comfortable with group_vars, it's the repetition in defining them that grates.

If you have a spare few minutes, take a look at Neil Watson's blog, specifically this post explaining the evolution of his Cfengine policies. I've used this approach at a previous shop that had CFengine deployed and it worked well.

It makes viewing and managing configuration much easier, the idea of data-driven policies appeals.It also simplifies operations view of managing the infrastructure, one line specifies multiple configuration parameters for an object.

So...not what I wanted to hear but, not the end of the world either.

Just noticed I hadn't set the delimiter parameter, however still get the "...unable to convert to bool" message.

Asil

[J Hawkesworth]

Not read this yet but you can define cross group_vars that are common across inventory as well as specific to inventory - it's a trick I use to save having to copy and paste stuff which is shared across most, but not all inventory.

Thanks for the link though.

[Kai Stian Olstad]

On 03. okt. 2016 11:18, Asil Carlin wrote:
> Hi,
>
> I'm trying to use csvfile lookups to populate values in the groups module,
> but not having much luck. My csvfile:
>
> # groups.csv
> # name, gid [optional - leave blank], state [present|absent], system
> [yes|no]
> accounts,502,present,no
> engineering,504,present,no
>
> The playbook:
>
> ---
>
> - hosts: localhost
> become: True
> become_user: root
>
> tasks:
>
> - name: get groups
> command: /usr/bin/awk -F',' '!/^#/ && !/^$/ { print $1 }' groups.csv
> register: groups_out
>
> - name: Process groups
> group: >
> name="{{ lookup('csvfile', 'item file=groups.csv col=0') }}"
> gid="{{ lookup('csvfile', 'item file=groups.csv col=1') }}"
> state="{{ lookup('csvfile', 'item file=groups.csv col=2') }}"
> system="{{ lookup('csvfile', 'item file=groups.csv col=3') }}"

As you mention in a later post you are missing the delimiter, TAB is the
default.

And you key is literally "item" on all you lookup. To use the variable
item you'll have to concatenate like this.

lookup('csvfile', item + ' file=groups.csv delimiter=, col=n')

> # with_lines: "/usr/bin/awk -F',' '!/^#/ && !/^$/ { print $1 }'
> groups.csv"
> # with_items: "{{ groups_out.stdout_lines }}"
> with_lines: "{{ groups_out.stdout_lines }}"

You can't only have a variable in with_items, it must be a command
https://docs.ansible.com/ansible/playbooks_loops.html#iterating-over-the-results-of-a-program-execution

>
> Using with_lines and groups_out.stdout_lines gives me:
>
> TASK [Process groups]
> **********************************************************
> /bin/sh: accounts: command not found

You don't have /bin/sh, that can be a problem, since default, Ansible is
depending on it.

--
Kai Stian Olstad

[Asil Carlin]

Hi Kai,

Thanks for taking the time to reply. I did first try using with_lines and the awk command and using "item +", that throws up another error: This code

---

- hosts: localhost
  become: True
  become_user: root

  tasks:

  - name: get

groups
    command: /usr/bin/awk -F',' '!/^#/ && !/^$/ { print $1 }' /var/tmp/groups.csv
    register: groups_out

  - debug: var=groups_out.stdout_lines

  - name: Process groups one
    group: >
      name={{ lookup('csvfile', item + 'file=groups.csv col=0 delimiter=,') }}
      gid={{ lookup('csvfile', item + 'file=groups.csv col=1 delimiter=,') }}
      state={{ lookup('csvfile', item + 'file=groups.csv col=2 delimiter=,') }}
      system={{ lookup('csvfile', item + 'file=groups.csv col=3 delimiter=,') }}
    with_items: "{{ groups_out.stdout_lines }}"
    ignore_errors: True

  - name: Process groups two
    group: >
      name={{ lookup('csvfile', item + 'file=groups.csv col=0 delimiter=,') }}
      gid={{ lookup('csvfile', item + 'file=groups.csv col=1 delimiter=,') }}
      state={{ lookup('csvfile', item + 'file=groups.csv col=2 delimiter=,') }}
      system={{ lookup('csvfile', item + 'file=groups.csv col=3 delimiter=,') }}
    with_lines: /usr/bin/awk -F',' '!/^#/ && !/^$/ { print $1 }' /var/tmp/groups.csv

with this csv:

accounts,502,present,no
engineering,504,present,no

Both tasks complain about ansible.csv not existing in my CWD.

fatal: [localhost]: FAILED! => {"failed": true, "msg": "csvfile: [Errno 2] No such file or directory: u'/var/tmp/ansible.csv'"}

No. If I touch /var/tmp/ansible.csv then the we're back to the following:

failed: [localhost] (item=accounts) => {"failed": true, "item": "accounts", "msg": "argument system is of type <type 'list'> and we were unable to convert to bool"}
failed: [localhost] (item=engineering) => {"failed": true, "item": "engineering", "msg": "argument system is of type <type 'list'> and we were unable to convert to bool"}

To add, if I were to use the following syntax of "item":

lookup('csvfile', 'item file=groups.csv col=0 delimiter=,') }}

Then ansible doesn't look for a file called ansible.csv in my cwd and we get the same "...unable to convert to bool" message.

/bin/sh exists and is a link to /bin/bash.

Assuming I've understood all the advice given, should I file a bug report? 

Can anyone else reproduce this?

Thanks and regards,

Asil

[J Hawkesworth]

I wonder if you might be better off using include_vars to load all of your groups details in one pass, without multiple lookup calls.

http://docs.ansible.com/ansible/include_vars_module.html

There's a little bit more syntax to make your csv into yaml.  Instead of...

accounts,502,present,no
engineering,504,present,no

You'd write:

groups:

 - {name: accounts, gid: 502, state: present, system: no}

 - {name: engineering, gid: 504, state: present, system: no}

or 

--- 

groups: 

  - 

    gid: 502

    name: accounts

    state: present

    system: false

  - 

    gid: 504

    name: engineering

    state: present

    system: false

Then your playbook might look like:

     - name: load groups one

    include_vars: file=groups1.yml

  - name: Process groups one
    group: 

      name: {{ item.name }}
      gid: {{ item.gid }}
      state: {{ item.state }}
      system: {{ item.state }}
    with_items: "{{ groups }}"
    ignore_errors: True

If you still prefer your csv syntax, you might want to change to the key: value style instead of the key=value style as this preserves type information in yaml (so ints stay as ints, booleans as booleans etc).

Hope this helps,

Jon

[Kai Stian Olstad]

If you look carefully at my example I have a space in front of file.
Without it your name would be like this on first iteration

lookup('csvfile', 'accountfile=groups.csv col=0 delimiter=,')

And this is not a valid lookup.

--
Kai Stian Olstad

[Asil Carlin]

Kai - Thank you, I missed the space! This now works as I would have hoped.

Jon - Thank you too! Using a include_vars rather than a lookup achieves the same goal, the single-line format of each item was something I completely forgot you could do, for some reason I've adopted the multi-line  format and never let go.

Kind regards,

Asil