Complex Loops and Conditionals (a.k.a Dude, where's my include + with_items + when)

[Christopher O'Connell]

Hello List,

I understand that a decision has been made to remove include + with_items, and I am not trying to directly revisit that decision. What I am trying to figure out is how to effectively replace such constructs in our roles. If someone reads this and concludes that I'm approaching my problem entirely wrong, then please pipe up and let me know I'm an idiot.

Consider a custom module called "listdisks" which (as the name suggests), lists information about the physical disks installed in a system (whether they're formatted and mounted, or not). Example output:

{
    "changed": false,
    "drives": [
        {
            "hasPartitions": true,
            "name": "sda",
            "partitions": [
                {
                    "filesystem": "ext3",
                    "mount": "/boot",
                    "name": "1",
                    "size": "268",
                    "sizeUnits": "MB"
                },
                {
                    "filesystem": "linux-swap",
                    "mount": null,
                    "name": "2",
                    "size": "8590",
                    "sizeUnits": "MB"
                },
                {
                    "filesystem": "ext4",
                    "mount": "/",
                    "name": "3",
                    "size": "111",
                    "sizeUnits": "GB"
                }
            ],
            "size": "120",
            "sizeUnits": "GB"
        },
        {
            "hasPartitions": false,
            "name": "sdb",
            "partitions": [],
            "size": "2000",
            "sizeUnits": "GB"
        }
    ],

"item":""

}

As a simple example of the problem, consider a playbook to simply print a debug message for each existing partition which does not have an identified mount point.

This playbook snippet works without using include or with_items:

- name: get outstanding drives
  listdisks: unpartitioned=true
  register: system_disks
- name: identify unmounted partitions
  debug: msg="/dev/{{ item[0].name }}{{ item[1].name }} ({{ item[1].size }}{{ item[1].sizeUnits }}) not mounted"
  when: item[0].hasPartitions == True and item[1].mount == None
  with_subelements:
    - system_disks.drives
    - partitions

Sample output (with some specific domain name information elided):

PLAY [configure ceph systems] *************************************************

GATHERING FACTS ***************************************************************
ok: [system1232]
ok: [system1233]
ok: [system1234]

TASK: [ceph | get outstanding drives] *****************************************
ok: [system1233]
ok: [system1232]
ok: [system1234]

TASK: [ceph | identify unmounted partitions] **********************************
skipping: [system1233] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'MB', u'mount': u'/boot', u'size': u'268', u'name': u'1', u'filesystem': u'ext3'}))
skipping: [system1232] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'MB', u'mount': u'/boot', u'size': u'268', u'name': u'1', u'filesystem': u'ext3'}))
skipping: [system1234] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'MB', u'mount': u'/boot', u'size': u'268', u'name': u'1', u'filesystem': u'ext3'}))
ok: [system1233] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'MB', u'mount': None, u'size': u'8590', u'name': u'2', u'filesystem': u'linux-swap'})) => {
    "item": [
        {
            "hasPartitions": true,
            "name": "sda",
            "size": "120",
            "sizeUnits": "GB"
        },
        {
            "filesystem": "linux-swap",
            "mount": null,
            "name": "2",
            "size": "8590",
            "sizeUnits": "MB"
        }
    ],
    "msg": "/dev/sda2 (8590MB) not mounted"
}
ok: [system1232] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'MB', u'mount': None, u'size': u'8590', u'name': u'2', u'filesystem': u'linux-swap'})) => {
    "item": [
        {
            "hasPartitions": true,
            "name": "sda",
            "size": "120",
            "sizeUnits": "GB"
        },
        {
            "filesystem": "linux-swap",
            "mount": null,
            "name": "2",
            "size": "8590",
            "sizeUnits": "MB"
        }
    ],
    "msg": "/dev/sda2 (8590MB) not mounted"
}
skipping: [system1233] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'GB', u'mount': u'/', u'size': u'111', u'name': u'3', u'filesystem': u'ext4'}))
ok: [system1234] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'MB', u'mount': None, u'size': u'8590', u'name': u'2', u'filesystem': u'linux-swap'})) => {
    "item": [
        {
            "hasPartitions": true,
            "name": "sda",
            "size": "120",
            "sizeUnits": "GB"
        },
        {
            "filesystem": "linux-swap",
            "mount": null,
            "name": "2",
            "size": "8590",
            "sizeUnits": "MB"
        }
    ],
    "msg": "/dev/sda2 (8590MB) not mounted"
}
skipping: [system1232] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'GB', u'mount': u'/', u'size': u'111', u'name': u'3', u'filesystem': u'ext4'}))
skipping: [system1234] => (item=({u'hasPartitions': True, u'sizeUnits': u'GB', u'size': u'120', u'name': u'sda'}, {u'sizeUnits': u'GB', u'mount': u'/', u'size': u'111', u'name': u'3', u'filesystem': u'ext4'}))

PLAY RECAP ********************************************************************
system1232        : ok=3    changed=0    unreachable=0    failed=0  
system1233        : ok=3    changed=0    unreachable=0    failed=0  
system1234        : ok=3    changed=0    unreachable=0    failed=0

This playbook works perfectly. The problem is that it is virtually unreadable. Take a look at just the command to debug information about the unmounted drives:

  debug: msg="/dev/{{ item[0].name }}{{ item[1].name }} ({{ item[1].size }}{{ item[1].sizeUnits }}) not mounted"
  when: item[0].hasPartitions == True and item[1].mount == None
  with_subelements:
    - system_disks.drives
    - partitions

Simply looking at that command makes very little sense of what it actually does. Furthermore, the when and with_subelements clauses must be repeated for every single item in a long role. In the role I've drawn this example from, I need to partition, format and mount drives newly connected (or reconnected after a drive replacement) to the system and then inform the ceph agent about these devices. At the moment I have about 14 tasks with the ugly when + with_subelements clauses. This wildly violates the principle of "Don't Repeat Yourself", and not just in an abstract way -- the impetus to write this email came after spending nearly 4 hours tracking down a typo in a single one of these repeated clauses. Surely there must be a better way, either already baked into Ansible, or if not, then added to the road map.

On a related note, since updating to the 1.5 working branch, I've noticed a regression using include + when where some of the included tasks also have a when clause. Based on reading this closed bug [https://github.com/ansible/ansible/issues/3269] it appears that include + when and an inner when clause should work, but I have not found this to be the case. I haven't spent considerable time trying to debug this since I'm moving away from using include at all and towards really long roles, but if this behavior is being intentionally removed, some documentation seems like a good idea.

All the best,

~ Christopher

P.S. If the listdisks module seems like something of interest, I'll consider adding it to the ansible-galaxy thingy.

[Kiran Kumar]

Hi Christopher,

Am trying to work on something where the listdisks module can perfectly fit, do you mind sharing the code for listdisks module?