Custom Module Output parsing

[Adam Shantz]

Hi all - 

I'm working on a custom module in python that takes a subset of an XML config, and turns it into JSON so I can use it with an Ansible playbook & Jinja2 template.  I want to make sure my output will be compatible with Ansible playbook processing.  Here's the output:

[

{"q1": {"flowcontrol-start-queuesize": "-1", "propertyCount": 5, "cleanup-interval": "-1", "cache-size": "5000", "persistence-mode": "non_persistent"}}, 

{"q2": {"flowcontrol-start-queuesize": "-1", "propertyCount": 5, "cleanup-interval": "-1", "cache-size": "5000", "persistence-mode": "non_persistent"}}, 

{"test": {"propertyCount": 1}}

]

I've added line-breaks to make it more readable.  Can this output be used in an Ansible playbook?  Possibly with the with_items loop?

Thanks,

Adam

[Adam Shantz]

My python code works fine to parse the output, but my module continually fails for reasons I can't explain.  Can someone give me a hint?

Module code:

#!/usr/bin/python

ANSIBLE_METADATA = {

    'metadata_version': '1.0',

    'status': ['preview'],

    'supported_by': 'curated'

}

import os

import shutil

import tempfile

import traceback

import xml.etree.cElementTree as etree

import json

from ansible.module_utils.basic import AnsibleModule

from ansible.module_utils._text import to_bytes, to_native

from collections import defaultdict

class Xml2Dict(dict):

    def __init__(self, parent_element):

        if parent_element.items():

            self.updateDict( dict(parent_element.items()) )

        for element in parent_element:

            if len(element):

                aDict = Xml2Dict(element)

                self.updateDict({element.tag: aDict})

            elif element.items():    # items() is special for attributes

                elementattrib= element.items()

                if element.text:

                    elementattrib.append((element.tag,element.text ))     # add tag:text if exist

                self.updateDict({element.tag: dict(elementattrib)})

            else:

                self.updateDict({element.tag: element.text})

    def updateDict (self, aDict ):

        for key in aDict.keys():   # keys() includes tag and attributes

            if key in self:

                value = self.pop(key)

                if type(value) is not list:

                    listOfDicts = []

                    listOfDicts.append(value)

                    listOfDicts.append(aDict[key])

                    self.update({key: listOfDicts})

                else:

                    value.append(aDict[key])

                    self.update({key: value})

            else:

                self.update({key:aDict[key]})  # it was self.update(aDict)

def run_module():

    # define the available arguments/parameters that a user can pass to

    # the module

    module_args = dict(

        src=dict(type='path')

    )

    # seed the result dict in the object

    # we primarily care about changed and state

    # change is if this module effectively modified the target

    # state will include any data that you want your module to pass back

    # for consumption, for example, in a subsequent task

    result = dict(

        changed=False,

        original_message='',

        message='',

        failed=''

    )

    # the AnsibleModule object will be our abstraction working with Ansible

    # this includes instantiation, a couple of common attr would be the

    # args/params passed to the execution, as well as if the module

    # supports check mode

    module = AnsibleModule(

        argument_spec=module_args,

        supports_check_mode=True

    )

    src = module.params['src']

#    b_src = to_bytes(src, errors='surrogate_or_strict')

#    if not os.path.exists(b_src):

#        module.fail_json(msg="Source %s not found" % (src))

#    if not os.access(b_src, os.R_OK):

#        module.fail_json(msg="Source %s not readable" % (src))

#    if os.path.isdir(b_src):

#        module.fail_json(msg="Directory specified as the source instead of a file: %s" % (src))

#    if os.path.exists(b_src):

#        b_src = os.path.realpath(b_src)

#        src = to_native(b_src, errors='surrogate_or_strict')

#        if os.access(b_src, os.R_OK):

    tree = etree.ElementTree(file=src)

    #print tree.getroot()

    root = tree.getroot()

    #print "tag=%s, attrib=%s" % (root.tag, root.attrib)

    from pprint import pprint

    #router = pprint(etree_to_dict(root))

    router = Xml2Dict(root)

    for key in router['swiftlet'][4].iteritems():

        if key[0] == 'aliases':

            for item in enumerate(key[1]['alias']):

                dest = str(item[1]['map-to'])

                name = str(item[1]['name'])

    #            print dest + ":" + name

    queuesList = []

    for key in router['swiftlet'][9].iteritems():

        if key[0] == 'queues':

            for item in enumerate(key[1]['queue']):

                name = str(item[1]['name'])

                attrList = item[1]

                print attrList

                queueDict = {}

                if len(attrList) == 1:

                    queueDict[name] = {"propertyCount" : len(attrList)}

                else:

                    for attrib, value in attrList.iteritems():

                        if attrib != "name":

                            if name in queueDict:

                                queueDict[name][attrib] = value

                            else:

                                queueDict[name] = {attrib : value}

                    if name in queueDict:

                        queueDict[name]["propertyCount"] = len(attrList)

                    else:

                        queueDict[name] = {"propertyCount" : len(attrList)}

                queuesList.append(queueDict)

    output = json.dumps(queuesList)

    # if the user is working with this module in only check mode we do not

    # want to make any changes to the environment, just return the current

    # state with no modifications

#    if module.check_mode:

#        return result

    # manipulate or modify the state as needed (this is going to be the

    # part where your module will do what it needs to do)

    result['original_message'] = module.params['src']

    result['message'] = 'goodbye'

    # use whatever logic you need to determine whether or not this module

    # made any modifications to your target

#    if module.params['src']:

#        result['changed'] = True

    # during the execution of the module, if there is an exception or a

    # conditional state that effectively causes a failure, run

    # AnsibleModule.fail_json() to pass in the message and the result

#    if module.params['src'] == 'fail me':

#        module.fail_json(msg='You requested this to fail', **result)

    # in the event of a successful module execution, you will want to

    # simple AnsibleModule.exit_json(), passing the key/value results

    module.exit_json(

        changed=True,

        failed=False)

def main():

    run_module()

if __name__ == '__main__':

    main()

Results:

TASK [test module] *******************************************************************************************************************************************

task path: /home/ec2-user/ansible/testmod.yml:7

Using module_utils file /home/ec2-user/ansible/lib/ansible/module_utils/_text.py

Using module_utils file /home/ec2-user/ansible/lib/ansible/module_utils/basic.py

Using module_utils file /home/ec2-user/ansible/lib/ansible/module_utils/six/__init__.py

Using module_utils file /home/ec2-user/ansible/lib/ansible/module_utils/parsing/convert_bool.py

Using module_utils file /home/ec2-user/ansible/lib/ansible/module_utils/parsing/__init__.py

Using module_utils file /home/ec2-user/ansible/lib/ansible/module_utils/pycompat24.py

Using module file /home/ec2-user/ansible/lib/ansible/modules/messaging/new_module.py

<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: ec2-user

<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'

<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ec2-user/.ansible/tmp/ansible-tmp-1502132102.2-141765875120224 `" && echo ansible-tmp-1502132102.2-141765875120224="` echo /home/ec2-user/.ansible/tmp/ansible-tmp-1502132102.2-141765875120224 `" ) && sleep 0'

<127.0.0.1> PUT /tmp/tmpHHjMUg TO /home/ec2-user/.ansible/tmp/ansible-tmp-1502132102.2-141765875120224/new_module.py

<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/ec2-user/.ansible/tmp/ansible-tmp-1502132102.2-141765875120224/ /home/ec2-user/.ansible/tmp/ansible-tmp-1502132102.2-141765875120224/new_module.py && sleep 0'

<127.0.0.1> EXEC /bin/sh -c '/home/ec2-user/ansible/venv/bin/python /home/ec2-user/.ansible/tmp/ansible-tmp-1502132102.2-141765875120224/new_module.py; rm -rf "/home/ec2-user/.ansible/tmp/ansible-tmp-1502132102.2-141765875120224/" > /dev/null 2>&1 && sleep 0'

fatal: [localhost]: FAILED! => {

    "changed": false, 

    "failed": true, 

    "module_stderr": "", 

    "module_stdout": "{'cache-size': '5000', 'flowcontrol-start-queuesize': '-1', 'name': 'q1', 'cleanup-interval': '-1', 'persistence-mode': 'non_persistent'}\n{'cache-size': '5000', 'flowcontrol-start-queuesize': '-1', 'name': 'q2', 'cleanup-interval': '-1', 'persistence-mode': 'non_persistent'}\n{'cache-size': '5000', 'flowcontrol-start-queuesize': '-1', 'name': 'q3', 'cleanup-interval': '-1', 'persistence-mode': 'non_persistent'}\n{'name': 'test'}\n\n{\"invocation\": {\"module_args\": {\"src\": \"/home/ec2-user/ansible/routerconfig.xml\"}}, \"failed\": false, \"changed\": true}\n", 

    "msg": "MODULE FAILURE", 

    "rc": 0

}

        to retry, use: --limit @/home/ec2-user/ansible/testmod.retry

PLAY RECAP ***************************************************************************************************************************************************

localhost                  : ok=1    changed=0    unreachable=0    failed=1   

[Adam Shantz]

My playbook is below:

- name: test

  gather_facts: yes

  hosts: localhost

  connection: local

  tasks:

    - name: test module

      new_module:

        src: /home/ec2-user/ansible/routerconfig.xml

      register: testout

    - name: debug

      debug:

        msg: '{{testout}}'

[Kai Stian Olstad]

On 07. aug. 2017 19:14, Adam Shantz wrote:
> Hi all -
>
> I'm working on a custom module in python that takes a subset of an XML
> config, and turns it into JSON so I can use it with an Ansible playbook &
> Jinja2 template. I want to make sure my output will be compatible with
> Ansible playbook processing. Here's the output:

For development question you might have better luck in the development
mailing list https://groups.google.com/forum/#!forum/ansible-devel


--
Kai Stian Olstad

[Adam Shantz]

thanks