How to pylint and unit test module_utils code in my local module/role?

[Rich Megginson]

When developing a role that has a local module with some code split into
module_utils, how do you set up your pythonpath or venv so that pylint
and unit testing resolves the imports correctly?

For example: https://github.com/linux-system-roles/network/

The module code is in
https://github.com/linux-system-roles/network/blob/master/library/network_connections.py

The module_utils code is in
https://github.com/linux-system-roles/network/tree/master/module_utils/network_lsr

The module code imports the module_utils code like this:

# pylint: disable=import-error, no-name-in-module
from ansible.module_utils.network_lsr import MyError

As you can see, we have to disable pylint checking, because there is no
ansible.module_utils.network_lsr - there is a module_utils.network_lsr
however. We have to do something similar in the unit test code:
https://github.com/linux-system-roles/network/blob/master/tests/unit/test_nm_provider.py

sys.modules["ansible"] = mock.Mock()
sys.modules["ansible.module_utils.basic"] = mock.Mock()
sys.modules["ansible.module_utils"] = mock.Mock()
sys.modules["ansible.module_utils.network_lsr"] = __import__("network_lsr")

This seems pretty hackish, and I'm hoping there is a better or standard
way to do this.

[Matt Martz]

There isn't a great answer here.  roles weren't originally designed to work this way.

You may want to look into collections instead, which have proper python imports, that aren't effectively "fake".  A collection can include modules/plugins, as well as roles.  However the location of the module moves.

This doesn't really solve things for older versions of Ansible however, as collections are a recent addition.

--
You received this message because you are subscribed to the Google Groups "Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-deve...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-devel/d5f0c3ad-f7e5-09de-b5b4-3e9c741cd567%40redhat.com.

--

Matt Martz
@sivel
sivel.net

[Alan Rominger]

If you convert it to a collection you can use relative imports. I am in a very similar situation to you with the PYTHONPATH manipulation for testing.

https://github.com/ansible/awx/blob/a93b1aa3395651f01b39a03a6f122d3bdad99897/Makefile#L411

I put the root of the collection project (it's in a folder, not the top-level of source control) in the python path. Technically, this means that imports are screwed up, because module_utils will import as `import plugins.module_utils.tower` as opposed to the proper import of `import ansible_collections.awx.awx.plugins.module_utils.tower`, but the extra step to nest those folders has been a pain point. So to keep the development-test cycle fast, I knowingly use the wrong imports in tests:

https://github.com/ansible/awx/blob/a93b1aa3395651f01b39a03a6f122d3bdad99897/awx_collection/test/awx/conftest.py#L112

This doesn't affect the functionality of the modules that import from module_utils, because they use relative imports.

https://github.com/ansible/awx/blob/a93b1aa3395651f01b39a03a6f122d3bdad99897/awx_collection/plugins/modules/tower_credential.py#L221

Again, I think this is a trick you only get with collections.

----

If you keep the current folder structure, I wonder if instead of messing with sys.modules in the test code, you could just make the folders `ansible/module_utils/basic`, etc. in some far-off place, have those import from the path you know works, and put that in the python path too. Same as your current idea, it's very hacky but still isolated to your testing setup.

Alan

github: AlanCoding

[Rich Megginson]

On 3/6/20 10:22 AM, Alan Rominger wrote:
> If you convert it to a collection you can use relative imports. I am in
> a very similar situation to you with the PYTHONPATH manipulation for
> testing.
>
> https://github.com/ansible/awx/blob/a93b1aa3395651f01b39a03a6f122d3bdad99897/Makefile#L411
>
> I put the root of the collection project (it's in a folder, not the
> top-level of source control) in the python path. Technically, this means
> that imports are screwed up, because module_utils will import as `import
> plugins.module_utils.tower` as opposed to the proper import of `import
> ansible_collections.awx.awx.plugins.module_utils.tower`, but the extra
> step to nest those folders has been a pain point. So to keep the
> development-test cycle fast, I knowingly use the wrong imports in tests:
>
> https://github.com/ansible/awx/blob/a93b1aa3395651f01b39a03a6f122d3bdad99897/awx_collection/test/awx/conftest.py#L112
>
> This doesn't affect the functionality of the modules that import from
> module_utils, because they use relative imports.
>
> https://github.com/ansible/awx/blob/a93b1aa3395651f01b39a03a6f122d3bdad99897/awx_collection/plugins/modules/tower_credential.py#L221
>
> Again, I think this is a trick you only get with collections.

Good to know - we will eventually support collections, but we will have
to support the current roles for a while.

>
> ----
> If you keep the current folder structure, I wonder if instead of messing
> with sys.modules in the test code, you could just make the folders
> `ansible/module_utils/basic`, etc. in some far-off place, have those
> import from the path you know works, and put that in the python path
> too. Same as your current idea, it's very hacky but still isolated to
> your testing setup.

I was thinking of something like that, for tox/venv testing:
install ansible in the venv
cp or ln myrole/module_utils/* inside .tox/path/to/ansible/module_utils

That way I can script a solution for all of our system roles rather than
having to disable pylint checks and custom sys.modules hacking inside
unit test code.

> <mailto:ansible-devel%2Bunsu...@googlegroups.com>.

[Rich Megginson]

This works: https://github.com/linux-system-roles/template/pull/12