How to have ansible collection installed in the "edit mode"?

[Roman Dodin]

Hi all,

I am developing a collection following the dir layout as prescribed by the skeleton.

I have the following 2 questions re dev workflows

1) How do you install the collection that you're developing in the "edit mode", like with pip you do `pip install -e`, and then you can do changes to the modules on-the-fly and they will be picked up.
What is the alternative to that process with galaxy collection? Should I always do a `galaxy collection install ...` and then run my playbooks every time I touch the contents of my collection?

2) When developing modules inside the collection people reference funcs/methods from inside the very same collection, example - https://github.com/ansible-collections/arista.eos/blob/main/plugins/modules/eos_config.py#L319

How does this path get resolved? Who adds ansible_collection to the PYTHONPATH?

How do I make my python IDE to resolve that path during the developments

Thank you, and appreciate your comments.

[Matt Martz]

1) You just use normal `git` or whatever your version control system is:

cd ~/.ansible/collections/ansible_collections
rm -rf amazon/aws
mkdir -p amazon
cd amazon
git clone https://github.com/ansible-collections/amazon.aws.git aws

2) I don't know about IDEs, but the PYTHONPATH for those is managed by ansible during runtime effectively.  But basically, for the example above: PYTHONPATH=~/.ansible/collections/.  You can use `ansible-config dump | grep COLLECTIONS_PATHS` to see the default paths used. 

--
You received this message because you are subscribed to the Google Groups "Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-deve...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-devel/3e0a9a67-fbc0-44aa-846b-85772b710dbcn%40googlegroups.com.

--

Matt Martz
@sivel
sivel.net

[Roman Dodin]

Thanks Matt,

Q1 was not about vcs, maybe I poorly explained the problem. I was wondering how to make a collection source to be picked up by the playbooks without installing collection with galaxy-install.
I found only one way of doing it: tuning collections_path, but the problem is that this cfg item expects a strict folder structure. e.g. it needs to point to a dir D that has X/Y subdirs, where X is the namespace of the collection and Y is the collection name. Clearly, this messes up with whatever dir structure you have on your dev machine.

So if you started your project in dir structure such as ~/E/F, then you can't use collections_path.

[Matt Martz]

Not positive I understand, but if you want to use some arbitrary directory, it must be in a path that matches `collections/ansible_collections/<namespace>/<name>`

But my instructions still stand, regardless of using a vcs, they show how to put a collection in place without use of ansible-galaxy. You just need to drop the collection in a path that ansible is configured to look for, and it must be structured as I mentioned above.

You can configure the paths that ansible looks using https://docs.ansible.com/ansible-core/devel/reference_appendices/config.html#collections-paths

[Roman Dodin]

Yes, clear, thanks Matt.
it just felt wrong to me - imposing a predefined dir structure to import a collection from an arbitrary path. I tried symlinked dir, but didn't work either :D.

Nevertheless, appreciate your comments, I will likely resort to using `galaxy-install` as a step before running any playbooks during the dev time.
I was just feeling that I miss something obvious, but seems that I do not.

[Brian Coca]

symlinks do work, but you still need an ansible_collections/ dir
before your collection




--
----------
Brian Coca

[Brian Coca]

ls -l ~/.ansible/collections/ansible_collections/
total 20
lrwxrwxrwx 1 bcoca bcoca 54 Oct 7 2021 bcoca ->
/home/bcoca/work/collections/ansible_collections/bcoca

--
----------
Brian Coca

[Roman Dodin]

As for the 2nd question the only workaround I found is to use relative imports, which I want to get rid of.

Let me give you an example. In my collection dir I have the plugins subdir that has:

plugins
├── httpapi
│   └── sr.py
├── module_utils
│   └── sr.py
└── modules
    └── get.py

In my get.py I have to import a class from module_utils, and the full import path looks like

from ansible_collections.<namespace>.<collection>.plugins.module_utils.srlinux import MyClass

How do I make my python instance to resolve this path? I understand that at runtime ansible does add collections dir to the PYTHONPATH (or something of sorts), but without running ansible, how do i make my dev environment to resolve this path?

Thank you all for your help and valuable inputs.

PS. I can solve this with a relative path like

from ..module_utils.sr import MyClass

but I would like to not use relative imports if possible.

[Matt Martz]

As mentioned, you need to set `PYTHONPATH` in your IDE to the directory containing `ansible_collections`.  Like:  `PYTHONPATH=~/.ansible/collections`.

I don't use a fancy IDE, so I don't have any direct guidance on where you would need to set this.  Maybe the IDE doesn't respect this environment variable, or has other ways to configure it.

Effectively, the `ansible_collections/` directory is similar to a Python namespace package, and the `collections/` directory is similar to `site-packages` used by python.  So you just need to point `PYTHONPATH` or whatever else your IDE may use to all collections paths that ansible is configured to consult.  I mentioned this in my original reply.

[Roman Dodin]

Yes, this is clear, but then it will point to the "installed" collection. Unless developing directly in the ~/.ansible/collections or using symlinks like Brian mentioned above, this won't use the files from my dev directory.

I was looking for a way to make my local files in a dir that doesn't belong to ~/.ansible/collections or any other dir structure that I can use in a collections_path appear in dir that will resolve.

Will move my dev dir to a predefined structure otherwise.

[Roman Dodin]

I think I solved these two questions, thanks to Brian's comment on symlinks and general guidance by Matt. Appreciate.

TLDR: I wanted to have an ansible collection dir to be located anywhere, not following a particular dir structure, and satisfy the following requirements:

1. ansible-playbook should load the collection that is under development without doing galaxy-install. Similar to `pip install -e` mode.

2. Internal Import paths in my collection should use the full paths and resolve by the IDE.

To do that, I did the following:

1. create an ansible skeleton dir at ~/somedir/collection

2. create a temp dir that satisfies the collection's path structure: mkdir -p /tmp/somedir

3. create a symlink to the path where you develop the collection: ln -s "$(pwd)" /tmp/somedir/ansible_collections/<namespace>/<collection_name>

3. Put in ansible.cfg the following

[defaults]

collections_paths = /tmp/somedir

4. this will make ansible to use the collection without requiring an installation step

5. To make python import paths resolve, I created an .env file in the collection's root dir stating

PYTHONPATH=/tmp/somedir

6. This will make the python interpreter pick up import paths like: from ansible_collections.<namespace>.<coll_name>