Collections requiring additional python libraries

[kaleb....@gmail.com]

I'm trying to use a netapp collection - I have enabled collection and role download within AWX, but the error I'm getting is `the python NetApp-Lib module is required`

I've tried putting a requirements.yaml and collection/requirements.yaml in my SCM with the test playbook, but it doesn't seem to grab everything necessary.

Netapp documentation says I need to modify the build image for awx, but that's a step I'd rather not take if I don't have to.. are there other options?

AWX v 17.1.0

Ansible Version 2.9.18

Running in an Openshift 3.11 Cluster

Regards,

Kaleb

[kaleb....@gmail.com]

I received a direct reply (Thank you Stefan!) which pointed me in the direction of using Ansible-Builder.. I've got that working and creating a context... I'm now at the point of wanting to try running that in AWX but am uncertain how to proceed with that next step..

Any suggestions/thoughts would be greatly appreciated.

[Elijah DeLee]

I think the ansible-builder approach of creating a custom execution environment with the collection and using that for the job execution won't work until you upgrade to an AWX version 18+ ... which is not released yet :D

What does the output of your project update look like?

One problem I've faced is that if you don't have an ansible galaxy credential associated with the organization that the project belongs to (I know, this is obtuse), the collections download skips.

I'm of the opinion that this is opaque and is a UX bug, but has not been prioritized yet to get fixed (I'm not sure if there is an issue for this yet on awx).

Let me know if trying this changes the outcome, and if not, what the output of the project update is.

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/4a8c84ee-73ec-4693-a947-5265ac817792n%40googlegroups.com.

--

Elijah DeLee

------------------

Senior Quality Engineer

Red Hat Ansible Tower

[Kaleb S]

Hi Elijah,

The problem I have is when I try to run a playbook with a reference to the netapp.ontap collection, it errors out that the netapp-lib python module isn't available. The netapp documentation I was pointed at originally when I was looking at this suggested customizing the awx build to include the module at the base level.. but that's not something I'm wanting to do as I don't want to have to re-build that part of the project everytime I want to update awx (or if I have ot update the python library).

Is there another solution while I'm waiting for 18+ ? 

Regards,

Kaleb

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CAHmbr-8Q4bOeZ6breirc3UYt_%3DcmCUYDTaWh1ULXFKHStsPuMg%40mail.gmail.com.

[Elijah DeLee]

If the project where you playbook (e.g the project you use for the Job Template) has a file called:

$PROJECT_ROOT/collections/requirements.yml

And the organization that the project belongs to has a Ansible Galaxy credential associated with it,

the collection will be downloaded at project update time, and your playbook should work.

This does not require changing the base image of AWX.

[Elijah DeLee]

The collection requirements file format is detailed here:

https://docs.ansible.com/ansible/latest/galaxy/user_guide.html#install-multiple-collections-with-a-requirements-file 

[Kaleb S]

I wasn't aware of the requirement for the organization to have an ansible galaxy credential tie back.

I've added the credential to my organization, and confirmed I have a collections/requirement.yml with the following:

---
collections:
  - name: netapp.ontap
  - name: netapp.storagegrid

and it's still erroring out with needing the netapp-lib python module.

 "msg": "the python NetApp-Lib module is required",

I also tried with a $PROJECT_ROOT/requirements.txt with the netapp-lib in the thought that it would pull the python module in but no such luck...

[Elijah DeLee]

On Thu, Mar 18, 2021 at 3:37 PM Kaleb S <kaleb....@gmail.com> wrote:

I wasn't aware of the requirement for the organization to have an ansible galaxy credential tie back.

I've added the credential to my organization, and confirmed I have a collections/requirement.yml with the following:

---
collections:
  - name: netapp.ontap
  - name: netapp.storagegrid

and it's still erroring out with needing the netapp-lib python module.

 "msg": "the python NetApp-Lib module is required",

:facepalm:

I got fixated on the collection thing. This is saying that you need the python dependency for the collection.

Yeah, for 17.x you are going to need to either modify the built in ansible venv of your awx image or build a custom venv and install that python dependency. https://github.com/ansible/awx/blob/17.1.0/docs/custom_virtualenvs.md

For 18.x, the whole paradigm is changing and you'd build a custom Execution Environment with ansible builder.

[Kaleb S]

Ahh... okay that makes more sense as to why I'm having such troubles!

In looking at preparing the virtualenvs, I am not sure the best way to proceed.. I can connect to my awx instance through Openshift, but the awx local user is not configured with a known password, and from what i can tell, there needs to be an $AWX_ADMIN_USER and $AWX_ADMIN_PASSWORD variables set somewhere ( according to the config map).. 

I also see the K8s custom virtualenvs.. is that a better option? Add the virtual environment through the initial deployment, and then assign it to the organization, project, and job template?

Also - I also don't fully understand the idea of how to implement the PATCH as indicated at the bottom of the document.

[Elijah DeLee]

I am not an expert on custom venvs in awx deployed on k8s, but I think that is the section of the docs you want to use.

The PATCH request are describing how to associate the custom venv with organizations, projects, job templates, etc through the API. You can do this through the browsable API. I'm not sure if it is implemented in the new UI, since it is going away in the upcoming release.

[kaleb....@gmail.com]

Okay - the patch request makes sense after browsing the api on my awx install..thank you for the pointers there!

Do I have to put the venv under a specific folder? (I know I can change the folder through the custom_venv_dir in my inventory file, but is that actually supported?

If I try to add a 

---

custom_venvs:

  - name: foo

    python: python3

    python_ansible_vesrion: 2.9.1

    python_modules:

      - netapp-lib

my init-custom-venvs container crash-loops because pip isn't being updated, and one of the (I'm assuming base) modules fails because pip is too old (cryptography in this case.

You are using pip version 9.0.1, however version 21.0.1 is available.

"You should consider upgrading via the 'pip install --upgrade pip' command

I'm not quite sure what I can add to the custom venv file to update pip before it tries to run the rest of the venv creation process...any thoughts?

[Kaleb S]

I managed to make it through! I had to modify the roles/kubernetes/templates/deployment.yml.j2 and add:

@ line 153 (before the pip install, but after teh /bin/activate &&

{{ custom_venvs_path }}/{{ custom_venv.name }}/bin/pip install --upgrade pip &&

and then my custom venv was able to build successfully!

I know the plan is to move away from venvs, but is this something worth putting in a bug or clean it up and put in a PR?

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/b8da0666-bbd1-4e15-bee3-d7d61990f66fn%40googlegroups.com.

[selva...@gmail.com]

I am getting same issue in AWX 19.5.0.  Could some one please help me here.

"msg": "the python NetApp-Lib module is required"

[Wei-Yen Tan]

Create a custom venv with the dependencies in it  and map it to the container 

Get Outlook for iOS

---

From: awx-p...@googlegroups.com <awx-p...@googlegroups.com> on behalf of selva...@gmail.com <selva...@gmail.com>
Sent: Saturday, January 22, 2022 6:26:18 AM
To: AWX Project <awx-p...@googlegroups.com>
Subject: [awx-project] Re: Collections requiring additional python libraries

 

--

You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/36a0fe76-8fae-4296-b34b-34e9f7e30710n%40googlegroups.com.

[selva...@gmail.com]

shall  I get detailed step please.... Thanks.

[Kaleb S]

Sorry - my initial ask was using an older version of AWX (17.1.0) - for newer versions, the recommended approach nowadays is to use ansible-builder (stolen answer from a different post) - I'm sorry I don't have more information than that, as i'm still using 17.1.0 in my environment.

I found this on a quick google, though I don't know where you would need to go from here:
https://www.ansible.com/blog/introduction-to-ansible-builder

Regards,

Kaleb

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/8eb76ad4-99f2-4572-8fb2-28ddb55489c1n%40googlegroups.com.

[Wei-Yen Tan]

https://github.com/surfer190/fixes/blob/master/docs/ansible/awx-giving-a-project-a-custom-virtual-env-in-container-based-install.md

Get Outlook for iOS

---

From: awx-p...@googlegroups.com <awx-p...@googlegroups.com> on behalf of Kaleb S <kaleb....@gmail.com>
Sent: Saturday, January 22, 2022 7:36:06 AM
To: selva...@gmail.com <selva...@gmail.com>
Cc: AWX Project <awx-p...@googlegroups.com>
Subject: Re: [awx-project] Re: Collections requiring additional python libraries

 

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CACFMj%2Bt7pgAJ5xGq0p%2B6_V3fcc6aYQiw%2BJQh6v4VhX8bm0w7pQ%40mail.gmail.com.