No module named 'packaging'

[Kevin Knox]

I'm running 20.10.2 with operator 1.1.3 and having a lot of joy. Things are beginning to work, but this is a new problem in the ephemeral worker node:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'packaging'

Am I reading correctly that the worker node has a bad Python install? Is there anything I can do about this?

Here's the full error:

{"uuid": "19026f90-0653-464f-9cfa-a471fd82c20c", "counter": 87, "stdout": "\u001b[0;31mAn exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'packaging'\u001b[0m\r\n\u001b[0;31mfatal: [10.xx.xx.xx]: FAILED! => {\"changed\": false, \"msg\": \"Failed to import the required Python library (packaging) on automation-job-130-r4fwp's Python /usr/libexec/platform-python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter\"}\u001b[0m", "start_line": 215, "end_line": 217, "runner_ident": "130", "event": "runner_on_failed", "job_id": 130, "pid": 20, "created": "2023-01-25T15:23:47.775259", "parent_uuid": "0a580a3a-3980-308c-1e30-000000000033", "event_data": {"playbook": "Ansible/playbooks/AWX_NonProd/named_deploy-certificate-f5.yml", "playbook_uuid": "c1685fdf-ef45-473b-a9e0-3c70711566ab", "play": "Deploy named certificate to F5 as Traffic Certificate", "play_uuid": "0a580a3a-3980-308c-1e30-000000000008", "play_pattern": "int", "task": "Add SNI profile to existing virtual server /Common/app-vip.vs in nadc-int", "task_uuid": "0a580a3a-3980-308c-1e30-000000000033", "task_action": "bigip_virtual_server", "resolved_action": "f5networks.f5_modules.bigip_virtual_server", "task_args": "", "task_path": "/runner/project/Ansible/playbooks/roles/PutMicroservice/tasks/full_install.yml:81", "role": "PutMicroservice", "host": "10.xx.xx.xx", "remote_addr": "10.xx.xx.xx", "res": {"exception": "Traceback (most recent call last):\n  File \"/tmp/ansible_bigip_virtual_server_payload_2bbo10x6/ansible_bigip_virtual_server_payload.zip/ansible_collections/f5networks/f5_modules/plugins/modules/bigip_virtual_server.py\", line 927, in <module>\nModuleNotFoundError: No module named 'packaging'\n", "msg": "Failed to import the required Python library (packaging) on automation-job-130-r4fwp's Python /usr/libexec/platform-python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter", "invocation": {"module_args": {"state": "present", "partition": "Common", "name": "[appdetail]", "profiles": ["ASM_mwosapps-sbx.policy", "http-xff", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "[appdetail]", "tcp", "websecurity", "websocket"], "check_profiles": false, "provider": {"server": "10.xx.xx.xx", "validate_certs": false, "server_port": null, "user": "ansible-f5-mw", "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", "transport": "rest", "timeout": null, "no_f5_teem": null, "auth_provider": null}, "type": "standard", "insert_metadata": true, "rate_limit_mode": "object", "bypass_module_checks": false, "destination": null, "port": null, "policies": null, "irules": null, "enabled_vlans": null, "disabled_vlans": null, "pool": null, "description": null, "snat": null, "default_persistence_profile": null, "fallback_persistence_profile": null, "source": null, "metadata": null, "address_translation": null, "port_translation": null, "source_port": null, "ip_protocol": null, "mirror": null, "auto_last_hop": null, "mask": null, "firewall_staged_policy": null, "firewall_enforced_policy": null, "ip_intelligence_policy": null, "service_down_immediate_action": null, "security_log_profiles": null, "security_nat_policy": null, "rate_limit": null, "rate_limit_dst_mask": null, "rate_limit_src_mask": null, "clone_pools": null}}, "_ansible_no_log": false, "changed": false}, "start": "2023-01-25T15:23:46.873682", "end": "2023-01-25T15:23:47.775046", "duration": 0.901364, "ignore_errors": null, "event_loop": null, "uuid": "19026f90-0653-464f-9cfa-a471fd82c20c"}}

[AWX Project]

are you using a custom built execution environment image, or the default "quay.io/ansible/awx-ee:latest"?

[AWX Project]

Hi!

what is the ansible task that causes this failure, and can you run job (and past stdout) with increased verbosity?

AWX Team

[Kevin Knox]

Having a little trouble with scrubbing my data. I apologize. 

I wish I could be sure of my answer. I have attached a couple mounts to the EE image that are not attached in the ephemeral workers the EE spins up. I'm working on that in another thread. That said, I believe I'm actually running the stock latest. Here's my yaml stanza:
  ee_images:
    - name: aep-custom-awx-ee
      image: quay-remote.internalartifactoryproxy.com/ansible/awx-ee:latest
  control_plane_ee_image: quay-remote. internalartifactoryproxy .com/ansible/awx-ee:latest

Here is some of the debug 3 output:
Top:
ansible-playbook [core 2.12.5.post0]
  config file = /runner/project/ansible.cfg
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/project/collections:/home/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
  jinja version = 2.11.3
  libyaml = True
Using /runner/project/ansible.cfg as config file
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Read vars_file '../../inventory/certificates/ds.global.myextcompany.com_vars.yml'
Read vars_file 'certificates/{{ app_variables }}'
Read vars_file '../../inventory/certificates/ds.global.myextcompany.com_vars.yml'
Read vars_file 'certificates/{{ app_variables }}'
Read vars_file '../../inventory/certificates/ds.global.myextcompany.com_vars.yml'
Read vars_file 'certificates/{{ app_variables }}'
statically imported: /runner/project/Ansible/playbooks/roles/PutCertificate-F5/tasks/put_f5.yml
Read vars_file '../../inventory/certificates/ds.global.myextcompany.com_vars.yml'
Read vars_file 'certificates/{{ app_variables }}'
statically imported: /runner/project/Ansible/playbooks/roles/PutMicroservice/tasks/full_install.yml
redirecting (type: modules) ansible.builtin.bigip_profile_client_ssl to f5networks.f5_modules.bigip_profile_client_ssl
redirecting (type: modules) ansible.builtin.bigip_profile_client_ssl to f5networks.f5_modules.bigip_profile_client_ssl
redirecting (type: modules) ansible.builtin.bigip_virtual_server to f5networks.f5_modules.bigip_virtual_server
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

And here is the actual error line:
  File "/tmp/ansible_bigip_virtual_server_payload_afyprj4l/ansible_bigip_virtual_server_payload.zip/ansible_collections/f5networks/f5_modules/plugins/modules/bigip_virtual_server.py", line 927, in <module>

ModuleNotFoundError: No module named 'packaging'

This says the problem is that the F5 module is calling out to Python and not finding a module. The thing is, this works great on AWX 20.0.1.

Thank you.

Kevin

[Kevin Knox]

Here's the whole log of the run at Debug 3. 

[AWX Project]

are you changing the ansible_python_interpreter variable (in the playbook or inventory level) at all?

we see these lines in your debug 3 logs

<10.xx.xx.xx> Attempting python interpreter discovery

then

ok: [10.xx.xx.xx] => { "ansible_facts": { "discovered_interpreter_python": "/usr/libexec/platform-python" },

you may try setting this variable at the host level (or ansible task level)

ansible_python_interpreter: '{{ ansible_playbook_python }}'

AWX Team

[Kevin Knox]

I certainly have no desire to do so and I do not believe I am. I've never written such a line. I will go away and search my code for it now.

[Kevin Knox]

I've grepped everything in my repo and a full export of the AWX server, and I only see that line in the Demo Inventory, which I'm not using. Looking harder at the play itself now. 

[Kevin Knox]

OK. Adding that in the play worked like magic. 
  vars:
    - ansible_python_interpreter: '{{ ansible_playbook_python }}'

Now I'm looking for what I could possibly have done to make that line necessary.