Custom awx-ee image not used properly by AWX

[Stucksam]

Hi there

We are currently trying to create a new awx-ee image for testing purposes on a minikube installation. However, while the creation step works properly, the integration into awx itself fails miserably.

When executing jobs with this new image the process always fails saying it is missing python packages such as xmltodict which are required by the task. When checking the image for all its installed packages the "missing" packages are listed as installed after running "docker run our_container_registry_url/custom-awx-ee-test pip list".

After checking the event logs with "kubectl get events" we see that the image is successfully downloaded by minikube. Only after getting to the specific step in the ansible playbook where the package is needed will the jobs fail.

Has anyone had a similar experience or knows of ways to mitigate this? We suspect that maybe the job reverts to the default image which then fails because it lacks these packages but this is just speculation on our part. It wouldn't make much sense as well because the jobs clearly state which image to use...

Thanks for your help in advance!

[Arlan Lloyd]

There is not a lot of troubleshooting information here.  No source code for your image, no reference to AWX version, or how you are specifying execution environment the job is using.   My guess is you are missing the ansible-runner python package.  

Here is a functional execution environment reference, base image is Ubuntu   https://github.com/acelinkit/ansibletools

Cheers

Arlan

[Stucksam]

Thanks for your response Arlan and sorry for not providing more information, I added our image setup as an attachment. The thing is we were able to create a functioning image one month ago with a similar setup (only a package changed) which is based on the linkedin blog post at https://www.linkedin.com/pulse/creating-custom-ee-awx-phil-griffiths/?trk=public_profile_article_view

Concerning our environment:

We have AWX 19.5.0 installed on minikube running on a Debian 11 host. We use an ansible playbook to create the python virtual env which is the "modify_awx_ee_image.yml" file in the attachment. The requirements.txt/.yml and bindep.txt files are all in the context/_build folders.

Concerning the specification of which awx-ee image to use we configured the settings of the templates to use the image as follows:

Hope this helps a bit more.

Cheers

Samuel

[Stucksam]

Well it seems the screenshot didn't work, here it is again as an attachment

[phil.gr...@gmail.com]

This info doesn't help, doesn't show anything for troubleshooting.

You need to provide the output from the job template run showing the full output where it's not finding the modules.

I suspect you're using localhost in the JT, and it's picking up /usr/libexec/python or similar instead of the correct python built into the EE.

The normal fix for this is setting 

vars:

  ansible_python_interpreter: /usr/bin/python3 (or your installed version)

in the JT