Speed up automation_job pod's creation

[Danny Duménigo]

Hello! I was able to deploy AWX using kurokobo's tutorial describe here: https://github.com/kurokobo/awx-on-k3s
Everything works fine, but I have notice that whole process of playbook's execution takes a bit longer than my prior environment (awx 14.10.1 using docker-compose)
I consider this behavior normal and logical due the automation_job pod creation.
For example a job that before takes 30 sec, in this new environment takes about 1:30 min, I notice that playbooks execution per se, maintain the same times
of my prior environment (in this example about 30 sec both), but is the pod creation that takes between 1~1:20 min, which is the reason of the whole process takes longer to complete.
Due to intensive use of AWX at my workplace I need to short this times to minimal, so... With this in mind and looking for a way to speed up the whole process I have tried setup capacity values that best fit my env (https://github.com/ansible/awx/blob/devel/docs/capacity.md),
try using mitogen (https://github.com/mitogen-hq/mitogen/blob/master/docs/ansible_detailed.rst) and try to set up (without luck) an execution node (https://github.com/ansible/awx/blob/devel/docs/execution_nodes.md).

So my question: is there a way to make automation_job's creation faster than what it is today? Or in general what do you suggest me to do in n this case?

My host environment is:
cpu: 4vcpus
memory: 8Gb RAM.
awx-operator: 0.29.0    
awx: 21.6.0
k3s: v1.24.7+k3s1

Thanks in advance for any help or guidance you can give me.

[AWX Project]

Hello,

This is likely due to image pulls. We have a few questions for you:

Do you have pull always set in the execution environment?

Are you using a custom EE or are you using the latest EE image?

Where are you pulling the images from?

Please let us know how you are handling those particular instances and we can certainly further assist you. 

-AWX Team

[Danny Duménigo]

Hello! Thanks for the quick answer, I appreciate it!!

Do you have pull always set in the execution environment?

A/ No, I set: "Never pull container before running" because I have the image imported locally. I used this guide: https://github.com/kurokobo/awx-on-k3s/tree/main/builder#use-ee-in-awx-without-container-registry

Are you using a custom EE or are you using the latest EE image?

A/ I build my own custom EE image. I only install community.postgresql collection and python package psycopg2-binary, but essentially use this guide: https://github.com/kurokobo/awx-on-k3s/tree/main/builder#build-your-own-custom-ee

Where are you pulling the images from?

A/ I import locally the image using "k3s ctr images import" command, so I have it cached on the container runtime for Kubernetes.

Hope this can clarify you, if there is something else needed please let me know.

Once again, thanks a lot!

Best regards

[AWX Project]

Hello,

Thank you so much for providing that additional information!

Based on the information you provided regarding the manner in which you are pulling images, we do not feel that the timing is due to the image pull. 

Could you check if there is a  project sync for those jobs? Settings for project syncs can be adjusted, and this could help you. Please let us know if that is fruitful.

-AWX Team

[Danny Duménigo]

Hello!

I do not have set any project's sync for these particular jobs. Due to I have few projects, I sync them mannually once I have done a change.

Best Regards

Danny

[kurokobo]

Hi,

Does your project repository contain large files?

All files in project will be compressed, transfered from AWX to your EE for your job through Kubernetes API, and extracted. This happens every job run, so if your project contains some big file, it takes some time.

Regards,

--

@kurokobo

------- Original Message -------

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/78d7b85e-8ed3-47e9-af33-9bf5f39af908n%40googlegroups.com.

[Danny Duménigo]

Hello!! 

Yes, my repository has some large files in it. Following your idea, I did some tests with smaller repositories and as expected, the automation_pod was rapidly created and jobs run very fast.

Thanks a lot for clarify me!! I am good to go on what to do next.

Regards

PD: By the way @kurokobo, I cannot thanks enough your enourmous contribution and your amazing, detailed and always updated GitHub repository (which I use to deploy my AWX env). 

[kurokobo]

Hi Danny,

> Following your idea, I did some tests with smaller repositories and as expected, the automation_pod was rapidly created and jobs run very fast.

Good to hear that, thanks for updating. I don't know what kind of files are required for your job but to eliminate the need to transfer lage files, placing large files on additional PV and mounting it from EE using Container Group might be one of the solution I guess. 

> PD: By the way @kurokobo, I cannot thanks enough your enourmous contribution and your amazing, detailed and always updated GitHub repository (which I use to deploy my AWX env).

Wow you're very welcome, and I will do my best to keep updating it :)

Regards,

--

@kurokobo

------- Original Message -------

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/7e521911-83ea-41e9-98f4-5f5663f401b3n%40googlegroups.com.