Large task Issue

[Aaa Bbb]

1 playbook, 20 tasks, 10 targets/hosts. On 9 host tasks runs similar time - a couple of minutes. On 1 server 15'th tasks takes 1,5h. After that time last 5 task runs quickly and end result report is produced for whole 10 hosts.

I have two problems/questions.

Is there any simple way to debug/check which exactly task and on which host is a blocker? Now I have to analyze log output and check what task is "missing" from log - that task is running currently obviously.

I use "strategy: free" so my 9 hosts are done quite quickly(10 minutes) but I have report about it after my largest host/task - 2h after. I can't exclude that host and run it in separate job because I am not able to tell which one will be the blocker that day (I know it sounds weird - but it is). So is it possible to get some final report about a single host? It would be ofcourse between regular tasks but it is enough. Easy to grep and "tag"(in my excel) host as done (even without final successfull report).

[Dick Visser]

Maybe I'm stating the obvious but if one task is taking 1.5h then that
would be very visible while running the playbook.
So that task would be the culprit.
As to why it happens on different hosts, that's impossible to tell
without more information, such as what that task looks like.

Dick

> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/165a1067-818b-4dcf-a4d7-ea7134a6606d%40googlegroups.com.



--
Dick Visser
Trust & Identity Service Operations Manager
GÉANT

[Vladimir Botka]

On Mon, 16 Dec 2019 00:36:35 -0800 (PST)
Aaa Bbb <jsde...@gmail.com> wrote:

> 1 playbook, 20 tasks, 10 targets/hosts. On 9 host tasks runs similar time -
> a couple of minutes. On 1 server 15'th tasks takes 1,5h. After that time
> last 5 task runs quickly and end result report is produced for whole 10
> hosts.
>

> Is there any simple way to debug/check which exactly task and on which host
> is a blocker?

ansible-runner provides profiling
https://ansible-runner.readthedocs.io/en/latest/intro.html#runner-profiling-data-directory

> So is it possible to get some final report about a single host?

ansible-runner creates "artifacts" directory with the results
https://ansible-runner.readthedocs.io/en/latest/intro.html#runner-artifacts-directory-hierarchy

FWIW. See more details in "Ansible Logging Extreme Detail How-To?"
https://stackoverflow.com/questions/59184097/ansible-logging-extreme-detail-how-to/

Cheers,

-vlado

[Aaa Bbb]

Nope, It is not very visible. 10 hosts 20 tasks. 10x20 = 200 lines/tasks in output not in line. Please take a note that I am using "strategy free" -> So tasks are not realized one by one for all hosts in one step. Even without strategy free I woludn't know which exactly host is a blocker. I could see only previous - already done task. So the next one is running right now (I have to take a look in a code/playbook to check which one is next after already done task - info from output) - but on which host it takes the longest time - which one is still running when the rest is already done? I compare it (ansible/awx) to jenkins buid flow where in a split of second I can find really active task on a specyfic node. I would like to find out if it was possible on awx?

[Aaa Bbb]

Nope, It is not very visible. 10 hosts 20 tasks. 10x20 = 200 lines/tasks in output not in line. Please take a note that I am using "strategy free" -> So tasks are not realized one by one for all hosts in one step. Even without strategy free I woludn't know which exactly host is a blocker. I could see only previous - already done task. So the next one is running right now (I have to take a look in a code/playbook to check which one is next after already done task - info from output) - but on which host it takes the longest time - which one is still running when the rest is already done? I compare it (ansible/awx) to jenkins buid flow where in a split of second I can find really active task on a specyfic node. I would like to find out if it was possible on awx?

On Monday, December 16, 2019 at 9:50:05 AM UTC+1, Dick Visser wrote:

Maybe I'm stating the obvious but if one task is taking 1.5h then that
would be very visible while running the playbook.
So that task would be the culprit.
As to why it happens on different hosts, that's impossible to tell
without more information, such as what that task looks like.

Dick

On Mon, 16 Dec 2019 at 09:36, Aaa Bbb <jsde...@gmail.com> wrote:
>
> 1 playbook, 20 tasks, 10 targets/hosts. On 9 host tasks runs similar time - a couple of minutes. On 1 server 15'th tasks takes 1,5h. After that time last 5 task runs quickly and end result report is produced for whole 10 hosts.
>
> I have two problems/questions.
>
> Is there any simple way to debug/check which exactly task and on which host is a blocker? Now I have to analyze log output and check what task is "missing" from log - that task is running currently obviously.
>
> I use "strategy: free" so my 9 hosts are done quite quickly(10 minutes) but I have report about it after my largest host/task - 2h after. I can't exclude that host and run it in separate job because I am not able to tell which one will be the blocker that day (I know it sounds weird - but it is). So is it possible to get some final report about a single host? It would be ofcourse between regular tasks but it is enough. Easy to grep and "tag"(in my excel) host as done (even without final successfull report).
>
> --
> You received this message because you are subscribed to the Google Groups "Ansible Project" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ansible...@googlegroups.com.

[Aaa Bbb]

Thank You. Do You have any other advice in case of AWX usage (ansible-runner  is command line tool) ?