Job output/logs stopped

[Wajid Baig]

Hello, 

We have deployed AWX on EKS. When 500 Jobs are triggered to do scalability test, it stopped giving output. Jobs were running successfully but there was no output

Every job output has "This job has no logs"

wondering what could have caused this issue?

Please help me troubleshoot this condition. 

Thanks in advance.

Best wishes & Regards,

Wajid Ahmed Baig

[AWX Project]

Hi! So you are seeing that jobs are finishing with a successful status, but there is no stdout displayed on the Job Detail page? What happens if you refresh the page after the job is successful? After a job is finished, there is a period of time where events still being written to the database, and during this period they might yet display in the UI. Do the events eventually show (like, after 30 minutes after running)?

You may also look at the /api/v2/metrics endpoint, in particular the callback_receiver_events_queue_size_redis metric which is the queue length for events waiting to be written to the database. If this is high, then the delay for seeing these events in the UI may also be high.

Also, how many events typically are emitted from these jobs? (i.e. when you run less of them)

AWX Team

[Wajid Baig]

Hello AWX Project,

Thanks for your reply. 

The jobs which did not output don't have output now also. In details we are seeing: 

No output found for this job.

Thanks & Regards,

Ahmed

[AWX Project]

Hi! Do you get output when you run a single job (i.e. not 500)?

AWX Team

[Wajid Baig]

Hi, When this issue happened there was no output for single job also. After we did rollout restart we started seeing output for jobs that we run after restart.

Thanks & Regards,

AB