Receptor error from awxee2.contoso.com, detail: Work unit expired

[alex]

I have had this issue happen twice. I setup an external EE Instance so that i can run jobs in a remote DC. Works fine for a week or two and then all of the sudden it becomes unavailable. I'm kind of lost on how to further troubleshoot this. I followed the install guide here.

The error on the instance page is:

Receptor error from awxee2.contoso.com, detail: Work unit expired on Wed Oct 4 00:49:22

The logs that show on awx-ee container are:

WARNING 2023/10/04 00:48:54 Backend connection exited (will retry) 2023-10-04T00:48:54.457949020Z ERROR 2023/10/04 00:48:54 Backend error: awx-task-7896549d8-p29k8: rejected connection with node awxee2.contoso.com because it connected using a node ID we are already connected to

The logs that show for awx-task are:

2023-10-04 01:02:36,995 DEBUG    [34562f643d3b499ab15a7f177d4b60a5] awx.main.dispatch task 6bb12855-4aff-42d0-8f3a-7d5fe7b0c287 starting awx.main.tasks.system.execution_node_health_check(*['awxee2.contoso.com'])
2023-10-04T01:02:38.772376462Z 2023-10-04 01:02:38,771 DEBUG    [30d2c51691aa4ad49ce03227e44dc927] awx.main.dispatch task ba0be6c7-2686-4bad-8039-e9b60a01f8d8 starting awx.main.scheduler.tasks.task_manager(*[])
2023-10-04T01:02:38.787801171Z 2023-10-04 01:02:38,784 DEBUG    [30d2c51691aa4ad49ce03227e44dc927] awx.main.dispatch task ce07bc03-8e71-4782-a097-2cc92197b1f4 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[])
2023-10-04T01:02:38.787837735Z 2023-10-04 01:02:38,785 DEBUG    [30d2c51691aa4ad49ce03227e44dc927] awx.main.dispatch task 9b7cba51-a76b-4345-951c-7a6fa8f2e374 starting awx.main.scheduler.tasks.dependency_manager(*[])
2023-10-04T01:02:38.787961632Z 2023-10-04 01:02:38,787 DEBUG    [30d2c51691aa4ad49ce03227e44dc927] awx.main.scheduler Starting task_manager Scheduler
2023-10-04T01:02:38.800783976Z 2023-10-04 01:02:38,798 DEBUG    [30d2c51691aa4ad49ce03227e44dc927] awx.main.scheduler Starting dependency_manager Scheduler
2023-10-04 01:02:57,080 INFO     [34562f643d3b499ab15a7f177d4b60a5] awx.main.tasks.system Failed to find capacity of new or lost execution node awxee2.contoso.com, errors:
2023-10-04T01:02:57.081439995Z Receptor error from awxee2.contoso.com, detail:
2023-10-04T01:02:57.081443129Z Work unit expired on Wed Oct 4 01:02:57
2023-10-04T01:02:58.819226834Z 2023-10-04 01:02:58,817 DEBUG    [23c3314b7ebb4795a0569f15c7045ebf] awx.main.dispatch task 14b740a8-0ed6-4f77-81da-2a19104a8991 starting awx.main.scheduler.tasks.task_manager(*[])
2023-10-04T01:02:58.829227133Z 2023-10-04 01:02:58,826 DEBUG    [23c3314b7ebb4795a0569f15c7045ebf] awx.main.dispatch task 94a9d588-2925-4b03-9868-0fa4875a1734 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[])