Avoid waiting on long running remote tasks

[Daniel Friar]

I have a use case where I'm using Luigi to schedule tasks running on a remote cluster. I'm using a similar subclass of Luigi.Task to the k8s job task (https://luigi.readthedocs.io/en/stable/api/luigi.contrib.kubernetes.html) which waits for the remote job to complete.

Ideally, if I run N of these tasks in parallel, I'd like to avoid having N local processes that are mostly idle (infrequently polling for job completion). Is there any pattern to avoid doing this in Luigi?

(Possible solutions I've thought of are trying to decrease resources to zero in the task that runs the job - like here https://github.com/spotify/luigi/issues/2576 - or possibly using an ExternalTask).