I have a use case where I'm using Luigi to schedule tasks running on a remote cluster. I'm using a similar subclass of Luigi.Task to the k8s job task (https://luigi.readthedocs.io/en/stable/api/luigi.contrib.kubernetes.html
) which waits for the remote job to complete.
Ideally, if I run N of these tasks in parallel, I'd like to avoid having N local processes that are mostly idle (infrequently polling for job completion). Is there any pattern to avoid doing this in Luigi?