Avoid waiting on long running remote tasks

20 views
Skip to first unread message

Daniel Friar

unread,
Sep 14, 2021, 6:50:27 PM9/14/21
to luigi...@googlegroups.com
I have a use case where I'm using Luigi to schedule tasks running on a remote cluster. I'm using a similar subclass of Luigi.Task to the k8s job task (https://luigi.readthedocs.io/en/stable/api/luigi.contrib.kubernetes.html) which waits for the remote job to complete.

Ideally, if I run N of these tasks in parallel, I'd like to avoid having N local processes that are mostly idle (infrequently polling for job completion). Is there any pattern to avoid doing this in Luigi?

(Possible solutions I've thought of are trying to decrease resources to zero in the task that runs the job - like here https://github.com/spotify/luigi/issues/2576 - or possibly using an ExternalTask). 
Reply all
Reply to author
Forward
0 new messages