Problem with PENDING tasks that are never scheduled again

626 views
Skip to first unread message

Ruhan Bidart

unread,
Dec 9, 2015, 10:02:01 AM12/9/15
to Luigi
Hi all,

I think I'm misunderstanding something about Luigi. 

I would like to make a task that, if failed a first time, tries to execute 4 times again before give up.

My configuration file looks something like:

[core]
worker-timeout: 120
max-reschedules: 4

[scheduler]
disable-num-failures: 4
retry-delay: 6
disable-window-seconds: 60
disable-hard-timeout: 90
retry-external-tasks: true
worker-keep-alive: true

I thought it should consider the task as failed only after tries 4 times, but my task enters in PENDING and the central scheduler don't try to execute it again.

How can I put the luigi central scheduler to behave like I want? Do you have any tip?

I'm also executing my task informing more than one worker because of answers on this question: 


Can anyone help me?

Cheers,
--
Ruhan

Dave Buchfuhrer

unread,
Dec 9, 2015, 10:42:02 AM12/9/15
to Ruhan Bidart, Luigi
If you want to retry four times before considering the task failed, you need to do that in your run function. You can try using a decorator like https://wiki.python.org/moin/PythonDecoratorLibrary#Retry for that effect.

By setting disable-num-failures to 4 and disable-window-seconds to 60, your task will go into FAILED state after each run until it has 4 failures within 60 seconds. After that, it'll become DISABLED rather than PENDING and won't be able to run again for 1 day. This will also happen if the task ever fails more than 90 seconds after its first failure due to your disable-hard-timeout setting.

If your task is pending and not being executed, it's probably because your worker died after the task failed and there weren't any PENDING tasks remaining. When your task became PENDING again after 6 seconds, the worker was already gone and there was nothing for the scheduler to give your task to.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ruhan Bidart

unread,
Dec 9, 2015, 1:00:38 PM12/9/15
to Luigi, ruhan...@gmail.com
Thank you a lot by your answer, it was very clear.

I have one more doubt: what kind of problems are capable to make the worker die?

Cheers,
--
Ruhan

Dave Buchfuhrer

unread,
Dec 10, 2015, 1:02:34 AM12/10/15
to Ruhan Bidart, Luigi
The worker will die if two criteria are met:
  1. it's not working on anything
  2. the scheduler responds to a get work request saying there are no pending tasks
This happens in your example because your task is FAILED rather than PENDING and your worker isn't working on anything else.

Ruhan Bidart

unread,
Dec 11, 2015, 7:57:47 AM12/11/15
to Luigi, ruhan...@gmail.com
Oh, ok! I understood it now! Thank you a lot by your answers!

aditya...@gmail.com

unread,
Aug 17, 2016, 3:19:24 PM8/17/16
to Luigi, ruhan...@gmail.com
But shouldn't the worker-keep-alive option keep the workers alive?

On Wednesday, December 9, 2015 at 11:02:34 PM UTC-7, Dave Buchfuhrer wrote:
> The worker will die if two criteria are met:
> it's not working on anythingthe scheduler responds to a get work request saying there are no pending tasks

alexandre....@gmail.com

unread,
Sep 24, 2016, 11:32:56 AM9/24/16
to Luigi, ruhan...@gmail.com, aditya...@gmail.com
I wonder if you ever clarified this?

I'm running through the same problem. The failed task reports as failed on the UI, after the delay time it report as Pending, after that it never re-executes again ...

Thanks in advance.

Cheers,
Alex

Jake Alexander

unread,
Jul 14, 2022, 2:26:19 PM7/14/22
to Luigi
I appreciate this is from 6 years ago... but did anyone get this solved?

I killed a worker and have some pending tasks that are just stuck with no obvious way to delete.

Thanks!
Reply all
Reply to author
Forward
0 new messages