Problem with PENDING tasks that are never scheduled again

[Ruhan Bidart]

Hi all,

I think I'm misunderstanding something about Luigi. 

I would like to make a task that, if failed a first time, tries to execute 4 times again before give up.

My configuration file looks something like:

[core]

worker-timeout: 120

max-reschedules: 4

[scheduler]

disable-num-failures: 4

retry-delay: 6

disable-window-seconds: 60

disable-hard-timeout: 90

retry-external-tasks: true

worker-keep-alive: true

I thought it should consider the task as failed only after tries 4 times, but my task enters in PENDING and the central scheduler don't try to execute it again.

How can I put the luigi central scheduler to behave like I want? Do you have any tip?

I'm also executing my task informing more than one worker because of answers on this question: 

https://groups.google.com/forum/#!msg/luigi-user/mH7NWfOr-4I/yVEkc6g7DAAJ

Can anyone help me?

Cheers,

--

Ruhan

[Dave Buchfuhrer]

If you want to retry four times before considering the task failed, you need to do that in your run function. You can try using a decorator like https://wiki.python.org/moin/PythonDecoratorLibrary#Retry for that effect.

By setting disable-num-failures to 4 and disable-window-seconds to 60, your task will go into FAILED state after each run until it has 4 failures within 60 seconds. After that, it'll become DISABLED rather than PENDING and won't be able to run again for 1 day. This will also happen if the task ever fails more than 90 seconds after its first failure due to your disable-hard-timeout setting.

If your task is pending and not being executed, it's probably because your worker died after the task failed and there weren't any PENDING tasks remaining. When your task became PENDING again after 6 seconds, the worker was already gone and there was nothing for the scheduler to give your task to.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ruhan Bidart]

Thank you a lot by your answer, it was very clear.

I have one more doubt: what kind of problems are capable to make the worker die?

Cheers,

--

Ruhan

[Dave Buchfuhrer]

The worker will die if two criteria are met:

1. it's not working on anything
2. the scheduler responds to a get work request saying there are no pending tasks

This happens in your example because your task is FAILED rather than PENDING and your worker isn't working on anything else.

[Ruhan Bidart]

Oh, ok! I understood it now! Thank you a lot by your answers!

[aditya...@gmail.com]

But shouldn't the worker-keep-alive option keep the workers alive?

On Wednesday, December 9, 2015 at 11:02:34 PM UTC-7, Dave Buchfuhrer wrote:
> The worker will die if two criteria are met:

> it's not working on anythingthe scheduler responds to a get work request saying there are no pending tasks

[alexandre....@gmail.com]

I wonder if you ever clarified this?

I'm running through the same problem. The failed task reports as failed on the UI, after the delay time it report as Pending, after that it never re-executes again ...

Thanks in advance.

Cheers,
Alex

[Jake Alexander]

I appreciate this is from 6 years ago... but did anyone get this solved?

I killed a worker and have some pending tasks that are just stuck with no obvious way to delete.

Thanks!