Luigi takes much longer to retry than client.cfg retry-delay param

[Carmen Mardiros]

This is what my client.cfg looks like

[core]

max-reschedules = 1200

retry-delay = 10

retry-external-tasks = true

worker-keep-alive = true

worker-wait-interval = 2

[scheduler]

record_task_history = False

I have an external task A which needs to complete before task B can complete.

I execute Task B first, this goes into FAILED state (as Task A is yet to be done) and the console prints:

DEBUG: Asking scheduler for work...

INFO: Done

INFO: There are no more tasks to run at this time

INFO: There are 1 pending tasks possibly being run by other workers

INFO: There are 1 pending tasks unique to this worker

DEBUG: Sleeping for 3.413523 seconds

....

In another terminal window I execute Task B which completes successfully.

Switching back to Task A -- eventually this DOES get retried but it is certainly not retried every 10 seconds (I've tried various values). In fact, it can take up to 30 mins before it gets picked up again. I thought "retry delay" is in seconds.

Any ideas how to make Luigi retry "quicker"?

[David Thompson]

You have retry-delay in the [core] section, but it should go in [scheduler] (and also not just in client.cfg but in the scheduler's config if you are running a remote scheduler).

This configuration is read by the scheduler when deciding whether it is time to change the state of a task to allow a retry.

[Noah Maze]

I'm running into a similar problem so I'll piggyback on this post: 

The default configuration of my scheduler's retry-delay is 900 seconds, but I would like to disable retries completely for a particular project of mine.  Like the original post, I tried client.cfg and it didn't work for the reasons you just mentioned.  Using scheduling-related arguments (e.g. luigi --scheduler-retry-delay 10000 --module myModule MyTask) hasn't worked either, probably for the same reasons.

Am I missing something, or is retry-failures only configurable globally?  If so, are there any workarounds to do what I want?

[David Thompson]

Yes the retry-delay setting is global along with the other scheduler configurations. And I'm not sure of a good way (or workaround even) that allows disabling retry behavior just for that worker. =/

I'm curious, is your goal to be able to say "exit immediately on error" for this worker, or is it "continue running other tasks but just don't retry failed ones"?

[Noah Maze]

The latter.  I'm running a bunch of tasks with one script and the whole run takes more than 15 minutes so luigi retries any tasks that failed along the way.  But this project is sand-boxed in such a way that if a task fails once it is definitely going to fail every time so retries don't do anything but waste CPU and spam error emails.

I'll see what I can do about negotiating a configuration change upstream but there are other projects on this machine that can be retried so I don't think that is an option for me.

The scheduler doesn't expose the number of failures or retries to a worker, does it? I could just add some logic to skip the CPU intensive parts (and emails) if I knew that the task had already failed somehow.

[Dave Buchfuhrer]

If this pipeline is truly sandboxed, you could just use a second scheduler instance that uses different settings. Then you can either disable on 1 failure or just wait forever to retry on that scheduler. This is probably the simplest option for you.

If you want to know that the task failed in your worker, just note the failure in a local file or db so you can check for them later. This can be run by wrapping the run function in a try clause.

For a minor tweak to luigi, you could use a modified version of luigi.worker used in that sandbox to send multiple failures to the scheduler on each fail in order to disable the job so it won't come back to you after the first failure.

For a bigger tweak to get the effect you want, luigi could be modified to accept retry delay and other overrides from the worker on a per-task basis.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Thompson]

Note separate scheduler instance can also be accomplished by using --local-scheduler with your own custom options locally.

[hammer...@gmail.com]

Greetings to everyone out there,
I bring good news to your door My name is Mr. Hooper Geoge  we are company loan lender, we offer private and business loans with a minimum annual interest rates as low as 3% within 1 year to 20years period repayment duration for any part of the world. And we give loans in $ 1,000 range for the maximum value of USD $ 100,000,000.00, Our loans are well insured and maximum security is our priority, interested persons should contact us today via email (woodgat...@outlook.com) for more information about this loan transaction,
                   BORROWERS DATA
COMPANY  LOANBUSINESS  LOANCAR               LOANHOUSE        LOANSTUDENT   LOAN
we are always ready to make sure your loan delivered to you basically accordance of the company, if you are interested in getting please concact us. 
BEST REGARD HOOPER GEOGE.Greetings to everyone out there,
I bring good news to your door My name is Mr. Hooper Geoge  we are company loan lender, we offer private and business loans with a minimum annual interest rates as low as 3% within 1 year to 20years period repayment duration for any part of the world. And we give loans in $ 1,000 range for the maximum value of USD $ 100,000,000.00, Our loans are well insured and maximum security is our priority, interested persons should contact us today via email (woodgat...@outlook.com) for more information about this loan transaction,
                   BORROWERS DATA
COMPANY  LOANBUSINESS  LOANCAR               LOANHOUSE        LOANSTUDENT   LOAN
we are always ready to make sure your loan delivered to you basically accordance of the company, if you are interested in getting please concact us. 
BEST REGARD HOOPER GEOGE. to everyone out there,
I bring good news to your door My name is Mr. Hooper Geoge  we are company loan lender, we offer private and business loans with a minimum annual interest rates as low as 3% within 1 year to 20years period repayment duration for any part of the world. And we give loans in $ 1,000 range for the maximum value of USD $ 100,000,000.00, Our loans are well insured and maximum security is our priority, interested persons should contact us today via email (woodgat...@outlook.com) for more information about this loan transaction,
                   BORROWERS DATA
COMPANY  LOANBUSINESS  LOANCAR               LOANHOUSE        LOANSTUDENT   LOAN
we are always ready to make sure your loan delivered to you basically accordance of the company, if you are interested in getting please concact us. 
BEST REGARD HOOPER GEOGE.Greetings to everyone out there,
I bring good news to your door My name is Mr. Hooper Geoge  we are company loan lender, we offer private and business loans with a minimum annual interest rates as low as 3% within 1 year to 20years period repayment duration for any part of the world. And we give loans in $ 1,000 range for the maximum value of USD $ 100,000,000.00, Our loans are well insured and maximum security is our priority, interested persons should contact us today via email (woodgat...@outlook.com) for more information about this loan transaction,
                   BORROWERS DATA
COMPANY  LOANBUSINESS  LOANCAR               LOANHOUSE        LOANSTUDENT   LOAN
we are always ready to make sure your loan delivered to you basically accordance of the company, if you are interested in getting please concact us. 
BEST REGARD HOOPER GEOGE.I bring good news to your door My name is Mr. Hooper Geoge  we are company loan lender, we offer private and business loans with a minimum annual interest rates as low as 3% within 1 year to 20years period repayment duration for any part of the world. And we give loans in $ 1,000 range for the maximum value of USD $ 100,000,000.00, Our loans are well insured and maximum security is our priority, interested persons should contact us today via email (woodgat...@outlook.com) for more information about this loan transaction,
                   BORROWERS DATA
COMPANY  LOANBUSINESS  LOANCAR               LOANHOUSE        LOANSTUDENT   LOAN
we are always ready to make sure your loan delivered to you basically accordance of the company, if you are interested in getting please concact us. 
BEST REGARD HOOPER GEOGE.

[masuda....@opst.co.jp]

2016年2月23日火曜日 18時24分06秒 UTC+9 Carmen Mardiros: