Luigi Scheduling randomly gets stuck pending after upgrade to Luigi 2

[aha...@totango.com]

Hi All

We've just upgraded Luigi from v1.0.18 to v2.1.1 and are running into problems with the Luigi Scheduler.

We run here about 12000 Tasks daily , this problem affects roughly 10% of these jobs seemingly randomly.

The symptoms are the following:
Lets say I run a Task call ActiveListRealtimeTask with 2 params as follows:

DEBUG: Checking if ActivelistRealtimeTask(service_id=880, pipeline_id=61795892-cb8e-4dbd-b86f-686594775b8a) is complete
INFO: Informed scheduler that task ActivelistRealtimeTask_61795892_cb8e_4d_880_a118eec427 has status PENDING
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1

This code will sit pending for 20-60 minutes before actually running (which takes 2 minutes)

I know clearly from our system that we aren't running this task with the same 2 params anywhere else. So I'm really not sure why the scheduler seems to take an hour to allocate it.
Also a search through the logs of the Luigi server shows nothing out of the ordinary (just a whole lot of pings and completed tasks)

Any help greatly appreciated!

Aharon Levine
Totango

[Arash Rouhani Kalleh]

What does it say after "DEBUG: Pending tasks: 1"? Is that the last thing it says before sleeping for 20-60 minutes? If so, I think the error is on the client's side. It gets stuck somewhere in the worker.py code. If you can reproduce it, try Ctrl+C your way to a stack trace.

Also, half-relevant, the scheduler (if you look at the graph in browser) indicates that the task is ready to run, correct?

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[aha...@totango.com]

Hi Arash

Your correct, after further checking the problem was on our implementation code and had nothing to do with scheduling.

Thanks for the speedy reply!

[victoria lisa]

✅🔴▶️▶ Really Amazing ️You Can Try This ◀️◀️🔴✅

✅▶️▶️ CLICK HERE Full HD✅720p✅1080p✅4K✅

WATCH ✅💻📺📱👉https://co.fastmovies.org

ᗪOᗯᑎᒪOᗩᗪ ✅📺📱💻👉https://co.fastmovies.org

🔴WATCH>>ᗪOᗯᑎᒪOᗩᗪ>>LINK>👉https://co.fastmovies.org

✅WATCH>>ᗪOᗯᑎᒪOᗩᗪ>>LINK>👉https://co.fastmovies.org

💚WATCH>>ᗪOᗯᑎᒪOᗩᗪ>>LINK>👉https://co.fastmovies.org

💚 Really Amazing ️You Can Try This💚WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>
🔴ALL>Movies>ALL>TIME>HITE>Save>LINK👉https://co.fastmovies.org

💚CLICK HERE Full HD>720p>1080p>4K💚WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>
🔴ALL>Movies>ALL>TIME>HITE>Save>LINK👉https://co.fastmovies.org

🔴💻ALL>Movies>WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>👉https://co.fastmovies.org
🔴📱ALL>Movies>WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>👉https://co.fastmovies.org

🔴📺ALL>Movies>WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>👉https://co.fastmovies.org
🔴📺ALL>Movies>WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>👉https://co.fastmovies.org

🔴✅📺📱💻ALL>Movies>WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>👉https://co.fastmovies.org
🔴✅📺📱💻ALL>Movies>WATCH>ᗪOᗯᑎᒪOᗩᗪ>LINK>👉https://co.fastmovies.org