Triggering luigi tasks on a schedule

[Bilal Aslam]

We love and use luigi every day at my company. We'd like to start using it more for scheduled tasks (i.e. run a task every night). What schedulers does the community use (especially Python and Java shops) to kick off Luigi tasks? We're currently using cron and it's showing its age :)

Thanks!

~Bilal Aslam

[Erik Bernhardsson]

There's no built-in support in Luigi, we should probably add this (to encompass functionality like Chronos). Currently we just use cron at Spotify, but I think other people use other schedulers.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Erik Bernhardsson

Engineering Manager, Spotify, New York

[sara...@gmail.com]

Hi,

is there any update on this? I wanted to use similar scheduling in my project. is this feature being added to luigi.

thanks

[Lars Albertsson]

As indicated by the lack of response, it is not likely to happen. IMHO, it would be a mistake. Luigi's strength is the simplicity and well defined scope. It makes it easy to debug as well as integrate with other components. More features than necessary would make that worse.

People use whatever scheduling is available in the environment. If you are running Kubernetes, CronJob is a good fit. Good old cron works fine if you are not.

If you would prefer a scheduling tool with the opposite philosophy, i.e. leaning to more rather than fewer features, Airflow is a good option.

Lars Albertsson
Data engineering entrepreneur

www.mimeria.com, www.mapflat.com
https://twitter.com/lalleal
+46 70 7687109