Resource limit on luigi.task

[eri...@gogotech.hk]

Hi all,

I am a beginner of using luigi and usually just run python code to execute my task.
I would like to know is there any tools for restricting computer resource (e.g. CPU & memory) while executing the task?
Thanks!

[Arash Rouhani Kalleh]

Good question. Unfortunately luigi doesn't come with support for that. I've previously seen solutions that before you start luigi, you do some pre-luigi checks and simply abort if the system is overloaded.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lars Albertsson]

Luigi does have a primitive for resource overload protection, but at a
global level, i.e. enforced by the central scheduler. The
documentation is well hidden:
http://luigi.readthedocs.org/en/stable/configuration.html#resources

Are you running multiple Luigi workers, and would like to avoid the
sum of resources required to overload the machine? In that case, it
might make sense to port the resource logic from the central scheduler
to the worker side. It would probably be a straightforward
implementation, and would be useful as simple overload protection,
e.g. on a farm of worker machines. It would be less powerful, but
simpler than running frameworks like Mesos.

If you are only concerned with preventing a single job from consuming
too much resources, just run it in a container, e.g. with ulimit or
Docker.

Regards,



Lars Albertsson
Data engineering consultant
www.mapflat.com
+46 70 7687109