running shell script from luigi
1,854 views
Skip to first unread message
Imran Akbar
Dec 4, 2013, 1:59:12 PM12/4/13
to luigi...@googlegroups.com
Luigi is looking awesome - thanks for all the great work so far.
I'm trying to run a local shell script from Luigi, but I couldn't find any examples or documentation for that.
Am I supposed to use luigi.LocalTarget?
thanks,
imran
Martin Czygan
Dec 4, 2013, 2:36:47 PM12/4/13
to Imran Akbar, luigi...@googlegroups.com
I guess since you can just run any code in run, just utilize python
subprocess module[1] - which luigi uses internally as well (e.g.
https://github.com/spotify/luigi/blob/master/luigi/hadoop.py#L242 and
in a few other modules).
As for dealing with "targets", there are surely many possibilities.
We wrote simple helper function (source:
https://gist.github.com/miku/7793932), which helps us to write code
like this:
...
def run(self):
# without explicit "output" - "tmp" will be some random temp file
# will raise a RuntimeError, if subprocess exits with non-zero exit code
output = shellout("xsltproc {stylesheet} {input} > {output}",
stylesheet="/path/to/stylesheet",
input=self.input().fn)
# atomically move file
luigi.File(output).move(self.output().fn)
Basically, shellout will return the {output} path, which you can the
use to move the file to final place atomically via File.move.
I'm curious about other approaches.
Martin.
[1] http://docs.python.org/2/library/subprocess.html
> --
> You received this message because you are subscribed to the Google Groups
> "Luigi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to luigi-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
subprocess module[1] - which luigi uses internally as well (e.g.
https://github.com/spotify/luigi/blob/master/luigi/hadoop.py#L242 and
in a few other modules).
As for dealing with "targets", there are surely many possibilities.
We wrote simple helper function (source:
https://gist.github.com/miku/7793932), which helps us to write code
like this:
...
def run(self):
# without explicit "output" - "tmp" will be some random temp file
# will raise a RuntimeError, if subprocess exits with non-zero exit code
output = shellout("xsltproc {stylesheet} {input} > {output}",
stylesheet="/path/to/stylesheet",
input=self.input().fn)
# atomically move file
luigi.File(output).move(self.output().fn)
Basically, shellout will return the {output} path, which you can the
use to move the file to final place atomically via File.move.
I'm curious about other approaches.
Martin.
[1] http://docs.python.org/2/library/subprocess.html
> You received this message because you are subscribed to the Google Groups
> "Luigi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to luigi-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
Imran Akbar
Dec 4, 2013, 3:42:51 PM12/4/13
to luigi...@googlegroups.com, Imran Akbar
Hey Martin,
yeah, that's what I ended up doing - subprocess.call worked fine inside the run method of a task.
thanks,
imran
0 new messages