ImportError: No module named <module_name> on everything

[dyla...@dowjones.com]

Whenever I try to run any example for luigi after creating a virtual environment in python 2.7 and installing luigi right away either with source or pip I get the message below.

luigi --module top_artists AggregateArtists --local-scheduler --date-interval 2012-06
Traceback (most recent call last):
File "/usr/local/bin/luigi", line 11, in <module>
sys.exit(luigi_run())
File "/Library/Python/2.7/site-packages/luigi/cmdline.py", line 13, in luigi_run
luigi.interface.run(argv, use_dynamic_argparse=True)
File "/Library/Python/2.7/site-packages/luigi/interface.py", line 421, in run
tasks = interface.parse(cmdline_args, main_task_cls=main_task_cls)
File "/Library/Python/2.7/site-packages/luigi/interface.py", line 329, in parse
__import__(module)
ImportError: No module named top_artists

Has anyone gotten this before or know why luigi is doing this for everything I try to run with i?

[Uldis Barbans]

Looks like the example should have said

PYTHONPATH=. luigi --module top_artists AggregateArtists --local-scheduler --date-interval 2012-06

all the time.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Uldis "ulzha" Barbans - Software Engineer, Data Infrastructure - Spotify AB

[dyla...@dowjones.com]

Thank you so much Uldis that did the trick! I do not remember having to do that before, but it could have been an environment thing. I have started using luigi for all of my pipelines and I didn't package all of them so this is very useful.

[Arash Rouhani]

I actually just checked. It seems like it works with an empty PYTHONPATH too (run with PYTHONPATH='' luigi --module ...).

https://docs.python.org/2/tutorial/modules.html#standard-modules

Maybe we should still explicitly specify PYTHONPATH=. in the docs, so people can't do it wrong. :)

Cheers,

Arash

[Thomas Yager-Madden]

Oh, I've been meaning to bring this up for a while; I came across this thread while poking around to see if there might be a better way. Tbh, although I am in the habit of setting my PYTHONPATH before job runs, I find it sort of awkward to have to do that so that luigi can recognize modules in the PWD. For code organization I also prefer to keep more abstract task superclasses in a different subdirectory from specific pipeline scripts, which makes my needed PYTHONPATH a bit more complicated than just "."

I don't hate this, but I'm just wondering if something could be done so that luigi command line tool could find my code paths more readily/automagically. I'm happy to work on a PR for something that might help accomplish this, but I don't know enough to know what approach to take or where to begin (I am a humble database administrator who finds working with Luigi agreeable, not really a python developer by trade or expertise).

Any thoughts, suggestions or help appreciated here.

 Thomas

[Dave Buchfuhrer]

If you run Luigi from a virtualenv you can avoid a lot of these issues. With virtualenvwrapper, you can add to the python path of the virtualenv with the command add2virtualenv. This also somewhat isolates your Luigi installation from your system Python and related environment variables, which gives greater stability for runs.

[Uldis Barbans]

Regarding setting PYTHONPATH right, that ought to be taken care of by one's shell rc file or configuration management system...

A related idea, support for configurable plug-in module loading (of, say, loggers or event handlers), sounds like something related to what you want (so do we). It would be great to use something semi-standard, yet I seem to only find rather old articles googling for things like "python plugin frameworks"...