Luigi executable + RangeDailyBase

[Jerome Gagnon]

Hi everyone,

I am just starting with Luigi and I am writing my first job with it, the job is working when passing a parameter date directly; but now I want to execute it with RangeDaily parameter and it doesnt seems to work. I am probably missing obvious, but what ? I think I would need something to bing the parameter date to my task.

 

This works:

python project/tasks/user_activity.py RedisDailyUserActivity --local-scheduler --date 2015-01-01

This doesn't work:

python project/tasks/user_activity.py RedisDailyUserActivity --local-scheduler --RangeDailyBase-start 2015-01-01 --RangeDailyBase-stop 2015-02-01

Console output:

luigi.parameter.MissingParameterException: No value for 'date' (--date) submitted and no default value has been assigned.

Thanks!

[Arash Rouhani]

Hi Jerome!

See this: http://luigi.readthedocs.org/en/latest/luigi_patterns.html#efficiently-triggering-recurring-tasks

It shows how to use Range :)

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jerome Gagnon]

I've actually seen that; It's just that this is using luigi cmd line and I seem to be unable to use it. Here's the error I get;

luigi --module user_activity RedisDailyUserActivity ...

ImportError: No module named user_activity

[Arash Rouhani]

Try

luigi --module user_activity RangeDaily --of RedisDailyUserActivity --start 2015-03-05 ...

You were obviously a bit confused by the documentation. Will you consider to clarify it? :)

[Jerome Gagnon]

So I figured out how to make this work with python command line :

python project/tasks/user_activity.py RangeDaily --of RedisDailyUserActivity --RangeDailyBase-start=2015-04-01 --RangeDailyBase-stop=2015-07-01 --local-scheduler

Turns out its pretty similar to luigi command; it's just not that well explained in the documentaiton.

Thanks

[Arash Rouhani]

No that is wrong. You should still use the `luigi` command line. Not `python`. Also, you should just specify --start not --RangeDailyBase-start.

Also Do *not* rely on --RangeDailyBase-start when you use RangeDaily (without Base). It *might* work now. But that's a bug in luigi which I'm actively trying to fix.

[Jerome Gagnon]

I'd like to use luigi command but for some reason it doesn't find my module; any pointer on why that is hapenning ?

Traceback (most recent call last):

  File "/usr/local/bin/luigi", line 9, in <module>

    load_entry_point('luigi==1.3.0', 'console_scripts', 'luigi')()

  File "/usr/local/lib/python2.7/site-packages/luigi/cmdline.py", line 13, in luigi_run

    luigi.interface.run(argv, use_dynamic_argparse=True)

  File "/usr/local/lib/python2.7/site-packages/luigi/interface.py", line 421, in run

    tasks = interface.parse(cmdline_args, main_task_cls=main_task_cls)

  File "/usr/local/lib/python2.7/site-packages/luigi/interface.py", line 329, in parse

    __import__(module)

ImportError: No module named user_activity

[Jerome Gagnon]

I also get a really weird bug when trying to schedule 

DEBUG: Checking if RangeDaily(of=RedisDailyUserActivity, reverse=False, task_limit=50, now=None, start=2015-01-01, stop=2015-02-01, days_back=100, days_forward=0) is complete

DEBUG: Empty range. No RedisDailyUserActivity instances expected

2015-01-01 to 2015-02-01 empty range, how's that possible ?

[Jerome Gagnon]

Figured out that thing... days_back had precedence over the start and stop dates; so its like (BETWEEN 2015-01-01 and 2015-03-01) AND days_back < 100 which explain why it returns an empty set.

[Benjamin Bertrand]

Le mardi 14 juillet 2015 19:38:31 UTC+2, Jerome Gagnon a écrit :

I'd like to use luigi command but for some reason it doesn't find my module; any pointer on why that is hapenning ?

Try to export your PYTHONPATH with the path of your module.

[Jerome Gagnon]

That did it, thank you!