How to make luigi use config file

[badgley]

I have a luigi.cfg file that has this inside it


[GlobalParamters]
exportpath: /foo/bar/baz


Then I have my luigi pipeline:

import luigi

class GlobalParamters(luigi.Task):
  exportpath = luigi.Parameter(default=None, is_global=True)

class FooTask(luigi.Task)
  exportpath = GlobalParameters.exportpath.value
  # ... rest of task

if __name__ == "__main__":
  luigi.run()


The issue here -- my luigi.cfg never gets read - and exportpath in FooTask is equal to None. I cannot find good examples outside the docs about how I am supposed to interact with my .cfg file -- my reading of it [http://luigi.readthedocs.org/en/latest/configuration.html] made me think that this should "just work". But do I have to explicitly parse the config?

Confused -- would love an example.

[Arash Rouhani]

Don't rely on GlobalParameters.exportpath.value. That will not be possible in future luigi versions (I think), instead do GlobalParameters().exportpath. Also extend luigi.Config rather than luigi.Task. :)

There was a bug also that might be related here[1]. I see that you use a somewhat  old version as of you still use is_global. Maybe you just need to bump your luigi version? :)

Cheers,

Arash

[1]: https://github.com/spotify/luigi/pull/958

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[badgley]

Woah. and now I know there is a luigi.Config!

Bumped my version of luigi and did some more reading -- think I should be good from here out, thought I still have many default_from_config to get rid of...

 Thanks

[dsta...@bandwidth.com]

Do I understand correctly that Global Configuration Variables are being deprecated?

I am rewriting our ETL process using Luigi and I would like to specify various output paths, filenames, database creds, etc. in a Globally accessible location. Rather than passing these variables through the pipelines to their needed classes, I'd like to be able to declare them in one location and call them from anywhere.

Is there any Luigi-compliant way of doing this?

Thanks

[Arash Rouhani]

The is_global keyword is being deprecated for Parameters. But all old sensible use cases are still doable just in a slightly different way. It have actually been available since February this year. For example see luigi.Config and core()[1].

[1]: https://github.com/spotify/luigi/blob/85ce122ee8a679b062ce24230e70c1cf3f631a6f/luigi/interface.py#L62

[Arash Rouhani]

I just sent a PR for review, with this commit[1] explaining a bit more what I mean and how the API can be expected to change.

[1]: https://github.com/Tarrasch/luigi/commit/d8e8c5a20f43f3c54ef8522566ab6312d1ce9e74