Utilizing Luigi.cfg

[dlsta...@gmail.com]

We are migrating our ETL processes to Luigi and would really like to utilize the value of Luigi's configuration file; however, we are unable to get it to function.

We have created a luigi.cfg of the form (including some of our own sections):
_________________________________________
[core]
error-email: <email>
email-prefix: [Luigi]
...

[scheduler]
retry-delay: 60
...

[redshift]
host:
database:
user:
password:

[s3]
aws_access_key_id:
aws_access_key:
bucket:

[other]
...
__________________________________________

Parameters have been specified:

aws_key = luigi.Parameter(config_path=dict(section='s3', name='aws_access_key_id')

However, no such luck. Error emails don't even send.

Can anyone provide pointers or (preferably) example code?

Thank you all in advance! (If you need further details from me, just ask).

[Miguel Dickson]

Luigi is usually looking for the file to be called 'client.cfg'. Try that?

[Arash Rouhani]

Oh yea, only Luigi HEAD uses luigi.cfg. Latest pypi release only recognizes client.cfg.

*This communication (including any attachments) is for the use of the intended recipient(s) only and may contain information that is confidential, privileged or otherwise legally protected. Any unauthorized use or dissemination of this communication is prohibited. If you have received this communication in error, please immediately notify the sender by return e-mail message and delete all copies of the original communication. Thank you for your cooperation.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dlsta...@gmail.com]

Thanks guys, remaining to client.cfg and using
from luigi import configuration
...
configuration.get_config().get(<section>,<var>)
works great.

Is there a best practice to using configuration variables within many classes within the same file?

For example, a file includes all classes which involve BazaarVoice and I need to access the config variable for a local path. Is it best to make reference to the config variable in the parameter list for every class? Or is it best to reference the variable as a global variable for the entire file?

Thanks

[Arash Rouhani]

Yea, we kind of do that here: https://github.com/spotify/luigi/blob/0fb4ff6a53ff449c800d7f9bc2e8e2baef3b6f8b/luigi/contrib/hdfs/config.py#L37-L46

On Sun, Sep 6, 2015 at 12:42 AM, <dlsta...@gmail.com> wrote:

On Thursday, September 3, 2015 at 2:53:50 PM UTC-4, dlsta...@gmail.com wrote:

Thanks guys, remaining to client.cfg and using
   from luigi import configuration
   ...
   configuration.get_config().get(<section>,<var>)
works great.

Is there a best practice to using configuration variables within many classes within the same file?

For example, a file includes all classes which involve BazaarVoice and I need to access the config variable for a local path. Is it best to make reference to the config variable in the parameter list for every class? Or is it best to reference the variable as a global variable for the entire file?

Thanks

[dlsta...@gmail.com]

To note, Luigi documentation notes that 'client.cfg' is the legacy name and that the updated name should be 'luigi.cfg' (seen http://luigi.readthedocs.org/en/latest/configuration.html). However, since this isn't the case, shouldn't the documentation be changed in order to reflect the actual usage?

Thanks for everyone's help correcting me!

[Arash Rouhani]

Nah, we're still holding off for a 2.0 release for a moment. but if you want the 1.3.1 docs, you have them too http://luigi.readthedocs.org/en/stable/configuration.html. Basically replace "latest" with "stable".

[dlsta...@gmail.com]

Oh, I didn't realize that! Thanks!