parse an environment file

[Jason Friedman]

$ crontab -l
* * * * * env

This produces mail with the following contents:

HOME=/home/spjsf
LOGNAME=spjsf
PATH=/usr/bin:/bin
PWD=/home/spjsf
SHELL=/bin/sh
SHLVL=1
USER=spjsf
_=/usr/bin/env

On the other hand

$ env

produces about 100 entries, most of which are provided by my .bashrc;
cron provides only a limited number of environment variables.

I want my python 3.2.2 script, called via cron, to know what those
additional variables are. How?

[Dave Angel]

On 09/30/2012 08:11 PM, Jason Friedman wrote:

First comment: I don't know anything about an "environment file." An
environment is an attribute of a process, and it's inherited by
subprocesses that process launches. This is not a Python thing, it's an
OS thing, for whatever OS you're running.

> $ crontab -l
> * * * * * env
>
> This produces mail with the following contents:
>
> HOME=/home/spjsf
> LOGNAME=spjsf
> PATH=/usr/bin:/bin
> PWD=/home/spjsf
> SHELL=/bin/sh
> SHLVL=1
> USER=spjsf
> _=/usr/bin/env

In other words, these are the environment variables that cron has, and
that it gives to any children it launches.

> On the other hand
>
> $ env
>
> produces about 100 entries, most of which are provided by my .bashrc;
> cron provides only a limited number of environment variables.

Gee, those are variables specific to a particular (bash) shell, so how
would a program launched by cron know about them?

> I want my python 3.2.2 script, called via cron, to know what those
> additional variables are. How?

You'd have to change the environment that cron sees (before login), or
add to the crontab, or /etc/default/cron, or else run some shell from
cron that defines those variables and then runs the application. The
way to do this is specific to whatever OS you're running, and is
irrelevant to Python.

Try googling. For some examples, none of which use Python:

http://www.perlmonks.org/?node_id=624540
http://www.unix.com/solaris/164483-environment-variables-cron.html
http://www.computing.net/answers/unix/how-to-set-environment-variable/5689.html
http://linuxshellaccount.blogspot.com/2007/10/crontab-and-your-environment.html

--

DaveA

[Steven D'Aprano]

On Sun, 30 Sep 2012 18:11:09 -0600, Jason Friedman wrote:

> $ crontab -l
> * * * * * env
>
> This produces mail with the following contents:

[snip]

Yes, env returns the environment variables of the current environment.

> On the other hand
>
> $ env
>
> produces about 100 entries, most of which are provided by my .bashrc;
> cron provides only a limited number of environment variables.

That's because it's a different environment.

> I want my python 3.2.2 script, called via cron, to know what those
> additional variables are. How?

In general, you can't, because they may not even exist when your script
runs. There's no guarantee that "your environment" (which one? you might
have many, or none) exists at the time, and env certainly cannot guess
which one that might be.

But specifically, if you know the ID of a process, you may be able to see
that process' environment variables by reading the /proc/<PID>/environ
virtual file, which is a NULL-delimited list of environment variables.

As usual, permissions apply. In general, you can read your own processes,
but not those of other users.



--
Steven

[Chris Angelico]

On Mon, Oct 1, 2012 at 10:11 AM, Jason Friedman <ja...@powerpull.net> wrote:
> $ env
>
> produces about 100 entries, most of which are provided by my .bashrc;
> cron provides only a limited number of environment variables.
>

> I want my python 3.2.2 script, called via cron, to know what those
> additional variables are. How?

Looking into my crystal ball, I'm wondering if perhaps what you're
asking is for your cron job to have environment variables that aren't
set by cron, ones that you can see in your bash environment. This is a
common issue (usually with $PATH), and by no means Python-specific. A
quick web search will bring up some results, for instance:

http://www.google.com/search?q=cron+environment+variables
http://duckduckgo.com/?q=cron+environment+variables
http://www.bing.com/search?q=cron+environment+variables

If this isn't what you're asking about, please consider clarifying
your question :)

Chris Angelico

[Ulrich Eckhardt]

Am 01.10.2012 02:11, schrieb Jason Friedman:
> $ crontab -l
> * * * * * env
>
> This produces mail with the following contents:
>

[...]
> SHELL=/bin/sh
^^^^^^^
[...]

>
> On the other hand
>
> $ env
>
> produces about 100 entries, most of which are provided by my .bashrc;

bash != sh

Instead of running a script in default POSIX shell, you might be able to
run it in bash, which will then read your ~/.bashrc (verify that from
the docs, I'm not 100% sure). Maybe it is as easy as changing the first
line to '#!/bin/bash'.

> I want my python 3.2.2 script, called via cron, to know what those
> additional variables are.

To be honest, I would reconsider the approach. You could patch the cron
invokation, but that still won't fix any other invokations like starting
it from a non-bash shell, filemanager, atd etc. You could instead set
these variables in a different place that is considered by more
applications. I wonder if maybe ~/.profile would be such a place.

Alternatively, assuming these environment variables are just for your
Python program, you could store these settings in a separate
configuration file instead. Environment variables are always a bit like
using globals instead of function parameters.


Good luck!

Uli

[Alain Ketterlin]

Jason Friedman <ja...@powerpull.net> writes:

[...]

> I want my python 3.2.2 script, called via cron, to know what those
> additional variables are. How?

This is not a python question. Have a look at the crontab(5) man page,
it's all explained there.

-- Alain.

[Jason Friedman]

> I want my python 3.2.2 script, called via cron, to know what those
> additional variables are. How?

Thank you for the feedback. A crontab line of

* * * * * . /path/to/export_file && /path/to/script.py

does indeed work, but for various reasons this approach will not
always be available to me.

Let me restate my question. I have a file that looks like this:
export VAR1=foo
export VAR2=bar
# Comment
export VAR3=${VAR1}${VAR2}

I want this:
my_dict = {'VAR1': 'foo', 'VAR2': 'bar', 'VAR3': 'foobar'}

I can roll my own, but I'm thinking there is a module or existing code
that does this. I looked at the os and sys and configparse modules
but did not see it.

[Chris Angelico]

On Tue, Oct 2, 2012 at 12:12 AM, Jason Friedman <ja...@powerpull.net> wrote:
> Let me restate my question. I have a file that looks like this:
> export VAR1=foo
> export VAR2=bar
> # Comment
> export VAR3=${VAR1}${VAR2}
>
> I want this:
> my_dict = {'VAR1': 'foo', 'VAR2': 'bar', 'VAR3': 'foobar'}
>
> I can roll my own, but I'm thinking there is a module or existing code
> that does this. I looked at the os and sys and configparse modules
> but did not see it.

Is there a reason to use that format, rather than using Python
notation? I've at times made config files that simply get imported.
Instead of a dictionary, you'd have a module object:


# config.py
VAR1='foo'
VAR2='bar'
VAR3=VAR1+VAR2

# main file
import config as my_dict

ChrisA

[Chris Angelico]

On Tue, Oct 2, 2012 at 12:37 AM, Jason Friedman <ja...@powerpull.net> wrote:
>> Is there a reason to use that format, rather than using Python
>> notation? I've at times made config files that simply get imported.
>> Instead of a dictionary, you'd have a module object:
>>
>>
>> # config.py
>> VAR1='foo'
>> VAR2='bar'
>> VAR3=VAR1+VAR2
>>

> There is a reason: /path/to/export_file exists for Bash scripts, too,
> and I do not think I could get Bash to read config.py in the format
> stated above. I want to maintain only one file.

(Responding on-list and hoping it was merely oversight that had that
email come to me personally)

Ah, fair enough. Well, since you're using the full range of bash
functionality, the only viable way to parse it is with bash itself.
I'd recommend going with the version you have above:

> * * * * * . /path/to/export_file && /path/to/script.py

Under what circumstances is this not an option? That'd be the next
thing to consider.

Alternatively, you may want to consider making your own config file
format. If you consciously restrict yourself to a severe subset of
bash functionality, you could easily parse it in Python - for
instance, always look for "export %s=%s" with simple strings for the
variable name and value.

ChrisA

[88888 Dihedral]

I think one can ues some decorators to wrap OS or platform
dependent functions.

I am sure someone did that long time ago as the iron python
wrapped dot-net.

[88888 Dihedral]

On Monday, October 1, 2012 10:42:02 PM UTC+8, Chris Angelico wrote:

[Hans Mulder]

One tactic is to write a wrapper script in shellese that sets the
variables and then runs your script. Something like:

#/bin/bash

export VAR1=foo
export VAR2=bar
# Comment
export VAR3=${VAR1}${VAR2}

# Read some more settings from a file
. /path/to/file/with/more/exports

# Drum roll .....
/path/to/your/script.py


This allows you to copy-and-paste all sorts of weird and wonderful
shell syntax into your wrapper script.

AFAIK, there is no Python module that can read shell syntax.
You could translate all that shell syntax manually to Python,
but that may not be worth the effort.

Hope this helps,

-- HansM

[xDog Walker]

On Monday 2012 October 01 08:35, Hans Mulder wrote:
> AFAIK, there is no Python module that can read shell syntax.

The stdlib's shlex might be that module.

--
Yonder nor sorghum stenches shut ladle gulls stopper torque wet
strainers.

[Ramchandra Apte]

shlex can only split shell code into tokens.

[Ramchandra Apte]

On Tuesday, 2 October 2012 21:34:04 UTC+5:30, xDog Walker wrote:

[Jason Friedman]

> Ah, fair enough. Well, since you're using the full range of bash
> functionality, the only viable way to parse it is with bash itself.
> I'd recommend going with the version you have above:
>

>> * * * * * . /path/to/export_file && /path/to/script.py
>

> Under what circumstances is this not an option? That'd be the next
> thing to consider.
>
> Alternatively, you may want to consider making your own config file
> format. If you consciously restrict yourself to a severe subset of
> bash functionality, you could easily parse it in Python - for
> instance, always look for "export %s=%s" with simple strings for the
> variable name and value.
>

Thank you, Chris, off-list post unintentional. It may be the case
that I do not maintain /path/to/export_file; I might just be allowed
to read it.
Based on your responses and everyone's responses I'm guessing that
what I am doing is sufficiently novel that there is no canned
solution. I looked at shlex but did not see how that would be
helpful.
Thank you all for your thoughts.

[Chris Angelico]

On Wed, Oct 3, 2012 at 1:49 PM, Jason Friedman <ja...@powerpull.net> wrote:
> Based on your responses and everyone's responses I'm guessing that
> what I am doing is sufficiently novel that there is no canned
> solution. I looked at shlex but did not see how that would be
> helpful.

The only canned solution for parsing a bash script is bash. Think
about it the other way around: If you wanted to have a Python variable
made available to a bash script, the obvious thing to do is to invoke
Python. It's the same thing.

I recommend going with Hans Mulder's suggestion of a wrapper script;
that seems to be the cleanest option available.

ChrisA

[Jason Friedman]

> The only canned solution for parsing a bash script is bash. Think
> about it the other way around: If you wanted to have a Python variable
> made available to a bash script, the obvious thing to do is to invoke
> Python. It's the same thing.

I scratched my own itch:
http://code.activestate.com/recipes/578280-parse-profile/.

[Chris Angelico]

And there you have it! A subset of bash syntax and a Python parser. Job done!

ChrisA