Calling the source command from subprocess.popen to update the os.environ.

[Hongyi Zhao]

Hi all,

Based on the methods here:

http://stackoverflow.com/questions/7040592/calling-the-source-command-
from-subprocess-popen/18897007#comment30954741_12708396

I use the following code the update the os.environ:

import os
from subprocess import check_output

# POSIX: name shall not contain '=', value doesn't contain '\0'
output = check_output("source /home/werner/env-intel-toolchains.sh;
env -0", shell=True, executable="/bin/bash")

# replace env
os.environ.clear()
os.environ.update(line.partition('=')[::2] for line in output.split('\0'))

In the above code, the file /home/werner/env-intel-toolchains.sh will be
sourced and its settings should be combined into the current os.environ
of python.

But, I find that the above code will fail on the `os.environ.clear()'
command as follows:

In[24]: import os
from subprocess import check_output
# POSIX: name shall not contain '=', value doesn't contain '\0'
output = check_output("source /home/werner/env-intel-toolchains.sh; env
-0", shell=True,
executable="/bin/bash")
# replace env
os.environ.clear()
os.environ.update(line.partition('=')[::2] for line in output.split('\0'))

Traceback (most recent call last):
File "/home/werner/anaconda2/lib/python2.7/site-packages/IPython/core/
interactiveshell.py", line 3066, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-24-816c472b62f2>", line 10, in <module>
os.environ.clear()
File "/home/werner/anaconda2/lib/python2.7/os.py", line 501, in clear
unsetenv(key)
OSError: [Errno 22] Invalid argument

If I don't use the `os.environ.clear()' code line, it will work smoothly.

Any hints?

Regards
--
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

[Ben Finney]

Hongyi Zhao <hongy...@gmail.com> writes:

> I use the following code the update the os.environ:
>
> import os
> from subprocess import check_output
>
> # POSIX: name shall not contain '=', value doesn't contain '\0'
> output = check_output("source /home/werner/env-intel-toolchains.sh;
> env -0", shell=True, executable="/bin/bash")

That will start a new process (running ‘/bin/bash’), then execute some
commands from a script file in that process.

When that new process ends, any changes in its environment also
disappear.

At no point do changes to that new process's environment have any effect
on the Python process.

A Unix process is *completely unable* to change the environment
variables of its parent. It can change its own variables, and optionally
those of its child processes.

This is by design. It's a good thing.

So if you want the Python process's environment to change, then it needs
to be:

* Inherited from an *already* changed environment when it starts.

Or:

* Changed within the Python process, by the Python API for that purpose
(‘os.environ’).

You will not be able to change a Python process environment by starting
new processes.

--
\ “Faith, n. Belief without evidence in what is told by one who |
`\ speaks without knowledge, of things without parallel.” —Ambrose |
_o__) Bierce, _The Devil's Dictionary_, 1906 |
Ben Finney

[Oscar Benjamin]

On 27 Mar 2016 17:01, "Ben Finney" <ben+p...@benfinney.id.au> wrote:

>
> Hongyi Zhao <hongy...@gmail.com> writes:
>
> > I use the following code the update the os.environ:
> >
> > import os
> > from subprocess import check_output
> >
> > # POSIX: name shall not contain '=', value doesn't contain '\0'
> > output = check_output("source /home/werner/env-intel-toolchains.sh;
> > env -0", shell=True, executable="/bin/bash")
>

> That will start a new process (running ‘/bin/bash’), then execute some
> commands from a script file in that process.
>
> When that new process ends, any changes in its environment also
> disappear.

...unless the subprocess prints them out or something.

I think you skimmed the code and question a little too quickly.

[Chris Angelico]

On Mon, Mar 28, 2016 at 12:24 AM, Hongyi Zhao <hongy...@gmail.com> wrote:
> # replace env
> os.environ.clear()
> os.environ.update(line.partition('=')[::2] for line in output.split('\0'))
>
> Traceback (most recent call last):
> File "/home/werner/anaconda2/lib/python2.7/site-packages/IPython/core/
> interactiveshell.py", line 3066, in run_code
> exec(code_obj, self.user_global_ns, self.user_ns)
> File "<ipython-input-24-816c472b62f2>", line 10, in <module>
> os.environ.clear()
> File "/home/werner/anaconda2/lib/python2.7/os.py", line 501, in clear
> unsetenv(key)
> OSError: [Errno 22] Invalid argument
>
> If I don't use the `os.environ.clear()' code line, it will work smoothly.

Do you need to clear the environment first? Anything that's been
overwritten will replace stuff in os.environ, so the only reason to
clear it would be if you expect your env script to unset things. Is
that the case?

I'm not sure why the clear is a problem. Here's how I'd start debugging it:

for var in os.environ:
try: del os.environ[var]
except OSError: print(var)

Two possibilities: Either that raises OSError errno 22, same as
clear() does; or it doesn't. If it doesn't, you should have a clear
env, and all should be working (if a little odd) - and then you can
look into the possible bug with clear(). But more likely it will, and
then you'll be able to see exactly which key triggered the error.
Check out what that key is; maybe there's some kind of special
environment variable that can't be cleared??

Worst case, you can always do this "clear whatever you can", and then
do your environ.update(). Assuming you never need to clear one of the
unclearables, this'll give you the same env update as sourcing the
script did.

And worst worst case, you could probably exec to bash, source the
script, and exec back into Python. That should properly update the
environment, but it's a somewhat hamfisted way to go about it :)

ChrisA

[Hongyi Zhao]

On Sun, 27 Mar 2016 13:24:05 +0000, Hongyi Zhao wrote:

> # replace env
> os.environ.clear()

I find another method which can solve this issue, ie., changing the above
code into the follows:

# ref : http://unix.stackexchange.com/questions/178522/unsetting-
environment-variable-with-an-empty-name
if "" in os.environ.data: del os.environ.data[""]
os.environ.clear()