Command shell access within Python

[pie...@attglobal.net]

I am new to Python and i want to execute several shell commands within a Python function. Is there an easy way to execute several
shell command lines and get the return code of each of them using a Python library call?

Things like os.execv(), os.system() and os.startfile() allow me to execute one command at a time, starting one process for every
statement. I also played with the os.popen(), os.popen2(),...

These allow to get a pipe to a command shell and issue commands and read the results via stdin, stdout and stderr.

But there must be a better way so I can use Python to execute command line programs programmatically using Python the way you would
use a command line script.

I also need to get the return code (c exit()) of the program that ran.

I do not want to start one command shell process for every single command line .

Anyone knows how to do it?

Thanks

Pierre R.

[Stefan Antoni]

On Sun, Nov 04, 2001 at 01:02:46PM -0500, pie...@attglobal.net wrote:

> I also need to get the return code (c exit()) of the program that ran.

os.system() gives you the return code of your command.
try interactively:

import os

print os.system('ls')

... listing ...
0 <-- return code

--
Stefan Antoni

[Pierre Rouleau]

Thanks for taking the time to answer Stefan.

> > I also need to get the return code (c exit()) of the program that ran.
> os.system() gives you the return code of your command.
> try interactively:
>
> import os
>
> print os.system('ls')
>
> ... listing ...
> 0 <-- return code

Actually, i knew that i can execute a single program inside a command shell and get the return code.
What i want is to execute a large set of commands inside the same shell process and being able to get the return code for exach one
of the calls.

For example, i would like to be able to do someting like:

myShell = Process('cmd')
error = myShell.execute('make myproject')
if (error == 0) :
myShell.execute('ls')
myShell.execute('pj ci makefile')

I want to be able to issue a whole list of line commands whithout having to create a new process on every one of them.

Pierre

[Stuart D. Gathman]

In article <3BE5A1A7...@attglobal.net>, "Pierre Rouleau"
<pie...@attglobal.net> wrote:

> Actually, i knew that i can execute a single program inside a command
> shell and get the return code. What i want is to execute a large set of
> commands inside the same shell process and being able to get the return
> code for exach one of the calls.
>
> For example, i would like to be able to do someting like:
>
> myShell = Process('cmd')
> error = myShell.execute('make myproject') if (error == 0) :
> myShell.execute('ls')
> myShell.execute('pj ci makefile')
>
>
> I want to be able to issue a whole list of line commands whithout having
> to create a new process on every one of them.

When you run commands in a traditional shell script, it creates a new process
for every one of them!

The equivalent of the above is:

#!/usr/bin/python
import os
error = os.system('make myproject')
if error == 0:
os.system('ls')
os.system('pj ci makefile')

The equivalent bourne shell would be:

#!/bin/sh
make myproject
if [ "$?" == 0]; then
ls
pj ci makefile
fi

The number of 'fork' and 'exec' calls is exactly the same for both script
languages. It is very convenient to run subprocesses in bourne shell,
however bourne quickly becomes impenetrable when there is a lot of
internal processing. While python syntax is somewhat clumsier than bourne
for running external programs, internal processing is quite elegant.

In both cases, a new process is created for each command.

--
Stuart D. Gathman <stu...@bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flamis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

[Janko Hauser]

"Stuart D. Gathman" <stu...@bmsi.com> writes:

> In article <3BE5A1A7...@attglobal.net>, "Pierre Rouleau"
> <pie...@attglobal.net> wrote:
>
> > Actually, i knew that i can execute a single program inside a command
> > shell and get the return code. What i want is to execute a large set of
> > commands inside the same shell process and being able to get the return
> > code for exach one of the calls.
> >
> > For example, i would like to be able to do someting like:
> >
> > myShell = Process('cmd')
> > error = myShell.execute('make myproject') if (error == 0) :
> > myShell.execute('ls')
> > myShell.execute('pj ci makefile')
> >
> >
> > I want to be able to issue a whole list of line commands whithout having
> > to create a new process on every one of them.
>
> When you run commands in a traditional shell script, it creates a new process
> for every one of them!
>
> The equivalent of the above is:
>
> #!/usr/bin/python
> import os
> error = os.system('make myproject')
> if error == 0:
> os.system('ls')
> os.system('pj ci makefile')
>

> ...

>
> In both cases, a new process is created for each command.
>

AFAIK this is not right, as for the os.system() call first a shell is
started and then the process (ls) is started in this shell.

A better way is to use the commands module. It wraps os.popen and I'm
not sure if this also didn't open a shell beforehand.

Pure ways to start a new process are wrapped in the os module as
os.exe?? os.fork and os.spawn??. But using these is more complicated
(io handling, finding the path of the executable etc.)

HTH,
__Janko

--
Institut fuer Meereskunde phone: 49-431-597 3989
Dept. Theoretical Oceanography fax : 49-431-565876
Duesternbrooker Weg 20 email: jha...@ifm.uni-kiel.de
24105 Kiel, Germany

[Stuart D. Gathman]

In article <87ady25...@lisboa.ifm.uni-kiel.de>, "Janko Hauser"
<jha...@ifm.uni-kiel.de> wrote:

> AFAIK this is not right, as for the os.system() call first a shell is
> started and then the process (ls) is started in this shell.

Oops, you are right. I am a long way from a python guru...

> A better way is to use the commands module. It wraps os.popen and I'm
> not sure if this also didn't open a shell beforehand.

I checked, and the "commands" module also creates a shell process to
parse the command.

> Pure ways to start a new process are wrapped in the os module as
> os.exe?? os.fork and os.spawn??. But using these is more complicated (io
> handling, finding the path of the executable etc.)

It looks like os.spawn* is the way to go:

import os

def cmd(s):
"I do not handle shell style I/O redirections or substitutions"
sv = s.split()
return os.spawnvp(os.P_WAIT,sv[0],sv)

error = cmd('make myproject')
if error == 0:
cmd('ls')
cmd('pj ci makefile')

[Pierre Rouleau]

Janko Hauser wrote:

>
> A better way is to use the commands module. It wraps os.popen and I'm
> not sure if this also didn't open a shell beforehand.
>

Do you know if the commands module is supported under Win32 (NT) by python 2.1?
I tried it and it does not seem to work:

>>> import commands
>>> dir(commands)
['__all__', '__builtins__', '__doc__', '__file__', '__name__', 'getoutput', 'getstatus', 'getstatusoutput', 'mk2arg', 'mkarg']
>>> (exitstatus, outtext) = commands.getstatusoutput('ls')
>>> exitstatus
1
>>> outtext
'The name specified is not recognized as an\ninternal or external command, operable program or batch file.'

Pierre

[Pierre Rouleau]

Janko Hauser wrote:

>
> A better way is to use the commands module. It wraps os.popen and I'm
> not sure if this also didn't open a shell beforehand.

Do you know if the commands module is supported under Win32 (NT) by python 2.1? Or
2.2?
I tried it and without changing the source of commands.py this is the result i get
(under NT):

>>> import commands
>>> dir(commands)
['__all__', '__builtins__', '__doc__', '__file__', '__name__', 'getoutput',
'getstatus', 'getstatusoutput', 'mk2arg', 'mkarg']
>>> (exitstatus, outtext) = commands.getstatusoutput('ls')
>>> exitstatus
1
>>> outtext
'The name specified is not recognized as an\ninternal or external command, operable
program or batch file.'

Now, I looked at the commands.py and getstatusoutput() is:

def getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
import os
pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
pipe = os.popen(cmd + ' 2>&1', 'r')
text = pipe.read()
sts = pipe.close()
if sts is None: sts = 0
if text[-1:] == '\n': text = text[:-1]
return sts, text

The {ls;} does not work in the native NT command shell.
So if i modify the source to the following the problem is solved:

def getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
import os
if os.name in ['nt', 'dos', 'os2'] :
# use Dos style command shell for NT, DOS and OS/2
pipe = os.popen(cmd + ' 2>&1', 'r')
else :
# use Unix style for all others
pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
text = pipe.read()
sts = pipe.close()
if sts is None: sts = 0
if text[-1:] == '\n': text = text[:-1]
return sts, text

Now it works fine under NT:

>>> reload(commands)
<module 'commands' from 'c:\python21\lib\commands.py'>
>>> (stat, text) = commands.getstatusoutput('ls')
>>> stat
0
>>> print text
calc
freeware
mkssi
mkssi.py
mkssi.pyc
si_localChekpoint.py
testname.py
testname.pyc
testtk.py
testtk.pyc
>>>

So, in the end, it looks like commands could be used under Win32 and DOS-like
command shells.

The docstring in commands.py states that it is for Unix only. Is it worthed to add
Win32 compatibility so that modules using commands.py can run in Win32 as well?

Pierre

[Janko Hauser]

I do not know, why the commands module is only specified or Unix. I
assume there were problems with the popen function on windows in older
releases of Python.

As is seems your change makes this working on windows I would submit
it as a patch on the sourceforge site. But as noted in this thread,
popen also starts a shell for each command, so it doesn't really solve
your problem, if this is actually a problem :-).

[Pierre Rouleau]

> But as noted in this thread,
> popen also starts a shell for each command, so it doesn't really solve
> your problem, if this is actually a problem :-).

It's not that it is a problem. But I was trying to create an Python
object that I would be able to use where I would change directories,
change environment variables values (or add/delete environment variables).

Using os.system(), os.popenXX() and the like the environment of these
sub-processes are lost after every invocation.
I guess my other bet would be to re-direct stdin/stdout/stderr to a shell
process, but then I would not have access to the returned shelled program
value.

When i first tried os.system() i tried it within the IDLE system in NT.
Under IDLE in NT, every time your call os.system(), a shell window pops up
briefly while the command runs. I was trying to avoid that. Later, i
noticed that if i run python from a shell directly, os.system() does not
behave like that (no other window pops up).

Thanks for the help!

Pierre