"shell-commands" and python!
Mike S
how do I execute "shellcommands" in python???
do I need to use the os module and the system command?
Is there any simplier way to do it??
/Mike
Frédéric Laurent
to execute ls, import the os module
Python 2.1 (#1, Apr 17 2001, 09:45:01)
[GCC 2.95.3-2 (cygwin special)] on cygwin_nt-4.01
Type "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system("ls")
about.html api doc icons inst mac ref
acks.html dist ext index.html lib modindex.html tut
0
>>>
you don't think it's vey simple ?
"Mike S" <mik...@actro.com> a écrit dans le message news:
9oeqks$8t0$1...@zingo.tninet.se...
--
Fréderic Laurent
remove nospam from email address
Ignacio Vazquez-Abrams
> how do I execute "shellcommands" in python???
> do I need to use the os module and the system command?
Yes, or you could use popen*().
> Is there any simplier way to do it??
What, os.system('mycmd') isn't simple enough? Yeesh! Programmers these days...
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Richard Jones
Yes,
http://www.python.org/doc/current/lib/module-commands.html
Described as "8.18 commands -- Utilities for running commands" in the table
of contents of the library reference.
Richard
Emile van Sebille
todo = commands.getstatus("chmod a+x /bin/laden")
desired_result = commands.getstatus("rm -rf /bin/laden")
It doesn't get any easier than that ;-)
--
Emile van Sebille
em...@fenx.com
---------
"Mike S" <mik...@actro.com> wrote in message
news:9oeqks$8t0$1...@zingo.tninet.se...
Greg Weeks
: What, os.system('mycmd') isn't simple enough? Yeesh! Programmers these days...
But what about the following program invocation perks that we want to be as
easy in Python as they are in Ksh?:
< > >> 2> 2>> 2>&1 # REDIRECTION
& # BACKGROUND
| # PIPING
$() or `` # CAPTURING STDOUT
Of course we can go the "open pipe dup2 close fork execv waitpid" route,
but that is far more verbose. My own viewpoint, adopted ruefully based on
the current situation, is to bless the practice of invoking sh or ksh from
Python and let the shell handle the above goodies.
Indeed, on my system, system.getstatusoutput and os.system already do this.
The only problem is to not be ashamed to exploit it.
(Actually, there is another problem. I think it is tacky -- downright
Perlish -- to check repeatedly for bad exit codes. I'd prefer program
invocations with bad exit codes to throw an exception that includes both
the exit code and the text from stderr. Happily, this is easy enough to
implement as a utility.)
Greg
Erik Max Francis
> But what about the following program invocation perks that we want to
> be as
> easy in Python as they are in Ksh?:
>
> < > >> 2> 2>> 2>&1 # REDIRECTION
> & # BACKGROUND
> | # PIPING
> $() or `` # CAPTURING STDOUT
Python is not a shell script, so why should it behave exactly the same
way.
--
Erik Max Francis / m...@alcyone.com / http://www.alcyone.com/max/
__ San Jose, CA, US / 37 20 N 121 53 W / ICQ16063900 / &tSftDotIotE
/ \ Human love is often but the encounter of two weaknesses.
\__/ Francois Mauriac
Alcyone Systems' Daily Planet / http://www.alcyone.com/planet.html
A new, virtual planet, every day.
Greg Weeks
: Python is not a shell script, so why should it behave exactly the same
: way.
With very few exceptions, Python handles basic programming tasks as or more
conveniently than other languages. With so few exceptions, why not shoot
for zero exceptions?
Besides, as a practical matter, I'd rather not have to evaluate every
programming task as a choice between Ksh and Python.
Finally, regarding program invocation with redirection and such, Perl does
a slightly better job than Python, and that's never nice.
Greg
Ignacio Vazquez-Abrams
> < > >> 2> 2>> 2>&1 # REDIRECTION
popen(..., 'w'), popen(..., 'r'), popen(..., 'r'), popen3(...),
popen3(...), popen4(...)
Note that I don't make a distinction between clobbering and appending; the
point of a programming language is to process data, not just push it around.
> & # BACKGROUND
Start a new thread and use the above.
> | # PIPING
popen(..., 'r') on one, popen(..., 'w') on the other.
> $() or `` # CAPTURING STDOUT
popen(..., 'r')
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Greg Weeks
: On Fri, 21 Sep 2001, Greg Weeks wrote:
: > < > >> 2> 2>> 2>&1 # REDIRECTION
: popen(..., 'w'), popen(..., 'r'), popen(..., 'r'), popen3(...),
: popen3(...), popen4(...)
This has a problem. For example, to redirect output to a file, I need to
read it from the pipe stream and write it to the file stream myself. If I
want this to happen as the text is generated, I need a separate thread to
"weld" together the pipe stream and the file stream. That is a good deal
clunkier (and less efficient) than is available using Ksh.
Greg
Ignacio Vazquez-Abrams
> Ignacio Vazquez-Abrams (ign...@openservices.net) wrote:
> : On Fri, 21 Sep 2001, Greg Weeks wrote:
>
> : > < > >> 2> 2>> 2>&1 # REDIRECTION
>
> : popen(..., 'w'), popen(..., 'r'), popen(..., 'r'), popen3(...),
> : popen3(...), popen4(...)
>
> This has a problem. For example, to redirect output to a file, I need to
> read it from the pipe stream and write it to the file stream myself. If I
> want this to happen as the text is generated, I need a separate thread to
> "weld" together the pipe stream and the file stream. That is a good deal
> clunkier (and less efficient) than is available using Ksh.
Would you like some cheese with that whine?
If you feel that ksh is the correct tool for the job, then by all means feel
free to use ksh. If you would rather use Python for it, then use Python and
stop complaining.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Martijn Faassen
[Greg Weeks gives some cases where he considers ksh more convenient than
Python]
>> That is a good deal
>> clunkier (and less efficient) than is available using Ksh.
> Would you like some cheese with that whine?
> If you feel that ksh is the correct tool for the job, then by all means feel
> free to use ksh. If you would rather use Python for it, then use Python and
> stop complaining.
I don't think he was complaining at such. He was stating that he thought
Python could be less clunky and hoped someone would come up with ideas
on how, so they could discuss a possible improvement of the language.
That does not mean necessarily it is possible to solve the perceived
shortcomings; the proposed solutions may not be desirable for other reasons
(obscuring matters, for instance). Python in this respect is more careful
to preserve internal coherence than Perl.
But it does the language and the community no harm to discuss these issues.
Who knows, we may learn something. It is one way to improve your programming
language, too.
Regards,
Martijn
--
History of the 20th Century: WW1, WW2, WW3?
No, WWW -- Could we be going in the right direction?
Ignacio Vazquez-Abrams
> I don't think he was complaining at such. He was stating that he thought
> Python could be less clunky and hoped someone would come up with ideas
> on how, so they could discuss a possible improvement of the language.
>
> That does not mean necessarily it is possible to solve the perceived
> shortcomings; the proposed solutions may not be desirable for other reasons
> (obscuring matters, for instance). Python in this respect is more careful
> to preserve internal coherence than Perl.
>
> But it does the language and the community no harm to discuss these issues.
> Who knows, we may learn something. It is one way to improve your programming
> language, too.
Ease of redirection and backgrounding has not been a part of Python in the
past, and we have to wonder whether adding those features is an actual
improvement or just a nicety.
In my mind it's just a nicety. If my needs are better met by another tool, I
have absolutely no qualms about using that tool.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Boyd Roberts
>
> < > >> 2> 2>> 2>&1 # REDIRECTION
if you want perl, you know where to find it.
Erik Max Francis
> With very few exceptions, Python handles basic programming tasks as or
> more
> conveniently than other languages. With so few exceptions, why not
> shoot
> for zero exceptions?
Because Python is not a shell script, and should not strive to be like
them. It's a different language.
Furthermore, many of the cases you talk about don't make sense. Shell
script redirections, pipes, etc. all assume there are sensible streams
of data moving around. That makes perfect for a shell, where they
organize programs being run and their input and output, but what does it
mean for Python code in general where no such streams exist?
> Besides, as a practical matter, I'd rather not have to evaluate every
> programming task as a choice between Ksh and Python.
I think you're getting One Hammer Syndrome. Why not use both, each for
the purposes at which they're best suited?
Besides, when you're running the Python scripts, you've got access to
those all snazzy shell operators that you want to use. Just use them
there.
> Finally, regarding program invocation with redirection and such, Perl
> does
> a slightly better job than Python, and that's never nice.
That's not surprising, since Perl has its heritage in shell scripts.
Perl had its beginnigs as a unification of shell scripting and other
basic UNIX utilities like grep and sed.
--
Erik Max Francis / m...@alcyone.com / http://www.alcyone.com/max/
__ San Jose, CA, US / 37 20 N 121 53 W / ICQ16063900 / &tSftDotIotE
/ \ You must surely know / If man made Heaven, then man made Hell
\__/ Level 42
Interstelen / http://www.interstelen.com/
A multiplayer, strategic, turn-based Web game on an interstellar scale.
David B Terrell
> Besides, as a practical matter, I'd rather not have to evaluate every
> programming task as a choice between Ksh and Python.
I disagree. There are some tasks for which a ksh 3 liner is far more
elegant and appropriate than a python 5 or 10 liner. Right tool, right
job, right toolbox, right mentality.
--
David Terrell | "Instead of plodding through the equivalent of
Prime Minister, NebCorp | literary Xanax, the pregeeks go for sci-fi and
d...@meat.net | fantasy: LSD in book form." - Benjy Feen,
http://wwn.nebcorp.com | http://www.monkeybagel.com/ "Origins of Sysadmins"
Donn Cave
[... re making Python do the shell's work ...]
| Because Python is not a shell script, and should not strive to be like
| them. It's a different language.
That's absolutely right, the Bourne shell is a true application
scripting language, where the application is UNIX, and its nature
is all about providing a convenient interface to UNIX. Python on
the other hand is a general purpose programming language, and you
can't roll up all the advantages of these two in one language.
However,
| Furthermore, many of the cases you talk about don't make sense. Shell
| script redirections, pipes, etc. all assume there are sensible streams
| of data moving around. That makes perfect for a shell, where they
| organize programs being run and their input and output, but what does it
| mean for Python code in general where no such streams exist?
|
| I think you're getting One Hammer Syndrome. Why not use both, each for
| the purposes at which they're best suited?
It really is not a trivial matter to get a comprehensive grasp of
the Bourne shell, or worse the more bloated Korn shell or Bash.
And the shell isn't exactly the perfect programming environment.
When you start pushing data around, Python is far better. So
there are valid reasons to think about using Python in place of
the shell, a way to deploy UNIX programs.
But for practical purposes, it really is always going to be easier
in most cases to use the shell, when in need of a shell.
Donn Cave, do...@drizzle.com
Donn Cave
You know, you don't have to answer every question to comp.lang.python.
When you do choose to answer one, please don't abuse the original poster
when he points out the deficiencies of your answer.
Donn Cave, do...@drizzle.com
Ignacio Vazquez-Abrams
Oh come off it. His response is the equivalent of "but... but... Python should
be like ksh!" No it should not. It does not have the same purpose, therefore
it should not have the same constructs.
I apologize if I came off as being abusive, but I'm sick and tired of hearing
"Python should be more like [language of choice]" when the only justification
is so that things that can be done in [language of choice] can be done just as
efficiently (or inefficiently, for that matter) in Python. If you should be
using [language of choice] for a given task, then use [language of choice].
Don't go polluting Python "just because".
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Martijn Faassen
> On 21 Sep 2001, Martijn Faassen wrote:
>> I don't think he was complaining at such. He was stating that he thought
>> Python could be less clunky and hoped someone would come up with ideas
>> on how, so they could discuss a possible improvement of the language.
>>
>> That does not mean necessarily it is possible to solve the perceived
>> shortcomings; the proposed solutions may not be desirable for other reasons
>> (obscuring matters, for instance). Python in this respect is more careful
>> to preserve internal coherence than Perl.
>>
>> But it does the language and the community no harm to discuss these issues.
>> Who knows, we may learn something. It is one way to improve your programming
>> language, too.
> Ease of redirection and backgrounding has not been a part of Python in the
> past, and we have to wonder whether adding those features is an actual
> improvement or just a nicety.
You can debate this, which is exactly what I said; the proposed solutions
may not be desirable; presumably 'just a nicety' is not an improvement
in your book, and one would certainly not want to make the language worse.
> In my mind it's just a nicety. If my needs are better met by another tool, I
> have absolutely no qualms about using that tool.
Sure. That doesn't make anything I said less true; Python is one of my
favorite tools and I'd rather solve problems using Python than with some
other tool, because it is so nice. If I then find out that solving
parts of some problems is not so nice, I can look for ways to make
that easier. Idle intellectual pursuit perhaps, but no need to be
aggressive about it.
Paul Moore
While I don't have a problem with the philosophy of using the right tool for the
job, I think there is a fundamental "type of task" here - namely, pipe-type
plumbing of subprocesses - which is fairly common. It would not do any harm to
have a relatively simple way of doing this in Python.
However, this does *not* mean that there's any need to change anything - it's an
ideal task for a new module (like popen/fork and friends, but higher level). To
answer the original poster's question in a different way, I am not awaer of such
a module which currently exists, but I imagine that designing and writing such a
module would be an interesting, and generally useful task. Heck, i may even have
a look at it myself...
BTW, for people not averse to stealing design ideas from "the opposition",
there's a Perl module on CPAN - IPC::Run, I believe, which does this sort of
thing.
Paul.
Greg Weeks
: Ease of redirection and backgrounding has not been a part of Python in the
: past, and we have to wonder whether adding those features is an actual
: improvement or just a nicety.
Three points:
1. The tone of this discussion has been disappointing, although my thanks
to those who tried to keep it above board.
2. There is no point in *arguing* about this stuff, so I won't. But
silence does not imply consent.
3. Regarding "adding those features": I doubt you'll find anyone on the
planet less interested in adding new features to the Python *language* than
I am. Nor am I interested in adding a new module to the library, unless
someone thinks of something really neat. I was just interested in how to
program. (Misunderstanding my motives is still no excuse for rudeness.)
Blecch,
Greg
Ignacio Vazquez-Abrams
> Ignacio Vazquez-Abrams (ign...@openservices.net) wrote:
> : Ease of redirection and backgrounding has not been a part of Python in the
> : past, and we have to wonder whether adding those features is an actual
> : improvement or just a nicety.
>
> Three points:
>
> 1. The tone of this discussion has been disappointing, although my thanks
> to those who tried to keep it above board.
I apologize for my tone at times. I haven't been feeling 100% lately, but
there's no reason for it to spill over to an online discussion.
> 2. There is no point in *arguing* about this stuff, so I won't. But
> silence does not imply consent.
Agreed. Let's end this thread here.
> 3. Regarding "adding those features": I doubt you'll find anyone on the
> planet less interested in adding new features to the Python *language* than
> I am. Nor am I interested in adding a new module to the library, unless
> someone thinks of something really neat. I was just interested in how to
> program. (Misunderstanding my motives is still no excuse for rudeness.)
Once again, I apologize.
> Blecch,
> Greg
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Paul Moore
wrote:
>While I don't have a problem with the philosophy of using the right tool for the
>job, I think there is a fundamental "type of task" here - namely, pipe-type
>plumbing of subprocesses - which is fairly common. It would not do any harm to
>have a relatively simple way of doing this in Python.
>
>However, this does *not* mean that there's any need to change anything - it's an
>ideal task for a new module (like popen/fork and friends, but higher level). To
>answer the original poster's question in a different way, I am not awaer of such
>a module which currently exists, but I imagine that designing and writing such a
>module would be an interesting, and generally useful task. Heck, i may even have
>a look at it myself...
I did some looking. The attached is a fairly simple prototype of a module for
process handling. The simple examples
>>> output = process.run("echo hello")
>>> print output
hello
>>> print process.run("tr a-z A-Z", input="hello")
HELLO
are similar to shell "backticks", with the ability to provide input. There are
features to allow merging of stdout and stderr, raise an exception if the
command fails (the default action is to continue, but make the return code
available as process.retval()), and set teh mode of the pipe (text or binary).
It's a fairly simple wrapper around popen2/popen4, but it seems fairly useful to
me. Comments anyone?
I have ideas to work on this - things like allowing a sequence of strings as the
"command", which acts like a Unix pipe (the plumbing is messy to set up, so
encapsulating this might be nice...) or maybe some sort of class-based interface
(but the simplicity of the examples above is important, and should remain).
What do people think? Is this a useful idea?
Paul Moore.
Lee Harr
> >>> output = process.run("echo hello")
> >>> print output
> hello
> >>> print process.run("tr a-z A-Z", input="hello")
> HELLO
>
> What do people think? Is this a useful idea?
>
I think it is beautiful. Quite pythonic in my view.
I was looking to do something like this just last week. This would
have been _very_ useful to me.
Not sure what the big hullabaloo is about creating a module which
will allow this sort of functionality from python. Personally, I
find shell scripts to be generally ugly and unmaintainable.
To do something similar from python would clearly be better.
I hope something like this eventually gets in to the standard
distribution.
Ignacio Vazquez-Abrams
> I did some looking. The attached is a fairly simple prototype of a module for
> process handling. The simple examples
Could you please send the attachment to pytho...@python.org so that us poor
saps on the mailing list can see it too?
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Greg Weeks
: What do people think? Is this a useful idea?
I think it would be great if this functionality was available, slick, and
standard. But that might be pretty difficult:
1. The "re" module allows access to a *language* that describes regular
expressions. This language is then available within Python. In general,
this "language within a language" approach is kind of icky. But, as "re"
illustrates, it is sometimes the right solution. Could that be the case
here? And if so, what language? In any case, on Unix systems, os.system()
*currently* implements this approach with the POSIX shell as the language.
Is this sufficiently nonideal to merit the effort to do better?
2. I'm not a nut for efficiency, but I have been nagged *conceptually* by
the inefficiency of building standard I/O redirection on top of popen2.
For example, a traditional Unix pipe command (eg, "foo | bar") requires one
pipe. Implementing this with popen2 requires two pipes which then need to
be "welded" together somehow. (By the way, you have the same problem in
Java and Perl.) I normally wouldn't niggle about such a thing; but it
might be worth thinking about if we're really shooting for a standard
module.
3. Another minor problem with using popen2 to implement redirection:
Suppose the process you invoke starts up a daemon process. Then a read()
from child_stdout can hang, never reaching end-of-file. [I think.]
So, I just don't know. For now, though, I'm living with what is
essentially os.system() with the following additions:
- The shell is called with the -e option.
- ksh is called instead of sh (alas!) because "sh -e -c ..." doesn't work
on my system. (-e is ignored.)
- A nonzero exit code causes a "ShellError" to be thrown. The stderr
text is stored in the ShellError. Indeed, this is the only way to see
the stderr text (which, I admit, is dicey, given how some programs use
stderr).
- There is of course the option of capturing stdout to a string.
Greg
Donn Cave
[ ... pipe & fork wrapper ...]
| I did some looking. The attached is a fairly simple prototype of a module for
| process handling. The simple examples
|
| >>> output = process.run("echo hello")
| >>> print output
| hello
| >>> print process.run("tr a-z A-Z", input="hello")
| HELLO
|
| are similar to shell "backticks", with the ability to provide input. There are
| features to allow merging of stdout and stderr, raise an exception if the
| command fails (the default action is to continue, but make the return code
| available as process.retval()), and set teh mode of the pipe (text or binary).
|
| It's a fairly simple wrapper around popen2/popen4, but it seems fairly useful to
| me. Comments anyone?
|
| I have ideas to work on this - things like allowing a sequence of strings as the
| "command", which acts like a Unix pipe (the plumbing is messy to set up, so
| encapsulating this might be nice...) or maybe some sort of class-based interface
| (but the simplicity of the examples above is important, and should remain).
|
| What do people think? Is this a useful idea?
Sure! You probably won't be surprised to hear that it has been done
a couple of times before - because on one hand, it's useful, and as
you found, it isn't really terribly hard to do.
I don't know if anyone has pushed to get something like this into the
library. Haven't looked at yours, but since you're working with popen2,
I wonder if you could work it up to be submitted as an enhancement to
that module?
Are you thinking about reading stderr for use in the exception?
I find that very useful, but deadlock potential with two separate
input sources will make you earn your pay.
Donn Cave, do...@drizzle.com
Paul Moore
>On Sun, 23 Sep 2001, Paul Moore wrote:
>
>> I did some looking. The attached is a fairly simple prototype of a module for
>> process handling. The simple examples
>
>Could you please send the attachment to pytho...@python.org so that us poor
>saps on the mailing list can see it too?
Sorry. I don't know why the attachment got stripped into a separate message, but
here it is again, as inline text. Apologies if it gets word-wrapped - I get the
feeling I'm not going to win either way, here :-)
Paul.
"""
process - run an OS command and handle standard input and output.
The module exports one main function, run(). This is a preliminary version of
the module, and as such run() has far too many options. But changing the
interface to a class-based one seems to defeat the object of the exercise,
which is to have an extremely simple (one-liner) interface.
Examples:
>>> import process
>>> print process.run("echo Hello, world")
Hello, world
>>> print process.run("tr a-z A-Z", input = "Hello, world")
HELLO, WORLD
>>> print process.run("echo hello 1>&2") # No output, only stdout captured
>>> print process.run("echo hello 1>&2", capture_stderr = 1)
hello
>>> print process.run('exit 1')
>>> print process.retval()
1
>>> print process.run('exit 1', raise_exception=1)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "process.py", line 65, in run
if rv: raise Error
process.Error
"""
class Error(Exception):
pass
_retval = 0
try:
import win32pipe
_popen2 = win32pipe.popen2
_popen4 = win32pipe.popen4
except ImportError:
import os
_popen2 = os.popen2
_popen4 = os.popen4
def retval():
return _retval
def run(cmd, mode = 't', input = None,
capture_stderr = None,
raise_exception = None):
if capture_stderr:
(i, o) = _popen4(cmd, mode)
else:
(i, o) = _popen2(cmd, mode)
if input:
i.write(input)
i.close()
ret = o.read()
rv = o.close()
if raise_exception:
if rv: raise Error
else:
global _retval
_retval = rv
return ret
Ignacio Vazquez-Abrams
> On Sun, 23 Sep 2001 17:35:59 -0400 (EDT), in comp.lang.python you wrote:
>
> >On Sun, 23 Sep 2001, Paul Moore wrote:
> >
> >> I did some looking. The attached is a fairly simple prototype of a module for
> >> process handling. The simple examples
> >
> >Could you please send the attachment to pytho...@python.org so that us poor
> >saps on the mailing list can see it too?
>
> Sorry. I don't know why the attachment got stripped into a separate message, but
> here it is again, as inline text. Apologies if it gets word-wrapped - I get the
> feeling I'm not going to win either way, here :-)
It's because Agent is great for reading, writing, and leeching, but absolutely
TERRIBLE for posting.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Ignacio Vazquez-Abrams
How about these changes:
class Error:
def __init__(self, code=0):
self.code=code
def __str__(self):
return 'Error code %s' % self.code
retval = 0
>
> [snip]
>
> def run(cmd, mode = 't', input = None,
> capture_stderr = None,
> raise_exception = None):
global retval
> if capture_stderr:
> (i, o) = _popen4(cmd, mode)
> else:
> (i, o) = _popen2(cmd, mode)
>
> if input:
> i.write(input)
> i.close()
>
> ret = o.read()
retval = o.close()
> if raise_exception:
if retval: raise Error(retval)
> return ret
with those changes you can use proocess.retval instead of process.retval()
(not a big change, but why have extra code?), and you can get the return code
from the exception more easily.
In fact, you may also want to default the run() function to raising an
exception rather than just setting retval; force the user to have to
explicitly override the Pythonic behaviour.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Neil Schemenauer
use the shell (a feature IMHO) and it only works on Unix systems.
Suggestions for the module name are welcome. I would like to get
something like this in the standard library.
Neil
###########################################################################
# child_process.py
###########################################################################
import os
# created 2001/09/04, nas
class _ChildProcess:
"""
Instance attributes:
pid : int
the process id of the child process
stdin : file
open file connected to file descriptor 0 of the child process
stdout : file
open file connected to file descriptor 1 of the child process
stderr : file
open file connected to file descriptor 2 of the child process
"""
MAXFD = 256
def __init__(self, argv, bufsize=-1):
child_stdin, stdin = os.pipe()
stdout, child_stdout = os.pipe()
stderr, child_stderr = os.pipe()
self.stdin = os.fdopen(stdin, 'w', bufsize)
self.stdout = os.fdopen(stdout, 'r', bufsize)
self.stderr = os.fdopen(stderr, 'r', bufsize)
self.pid = os.fork()
if self.pid == 0:
os.dup2(child_stdin, 0)
os.dup2(child_stdout, 1)
os.dup2(child_stderr, 2)
for i in range(3, self.MAXFD):
try:
os.close(i)
except:
pass
try:
os.execvp(argv[0], argv)
finally:
os._exit(1)
os.close(child_stdin)
os.close(child_stdout)
os.close(child_stderr)
def read(self, n=-1):
"""Read from child stdout"""
return self.stdout.read(n)
def readline(self):
"""Readline from child stdout"""
return self.stdout.readline()
def readlines(self):
"""Readlines from child stdout"""
return self.stdout.readlines()
def write(self, s):
"""Write data to child stdin"""
self.stdin.write(s)
def flush(self):
"""Flush stdin pipe to child"""
self.stdin.flush()
def close(self):
"""Close stdin, stdout and stderr pipes to child process. Wait
for the exit status of the child and return it."""
for fd in (self.stdin, self.stdout, self.stderr):
if not fd.closed:
fd.close()
status = self.wait()
if status == 0:
return None # matches behavior of popen(...).close()
else:
return status
def wait(self, flags=0):
pid, status = os.waitpid(self.pid, flags)
return status
def execute(argv, bufsize=-1):
r"""execute(argv : (string*), bufsize=-1) -> _ChildProcess
Create a child process with pipes connected to its stdout, stdin and
stderr file descriptors. The child is created using fork() and
execvp() in order to avoid the argument quoting problems presented by
using system() or popen(). The bufsize argument has the same meaning as
for the open() builtin and applies to stdin, stdout, and stderr.
Examples:
>>> p = execute(("ls", "/"))
>>> p.readline()
'bin\n'
>>> p.readline()
'boot\n'
>>> p.close()
>>>
>>> p = execute(("cat",))
>>> p.write("hello world\n")
>>> p.stdin.close()
>>> p.read()
'hello world\n'
>>> p.close()
>>>
"""
return _ChildProcess(argv, bufsize)
Paul Moore
>Here's my humble attempt at a popen replacement. Note that it doesn't
>use the shell (a feature IMHO) and it only works on Unix systems.
>Suggestions for the module name are welcome. I would like to get
>something like this in the standard library.
I like the idea of having some form of "simplified piping" module in the
standard library. But I would *much* rather see something that is portable (at
least to Unix and Windows - I've no idea if the Mac has the concept of a command
line which would make sense here). The low-level building blocks are of
necessity non-portable, but the higher-level wrappers really ought to be (in
interface, if not implementation). Precisely because the lower levels are
non-portable - a portable higher level interface would make writing portable
programs that bit easier.
A portability requirement makes designing a suitable interface far more
difficult, but I think it's worth it.
(BTW, I could probably write a Win32 implementation of your interface (modulo
the availability of suitable low-level calls - I'd need to depend on either
win32all or dynwin). The only issue would be the argv tuples, which are *not*
the lowest level Win32 form - precisely the reverse of Unix (in Unix, execv()
takes argc/argv, and the shell converts command lines to argv arrays; on Win32,
CreateProcess() takes a command line, and the C runtime for an individual
program splits the command line into an argv array).
I'm relatively happy with the _ChildProcess class as a way of encapsulating the
various FDs relevant to the child. Although I miss the shortcuts my module has
for passing a string as stdin direct, or getting the contents of stdout direct.
Your class' equivalent is more verbose -
p = execute(...)
p.write(str)
p.stdin.close()
ret = p.read()
p.stdout.close()
I might simply add extra "convenience" methods,
def input(self, str):
self.stdin.write(str)
self.stdin.close()
return self # to allow chaining, see below
def output(self):
str = self.stdout.read()
self.stdout.close()
return str
which allows
ret = execute(...).input(str).output()
Hmm, no that still feels messy. I can't get away from the fact that my version
ret = run(..., input=str)
feels cleaner.
And by the way, your version doesn't offer any way of "merging" stdout and
stderr. Not reading both, but merging as in 2>&1.
And I'd *still* like to add a way of specifying a pipeline, with all the
inter-process plumbing handled automatically. Something like
child_obj = execute(("tr a-z A-Z", "sort", "uniq"))
(reverting to command-string form rather than argv-tuple, to avoid obscuring the
issue with nested tuples...)
OK, that's enough unstructured brain-dumping. I'll think about this some more,
and post a proposal.
[BTW, I assume that avoiding the shell is an important requirement on Unix? It's
less of an obvious thing to do on Win32, where a number of key commands like
DIR, COPY, DEL, RENAME and MOVE are shell built-ins, and not normally available
as separate EXEs]
Paul.
Paul Moore
<ign...@openservices.net> wrote:
>On Mon, 24 Sep 2001, Paul Moore wrote:
>
>How about these changes:
[...]
>with those changes you can use proocess.retval instead of process.retval()
>(not a big change, but why have extra code?), and you can get the return code
>from the exception more easily.
Yes, I like those. I originally had a retval variable rather than a retval()
function, but I changed as a function is read-only where a variable is
read-write. But that's probably excessive paranoia.
>In fact, you may also want to default the run() function to raising an
>exception rather than just setting retval; force the user to have to
>explicitly override the Pythonic behaviour.
Probably. I worried about programs like diff() which return a non-zero exit code
to signal differences, even though differences specifically *aren't* errors...
But you have some good points.
Paul.
Ignacio Vazquez-Abrams
> [BTW, I assume that avoiding the shell is an important requirement on Unix? It's
> less of an obvious thing to do on Win32, where a number of key commands like
> DIR, COPY, DEL, RENAME and MOVE are shell built-ins, and not normally available
> as separate EXEs]
It's not a requirement, but it's a really, Really, REALLY useful feature.
It's also handy for embedded systems that may not have a shell to speak of.
--
Ignacio Vazquez-Abrams <ign...@openservices.net>
Greg Weeks
Donn Cave (do...@drizzle.com) wrote:
: Are you thinking about reading stderr for use in the exception?
stderr is a messy issue, because there is no agreement on what stderr is
for. Some people use stderr only for fatal error messages. Othere use
stderr -- now renamed "diagnostic output" -- for any messages whatsoever
intended for human readers. The first group is like to want exceptions to
include stderr text; the second group isn't.
I'm in the first group, and I'm content to write a private utility that
handles stderr as if it were used only for fatal error messages (even
though I know this will get me into trouble with certain programs). But
I'd be hard-pressed to come up with a general solution. Look at the
evolution of the popen2 module, for instance. And if you have a pipeline
with multiple processes, it just gets that much worse.
Greg
Donn Cave
In mine (I see ftp://ftp.u.washington.edu/public/donn/cmdproc.py is
a fairly current version) there are 4 functions: system, systemr,
expand and expandr - expand returns output, system doesn't, and
*r raises an exception from stderr _if the command exits non-zero_.
I don't think many programmers would agree that stderr should be
reserved for fatal errors, and it isn't ideally suited for use in
an exception, but in practice it works very well for the general
run of utility programs. Use with discretion. The other problem
is error exits of a more "non-perfect" than "fatal" nature - for
example, you say "ls a b", and a exists, so ls displays a valid
result, but b doesn't, so ls writes to stderr and exits 1.
It might be like porting C to Python, where the C code has been
ignoring error codes --- for (i=getdtablesize(); i >= 0; --i)
close(i); and now you have to handle a bunch of exceptions because
Python doesn't ignore errors. Or you can make it an option to
ignore errors.
Some programs will misbehave and write errors to stdout and exit
with random values, but not many. As for pipelines - in shells,
the process at the downstream end of the pipe returns the exit
status for the pipeline. I have not had much need to construct
Python pipelines, though, and haven't thought about what would
be useful there.
Donn Cave, do...@drizzle.com