Python Embedding: redirecting output
Monty Brandenberg
and I need to gather up Python's normal and exceptional text output
and return it to the application's front end as part of a command's
return value. While driving from the top is easy with run_command(),
gathering the output seems non-trivial. So far, my strongest candidate
is to reimplement output on the standard sys.stderr and sys.stdout
objects and to diddle run_command() to initiate and terminate
result gathering. Have others done this and found better ways or
have suggestions to give?
tnx
--
Monty Brandenberg mon...@hks.com
Consulting for: mcb...@world.std.com
Hibbitt, Karlsson & Sorensen, Inc. Tel: (401) 727-4200 x4473
1080 Main Street, Pawtucket, RI 02860 Fax: (401) 727-4208
Guido.va...@cwi.nl
> and I need to gather up Python's normal and exceptional text output
> and return it to the application's front end as part of a command's
> return value. While driving from the top is easy with run_command(),
> gathering the output seems non-trivial. So far, my strongest candidate
> is to reimplement output on the standard sys.stderr and sys.stdout
> objects and to diddle run_command() to initiate and terminate
> result gathering. Have others done this and found better ways or
> have suggestions to give?
From Python, this is simple; use something like
import StringIO
import sys
myout = StringIO.StringIO()
myerr = StringIO.StringIO()
saveout = sys.stdout
saveerr = sys.stderr
try:
sys.stdout = myout
sys.stderr = myerr
exec <whatever>
finally:
sys.stdout = saveerr
sys.stderr = saveerr
and then myout.getvalue() and myerr.getvalue() return the standard
output and standard error text from <whatever>.
Also note that if you want a standard exception traceback as well, you
could replace "exec <whatever>" by
try:
except <whatever>
except:
import traceback
traceback.print_exc()
(This prints the traceback on stdout -- redirecting it to stderr is
left as an excercise :-)
--Guido van Rossum, CWI, Amsterdam <mailto:Guido.va...@cwi.nl>
<http://www.cwi.nl/cwi/people/Guido.van.Rossum.html>
Steven D. Majewski
> I'm busy embedding Python into a distributed application environment
> and I need to gather up Python's normal and exceptional text output
> and return it to the application's front end as part of a command's
> return value. While driving from the top is easy with run_command(),
> gathering the output seems non-trivial. So far, my strongest candidate
> is to reimplement output on the standard sys.stderr and sys.stdout
> objects and to diddle run_command() to initiate and terminate
> result gathering. Have others done this and found better ways or
> have suggestions to give?
>
Well - if you're embedding python and trying to deliver it's output
to the "outer" application, then it might make more sense to write
a module in C that mimics a file and reassign sys.{stdin,stdout,stderr)
to it. ( The write method could then call the appropriate routine to
handle the stream whenever it reached the right character or buffered
enough data in the stream. )
But, as an example of how to handle this in python, here's a copy of
my frequently reposted redirect module: it creates a file-like object
to be a sink for sys.output, and uses that object in the functions
'tostring( func, *args )' and 'tolines( func, *args )' - which will
call the function with the arguments and return the output as a single
string or a list of line strings, respectively. ( Also 'tofile', which
redirects sys.stdout to another file object: the test procedure at the
end does a 'tofile' of a command to unix 'more' so you can 'page' the
output - that really *ought* to be wrapped up into it's own function that
gets the environ variable PAGER, etc. and on a non-unix system, does
it's own mini-pager. )
---| Steven D. Majewski (804-982-0831) <sd...@Virginia.EDU> |---
---| Computer Systems Engineer University of Virginia |---
---| Department of Molecular Physiology and Biological Physics |---
---| Box 449 Health Science Center Charlottesville,VA 22908 |---
#!/usr/local/bin/python
#
# <module 'redirect'>
#
# - Steven D. Majewski <sd...@Virginia.EDU>
#
# Functions:
# tofile( file, func, *args )
# tostring( func, *args ) ==> string
# tolines( func, *args ) ==> [ line0, line1, ... lineN ]
#
# Functions apply a function to args, either redirecting the output
# or returning it as a string or a readlines() like list of lines.
#
# tofile will print (to the file) and return the value returned by
# apply( func, *args ). The value is also in the last string in
# tolines ( or the last line in tostring ). tolines and tostring,
# will print the value on the original sys.stdout as well (unless
# it's == None ).
#
#
# Class StringFile()
# Methods: [ all of the typical file object methods ]
# read(),write(),readline(),readlines(),writelines(),
# seek(),tell(),flush(),close(),isatty() [NO fileno()]
#
# Creates a file-like interface to a character array.
# Write's append to the array; Read's return the characters in the array.
#
# Class Tee( file1, file2 [, ... filen ] )
# create a fileobject that writes it's output to all of the files.
# Class Echo( filein, fileout )
# create a fileobject that automatically echo's whatever is read
# from filein to fileout.
#
# An instance of a Tee object can be assigned to sys.stdout and sys.stderr,
# and all Python output will be 'tee'-ed to that file.
# Unfortunately, 'Echo'-ing stdin does not reproduce YOUR typed input to
# the interpreter, whose input comes via a different mechanism.
# Implementing a 'dribble()' function, that logs all input and output to
# a file will require another trick.
#
#
#
# 'tofile()' temporarily reassigns sys.stdout while doing func.
# 'tostring()' and 'tolines()' both call 'tofile()' with an instance
# of StringFile().
#
#
# tofile( '|lpr', func, output )
#
import sys
import os
def filew( file ):
# file is a filename, a pipe-command, a fileno(), or a file
# returns file.
if not hasattr( file, 'write' ) :
if file[0] == '|' : file = os.popen( file[1:], 'w' )
else: file = open( file, 'w' )
return file
def filer( file ):
# file is a filename, a pipe-command, or a file
# returns file.
if not hasattr( file, 'read' ) :
if file[-1] == '|' : file = os.popen( file[1:], 'r' )
else: file = open( file, 'r' )
return file
def tofile( file, func, *args ):
# apply func( args ), temporarily redirecting stdout to file.
# file can be a file or any writable object, or a filename string.
# a "|cmd" string will pipe output to cmd.
# Returns value of apply( func, *args )
ret = None
file = filew( file )
sys.stdout, file = file, sys.stdout
try:
ret = apply( func, args )
finally:
print ret
sys.stdout, file = file, sys.stdout
return ret
def tostring( func, *args ):
# apply func( *args ) with stdout redirected to return string.
string = StringFile()
apply( tofile, ( string, func ) + args )
return string.read()
def tolines( func, *args ):
# apply func( *args ), returning a list of redirected stdout lines.
string = StringFile()
apply( tofile, ( string, func ) + args )
return string.readlines()
from array import array
# A class that mimics a r/w file.
# strings written to the file are stored in a character array.
# a read reads back what has been written.
# Note that the buffer pointer for read is independent of write,
# which ALWAYS appends to the end of buffer.
# Not exactly the same as file semantics, but it happens to be
# what we want!
# Some methods are no-ops, or otherwise not bery useful, but
# are included anyway: close(), fileno(), flush(),
class StringFile:
def __init__( self ):
self._buf = array( 'c' )
self._bp = 0
def close(self):
return self
# On second thought, I think it better to leave this out
# to cause an exception, rather than letting someone try
# posix.write( None, string )
# def fileno(self):
# return None
def flush(self):
pass
def isatty(self):
return 0
def read(self, *howmuch ):
buf = self._buf.tostring()[self._bp:]
if howmuch:
howmuch = howmuch[0]
else:
howmuch = len( buf )
ret = buf[:howmuch]
self._bp = self._bp + len(ret)
return ret
def readline(self):
line = ''
for c in self._buf.tostring()[self._bp:] :
line = line + c
self._bp = self._bp + 1
if c == '\n' : return line
def readlines(self):
lines = []
while 'True' :
lines.append( self.readline() )
if not lines[-1] : return lines[:-1]
def seek(self, where, how ):
if how == 0 :
self._bp = where
elif how == 1 :
self._bp = self._bp + where
elif how == 2 :
self._bp = len(self._buf.tostring()) + where
def tell(self):
return self._bp
def write(self, what ):
self._buf.fromstring( what )
def writelines( self, lines ):
for eachl in lines:
self._buf.fromstring( eachl )
class Tee:
# Tee( file1, file2 [, filen ] )
# creates a writable fileobject where the output is tee-ed to all of
# the individual files.
def __init__( self, *optargs ):
self._files = []
for arg in optargs:
self.addfile( arg )
def addfile( self, file ):
self._files.append( filew( file ) )
def remfile( self, file ):
file.flush()
self._files.remove( file )
def files( self ):
return self._files
def write( self, what ):
for eachfile in self._files:
eachfile.write( what )
def writelines( self, lines ):
for eachline in lines: self.write( eachline )
def flush( self ):
for eachfile in self._files:
eachfile.flush()
def close( self ):
for eachfile in self._files:
self.remfile( eachfile ) # Don't CLOSE the real files.
def CLOSE( self ):
for eachfile in self._files:
self.remfile( eachfile )
self.eachfile.close()
def isatty( self ):
return 0
class Echo:
def __init__( self, input, *output ):
self._infile = filer( input )
if output : self._output = filew(output[0])
else: self._output = None
def read( self, *howmuch ):
stuff = apply( self._infile.read, howmuch )
if output: self._output.write( stuff )
return stuff
def readline( self ):
line = self._infile.readline()
self._output.write( line )
return line
def readlines( self ):
out = []
while 1:
out.append( self.readline() )
if not out[-1]: return out[:-1]
def flush( self ):
self._output.flush()
def seek( self, where, how ):
self._infile.seek( where, how )
def tell( self ): return self._infile.tell()
def isatty( self ) : return self._infile.isatty()
def close( self ) :
self._infile.close()
self._output.close()
if __name__ == '__main__':
def testf( n ):
for i in range( n ):
print '._.'*10 + '[', '%03d' % i, ']' + 10*'._.'
if hasattr( os, 'popen' ):
tofile( '|more', testf, 300 )
print '\n# Last 10 lines "printed" by testf(): '
print '# (the Python equivalent of \'tail\'.)'
for line in tolines( testf, 300 )[-10:] :
print line[:-1]
Hammond, Mark (@ JM)
>I'm busy embedding Python into a distributed application environment
>and I need to gather up Python's normal and exceptional text output
>and return it to the application's front end as part of a command's
>return value. While driving from the top is easy with run_command(),
>gathering the output seems non-trivial. So far, my strongest candidate
>is to reimplement output on the standard sys.stderr and sys.stdout
>objects and to diddle run_command() to initiate and terminate
>result gathering. Have others done this and found better ways or
>have suggestions to give?
The simplest way of grabbing output is to override sys.stdout and
sys.stderr. All that is required is that the object you redirect to has a
"write" method.
An apporach you could take would be to run_command() a command for
re-assigning sys.stdout/err to one of your objects. Then run_command() the
real command, and conther run_command() to restore it back. However, in
some imbedded applications, it is often OK to redirect stdout once at the
start for the life of the application.
Mark.
Monty Brandenberg
: The simplest way of grabbing output is to override sys.stdout and
: sys.stderr. All that is required is that the object you redirect to has a
: "write" method.
: An apporach you could take would be to run_command() a command for
: re-assigning sys.stdout/err to one of your objects. Then run_command() the
: real command, and conther run_command() to restore it back. However, in
: some imbedded applications, it is often OK to redirect stdout once at the
: start for the life of the application.
That's partly the idea. In my application, Python is truly an embedded
component. It doesn't have control of the command stream and is
driven by requests from a distributed user interface running well
above it. I believe that something like the StringIO package Guido
mentioned or the code Steve posted will get me close to what I want
(having only dove into Python code a few days ago I'm still on the
learning curve) but fitting it into an application raises some
general design questions about embedding.
1. Source of input. 'run_command()' or something like it seems
to be correct. As an embedded component, Python cannot assume
that it can poll/sleepwait for input. It may have to be
driven from the top as in my application. However, as
Python becomes thread-friendly someone may choose to
run Python in its own thread. In this case, it is suitable
to have Python perform a sleeping read on a modified or
subclassed fileobject that performs interthread communication.
2. Redirecting output and error. Everyone seems to understand
this so I won't elucidate much. Basically, the embedding
application wants a sequence something like:
Clear output and error buffers.
Execute Python command(s).
Return output and error buffers to calling code.
Posted code seems to do most of that though I'm having
difficulty with the latter at the moment.
3. Exception handling. Executing a Python command may be part
of a larger transaction in the embedding application. Such
an application may want to specify it's own top-level
exception handling. E.g. allow Python to emit an
elaborated exception message to sys.stdout or sys.stderr,
return the exception status and reason to the caller
of run_command() or run_script(), etc. This may actually
be available now, I just haven't found the right way to
access it.
4. Mechanism for providing the above. This is partly a matter
of aesthetics. Most suggestions so far have centered
on a methods based on Python scripts to redefine classes
to do IO and that is elegant from a language-user's
perspective. But it could also be specified as part of
an initializion step via C interface. In this way,
run_command() and friends are never invoked until *after*
I/O and exception handling are pointing to the desired
application facilities. There is no interim state where
Python might take an exception or babble to the wrong
data stream. There is no dependency on the availability
of external modules such as StringIO. From an application
developer's point of view, this is somewhat more elegant
and trustworthy. Consider rewriting the standard Python
main program as an embedded application:
main() {
initall();
// Construct standard fileobjects for
// sys.stdin, sys.stdout, and sys.stderr
// from main's stdin, stdout, stderr
// Establish Python exception handling
// mechanism (built-in default?).
for (;;) {
readline(...);
run_command(...);
}
}
Comments? And thanks to everyone for the replies so far, they are
appreciated.
monty
Dag Gruneau
>
> I'm busy embedding Python into a distributed application environment
> and I need to gather up Python's normal and exceptional text output
> and return it to the application's front end as part of a command's
> return value.
Changing sys.stderr and sys.stdout works well with all Python
generated output. But the interpreter itself can produce error
messages on stderr that you might want to catch. These messages
are internal error messages and will not occur under normal conditions.
To catch the internal messages I redirected stderr to a file
so I know I will get the message in case anything appears.
On contract now, real adress:
-----------------------------o----------------------------------------------
Dag Gruneau \ 0 Voice: +46 (0)708 87 46 85
TeleTalk AB \_| Fax: +46 (0)8 31 31 34
Box 6342 \---------\---------/ Mail: dag.g...@teletalk.se
S102 35 Stockholm ^^^^^^^^^^^^^^0^^^^^^^^^^^^ NeXT mail, MIME
Sweden
----------------------------------------------------------------------------
Hammond, Mark
> Changing sys.stderr and sys.stdout works well with all Python
> generated output. But the interpreter itself can produce error
> messages on stderr that you might want to catch. These messages
> are internal error messages and will not occur under normal conditions.
>
> To catch the internal messages I redirected stderr to a file
> so I know I will get the message in case anything appears.
This is getting better with each Python release. AFAIK, there are only
about 3 or 4 places in the code that do the fprintf(stderr,...) thing -
usually followed by an abort().
Even this code seems to be moving towards a generic abort function (cant
remember the name) so an application should be able to override the abort
function at link time.
In my expericene imbedding Python, I have only seen such errors very few
times - stuffing reference counts is a pretty good way to get them. I have
never seen such an error message from bad Python code - just bad C code
wrapped around Python.
Basically I am saying that explicitely grabbing stderr is a good idea, but
probably not worth it for the number of times anything will be written to
it.
Mark.