Filtering data?

[Mark Shuttleworth]

This is a cryptographically signed message in MIME format.

--------------ms839E66D080CB3F583003B7B1
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hi folk

Is there any way to open a pipe to a command that you can read from and
write to? For example, say I have a program that performs BASE64
encoding, and I want to encode some text. I'd like to say say something
like:

t = os.popen('command -option -option -option', 'rw')
t.write(text)
result = t.read()

Or something like that ;-). Can this be done?

--
Mark Shuttleworth
Thawte Consulting

--------------ms839E66D080CB3F583003B7B1
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

<encoded_portion_removed>
yT4=
--------------ms839E66D080CB3F583003B7B1--

[David Arnold]

-->"Mark" == Mark Shuttleworth <ma...@thawte.com> writes:

Mark> Is there any way to open a pipe to a command that you can read
Mark> from and write to?

import popen2
rfd, wfd = popen2.popen2("some command")

etc, etc

-- David Arnold ,=================================================
=================' +617 33654310 (voice)
CRC for Distributed Systems Technology +617 33654311 (fax)
University of Queensland dav...@pobox.com (email)
Australia <http://www.pobox.com/~davida> (web)

C++ compilers rarely optimize for the joy of programming - lwall

[Mark Shuttleworth]

Hiya

Thanks! (And yes, I use the base64 module ATT, but it seemed like such a
nice example ;-)

[Fredrik Lundh]

>Is there any way to open a pipe to a command that you can read from and
>write to? For example, say I have a program that performs BASE64
>encoding, and I want to encode some text.

>like:

1. check the popen2.py script in the standard library
2. in this case, use the base64 module

for more info on (1), search for "popen2" in the newsgroup.
among other things, GvR posted an enhanced version of that
module not long ago (it might still be on your newsserver;
the subject was "executing an external program").

Cheers /F

[Guido van Rossum]

> Is there any way to open a pipe to a command that you can read from and
> write to? For example, say I have a program that performs BASE64

> encoding, and I want to encode some text. I'd like to say say something
> like:
>
> t = os.popen('command -option -option -option', 'rw')
> t.write(text)
> result = t.read()

You've received some answers already pointing to popen2. However I
must warn you that there is a severe chance of deadlocks if you code
it this way!

Pipes have a limited buffering capacity. While you're writing text to
the subprocess, you're not also reading its output. When the
subprocess produces enough output to fill up the buffer for its output
pipe, it is suspended, and won't read any more data. Then its input
pipe buffer starts filling up, and eventually the Python process that
is writing to it also gets suspended. Voila, deadlock.

One would think that the solution is to use select to read or write
parts of the data whenever either part is ready. Sure, this works.
But a much simpler solution is to use a temporary file:

import tempfile
tfn = tempfile.mktemp()
try:
tf = open(tfn, "w")
tf.write(text)
tf.close()
t = os.popen("...", "r")
result = t.read()
t.close()
finally:
os.unlink(tfn) # Always get rid of the temp file!

I bet this is more readable that the version with select.

Incidentally, I believe that the deadlock (which is quite subtle,
since it only happens for large amounts of data) is the real reason
that the Unix C library has no equivalent of popen2 -- it is simply a
bad idea.

--Guido van Rossum (home page: http://www.python.org/~guido/)