Python question about subprocess.Popen() and stdout

[Joshua Russo]

I'm creating a wxPython application that the client can run to make
sure the installation stack is functioning properly. I want to include
the unit test processing in this application but I've run into some
difficulties. An additional wrinkle I threw in was that I am using
py2exe so they don't also need to install wxPython, we'll see if that
will actually work or not.

It seems like the most appropriate thing to do is to use
subprocess.Popen() to trigger "manage.py test --noinput". The problem
I am trying to solve is that this runs for a while and I want to
display the stdio output into a text box on the window.

I had this working using django.core.management.call_command but
creating the exe broke the pathing within Django and even some Python
libraries (thus dumping py2exe could be what I need to do).

The problem with switching to Popen is that it expects a file type
object with a fileno() method. The file class cannot be extended in
the stdio instance because you can't write back to the __class__
property. I guess what I need to know is how do I mimic stdout? Stdout
doesn't output to a file, but it still uses the file object. It seems
like I want to open a file object onto a Nul output. Can this be
done?

Thanks
Josh

[Joshua Russo]

Ok, so it appears that (in Python 2.5 at least) there is no way to capture the stdout of subprocess.Popen(). I've tried many different suggestions from questions that have already been posted to Stack Overflow and everything either buffers until the end of processing or just plain doesn't work.

So I dumped Py2exe and went back to using django.core.management.call_command() and everything works like a dream.

In case you care.

[Hassan]

> Ok, so it appears that (in Python 2.5 at least) there is no way to capture
> the stdout of subprocess.Popen()

just do this

from subprocess import Popen, PIPE
p = Popen([cmd], stdout=PIPE)
p.stdout.readlines()

thats it!

[Joshua Russo]

The problem is that it waits for the process to end to output
anything. Unless I was doing something wrong, but I think I was doing
just what you describe here.

[Steve Holden]

Well, readlines() inherently has to see the end of the data stream
before it can return a list of all the lines that the data stream
contains, so that's hardly surprising is it?

Try using readline() in a loop and see if that gives you better results.
I don't guarantee it will, but at least you will have some chance if
subprocess.open() *isn't* buffering the whole stream.

regards
Steve
--
I'm no expert.
"ex" == "has-been"; "spurt" == "drip under pressure"
"expert" == "has-been drip under pressure".

[Joshua Russo]

On Thu, Aug 5, 2010 at 10:53 PM, Steve Holden <hold...@gmail.com> wrote:

On 8/5/2010 7:16 PM, Joshua Russo wrote:
> On Aug 4, 6:49 pm, Hassan <hsn.zam...@gmail.com> wrote:
>>> Ok, so it appears that (in Python 2.5 at least) there is no way to capture
>>> the stdout of subprocess.Popen()
>>
>> just do this
>>
>> from subprocess import Popen, PIPE
>> p = Popen([cmd], stdout=PIPE)
>> p.stdout.readlines()
>>
>> thats it!
>
> The problem is that it waits for the process to end to output
> anything. Unless I was doing something wrong, but I think I was doing
> just what you describe here.
>

Well, readlines() inherently has to see the end of the data stream
before it can return a list of all the lines that the data stream
contains, so that's hardly surprising is it?

Try using readline() in a loop and see if that gives you better results.
I don't guarantee it will, but at least you will have some chance if
subprocess.open() *isn't* buffering the whole stream.

regards
 Steve

Tried that too and even putting the reading of them in their own thread as someone else suggested on Stackoverflow. I just can't find anyway to capture stdout in real-time from Popen. The problem is that everything uses readline() that buffers the output.

http://stackoverflow.com/questions/874815/how-do-i-get-real-time-information-back-from-a-subprocess-popen-in-python-2-5

http://stackoverflow.com/questions/2082850/real-time-subprocess-popen-via-stdout-and-pipe

The way I got it to work with out Popen is to use a text box control as the stdout object. This works great, utilizing the object's write() method. 

The problem with Popen is that it expects the stdout object to have a fileno() method also, which the text box control does not have. So I tried extending the text box to add the fileno() and had it return 1 like the standard stdout, but still no dice. I didn't let the process (unit tests) finish but it definitely was not giving real time output, through it did accept the control with the fileno() method added. It really seems like that should have worked and I may have been able to dig further into Popen to make it work but I figured that I had spent too much time on it as it was.

If anybody has more suggestions I'd love to hear them.

[J. Cliff Dyer]

"Joshua Russo" <josh.r...@gmail.com> wrote:

>--
>You received this message because you are subscribed to the Google Groups "Django users" group.
>To post to this group, send email to django...@googlegroups.com.
>To unsubscribe from this group, send email to django-users...@googlegroups.com.
>For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>

If I understand your issue correctly, in ran into something similar a while back. The problem is not in subprocess, but in how stdout works. The output is buffered by default, so it only gets written on a line-by-line basis.

If you want to catch it one character at a time, you have to explicitly flush the output after each character using sys.stdout.flush(). Then, of course, you'll also have to read it from subprocess one character at a time, too,
using .read(1) instead of .readline()

I hope that addresses your issue.

Cheers,
Cliff
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[Joshua Russo]

On Aug 6, 9:57 am, "J. Cliff Dyer" <j...@sdf.lonestar.org> wrote:
> "Joshua Russo" <josh.r.ru...@gmail.com> wrote:

> >http://stackoverflow.com/questions/874815/how-do-i-get-real-time-info...
> >http://stackoverflow.com/questions/2082850/real-time-subprocess-popen...

>
> >The way I got it to work with out Popen is to use a text box control as the
> >stdout object. This works great, utilizing the object's write() method.
>
> >The problem with Popen is that it expects the stdout object to have a
> >fileno() method also, which the text box control does not have. So I tried
> >extending the text box to add the fileno() and had it return 1 like the
> >standard stdout, but still no dice. I didn't let the process (unit tests)
> >finish but it definitely was not giving real time output, through it did
> >accept the control with the fileno() method added. It really seems like that
> >should have worked and I may have been able to dig further into Popen to
> >make it work but I figured that I had spent too much time on it as it was.
>
> >If anybody has more suggestions I'd love to hear them.
>
> >--
> >You received this message because you are subscribed to the Google Groups "Django users" group.
> >To post to this group, send email to django...@googlegroups.com.
> >To unsubscribe from this group, send email to django-users...@googlegroups.com.

> >For more options, visit this group athttp://groups.google.com/group/django-users?hl=en.

>
> If I understand your issue correctly, in ran into something similar a while back. The problem is not in subprocess, but in how stdout works.  The output is buffered by default, so it only gets written on a line-by-line basis.
>
> If you want to catch it one character at a time, you have to explicitly flush the output after each character using sys.stdout.flush().  Then, of course, you'll also have to read it from subprocess one character at a time, too,
> using .read(1) instead of .readline()
>
> I hope that addresses your issue.
>
> Cheers,
> Cliff

Thanks for the tip, I'll check it out.

[Joshua Russo]

Nope, still no dice. Below is what I did. I modified the worker() function from one of the Stackoverflow examples. Let me know if I'm using flush properly. Keep in mind that everything you see here is already running in a separate thread from the wxPython GUI so I can update the GUI while all my tests run in the background.

import threading

import subprocess as sp

import sys

class ThreadWorker(threading.Thread):

    def __init__(self, callable, *args, **kwargs):

        super(ThreadWorker, self).__init__()

        self.callable = callable

        self.args = args

        self.kwargs = kwargs

        self.setDaemon(True)

    def run(self):

        try:

            self.callable(*self.args, **self.kwargs)

        except wx.PyDeadObjectError:

            pass

        except Exception, e:

            print e

def worker(pipe):

sys.stdout.write('**start worker**\n')

line = []

while True:

pipe.flush()

blob = pipe.read()

if blob == '':

sys.stdout.write('**break**\n')

break

for c in blob:

line.append(c)

if c == '\n':

sys.stdout.write(''.join(line))

line = []

tstP = sp.Popen("python.exe manage.py test --noinput --settings=adminTributaria.settings.test", stdout=sp.PIPE, stderr=sp.PIPE)

stdout_worker = ThreadWorker(worker, tstP.stdout)

stdout_worker.start()

tstP.wait()

[J. Cliff Dyer]

On Fri, 2010-08-06 at 18:35 -0100, Joshua Russo wrote:

>
>
> Nope, still no dice. Below is what I did. I modified the worker()
> function from one of the Stackoverflow examples. Let me know if I'm
> using flush properly. Keep in mind that everything you see here is
> already running in a separate thread from the wxPython GUI so I can
> update the GUI while all my tests run in the background.
>

Sorry. I didn't realize you're doing this in a threaded/wxPython
context. I really don't have any experience with that kind of
programming. I was confused, because you posted to a django mailing
list. You might want to try another list that deals with wxPython
directly.

Cheers,
Cliff