the trunk now has gevent.subprocess module

[Denis Bilenko]

The interface is the same as stdlib's subprocess on Python 2.7.2. It
works on Windows.

Please test it and post your feedback there.

[François-Xavier Bourlet]

Hey Denis,

Is it gevent < 1.0 compatible? Is there is anyway to expect the
gevent.subprocess backported to gevent < 1.0?

I actually maintain the package "gevent_subprocess" on pipy, but it
has never be tested on windows, either run against official python
subprocess tests (there is few tests unit test coming with it though).
So I would love to have subprocess in the stable version of gevent.

A little implementation detail question: it is waiting after a process
by catching the SIGCHL signal?

With gevent 0.13.7, when starting a lots of process, if they all
terminate at the same time, you easily miss half of the SIGCHLD (you
simply don't get them). So I did something... you know... polling
(shame on me).
Following the fact that UNIX signals are not reliable by design, this
problem is expected.

Is libev does something special about that? (threads for waitpid?)

Thanks for gevent,

Regards,
fx

--
François-Xavier Bourlet

[Denis Bilenko]

On Wed, May 2, 2012 at 11:25 PM, François-Xavier Bourlet
<bom...@gmail.com> wrote:
> Hey Denis,
>
> Is it gevent < 1.0 compatible?

No.

> Is there is anyway to expect the gevent.subprocess backported to gevent < 1.0?

No, because on POSIX it uses libev's child watchers and on Windows it
uses a threadpool. Neither is present in gevent 0.13, so backporting
would be very very hard.

> I actually maintain the package "gevent_subprocess" on pipy, but it
> has never be tested on windows, either run against official python
> subprocess tests (there is few tests unit test coming with it though).
> So I would love to have subprocess in the stable version of gevent.

Well, 1.0 is bound to become stable at some point :)
Is there anything that stops you from using 1.0beta now?

> A little implementation detail question: it is waiting after a process
> by catching the SIGCHL signal?

yes, that's what libev does.

> With gevent 0.13.7, when starting a lots of process, if they all
> terminate at the same time, you easily miss half of the SIGCHLD (you
> simply don't get them). So I did something... you know... polling
> (shame on me).
> Following the fact that UNIX signals are not reliable by design, this
> problem is expected.
>
> Is libev does something special about that? (threads for waitpid?)

What libev does is it calls non-blocking waitpid(ANY) in a loop until
there no more dead children left.
https://bitbucket.org/denis/gevent/src/f838056c793d/libev/ev.c#cl-2010
Thus multiple SIGCHLDs merging into one is not a problem.

Cheers,
Denis.

[François-Xavier Bourlet]

Thanks for the detailed response.

>
> Well, 1.0 is bound to become stable at some point :)
> Is there anything that stops you from using 1.0beta now?
>

For some lazy reasons, we are going to wait and use my
gevent_subprocess for now. But as soon as we can switch we will.

>> A little implementation detail question: it is waiting after a process
>> by catching the SIGCHL signal?
>
> yes, that's what libev does.
>
>> With gevent 0.13.7, when starting a lots of process, if they all
>> terminate at the same time, you easily miss half of the SIGCHLD (you
>> simply don't get them). So I did something... you know... polling
>> (shame on me).
>> Following the fact that UNIX signals are not reliable by design, this
>> problem is expected.
>>
>> Is libev does something special about that? (threads for waitpid?)
>
> What libev does is it calls non-blocking waitpid(ANY) in a loop until
> there no more dead children left.
> https://bitbucket.org/denis/gevent/src/f838056c793d/libev/ev.c#cl-2010
> Thus multiple SIGCHLDs merging into one is not a problem.
>

Thanks for the code link, I should have think about it :)


Regards,
--
François-Xavier Bourlet

[Denis Bilenko]

On Tue, Sep 4, 2012 at 9:50 PM, Dan Sully <dsu...@gmail.com> wrote:
> Calling:
>
> subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>
> Results in a: TypeError: child watchers are only available on the default
> loop
>
> The subprocess.Popen call is run in a greenlet that's spawned by a patched
> gevent.threadpool.ThreadPool()

How do you use thread pool ? If you use any gevent API there (directly
or through monkey patching), an event loop would be created for each
thread. Due to libev limitations only the original event loop can work
with child watchers, hence the exception.