subprocess + python-daemon - bug/problem?

949 views
Skip to first unread message

Andy Clegg

unread,
Jul 9, 2009, 5:26:10 AM7/9/09
to
Hi all,

I'm trying to daemonize a python program, and occasionally have it run
subprocesses, however I'm running into a nasty error, as follows:

"File "/users/rsg/ancl/devcocast/devcocast-svn/scripts/DaemonSpawnTes
t.py", line 5, in <module>
subprocess.Popen('echo 1').wait()
File "/usr/lib64/python2.5/subprocess.py", line 594, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1089, in _execute_ch
ild
os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes"

The minimal testcase showing this problem is as follows:

"import daemon
import subprocess

daemon.DaemonContext(stderr = open("fakeConsole.txt","w+")).open()
subprocess.Popen('echo 1').wait()"

So there is no threading going on (I've found some bugs relating to
subprocess and threading). I'm using Fedora 10, Python 2.5.2, and
python-daemon 1.4.6 from here http://pypi.python.org/pypi/python-daemon/
.

If anyone can shed some light on the situation, I'd be extremely
grateful!

Yours,

Andy

Andy Clegg

unread,
Jul 9, 2009, 5:42:54 AM7/9/09
to
My apologies, the python code should have been:

"import daemon
import subprocess

daemon.DaemonContext(stderr = open("fakeConsole.txt","w+")).open()

subprocess.Popen(['echo','1']).wait()"

However the error remains the same.

Yours,

Andy

On Jul 9, 10:26 am, Andy Clegg <andy...@gmail.com> wrote:
> Hi all,
>
> I'm trying to daemonize a python program, and occasionally have it run
> subprocesses, however I'm running into a nasty error, as follows:
>
> "File "/users/rsg/ancl/devcocast/devcocast-svn/scripts/DaemonSpawnTes
> t.py", line 5, in <module>
>     subprocess.Popen('echo 1').wait()
>   File "/usr/lib64/python2.5/subprocess.py", line 594, in __init__
>     errread, errwrite)
>   File "/usr/lib64/python2.5/subprocess.py", line 1089, in _execute_ch
> ild
>     os.waitpid(self.pid, 0)
> OSError: [Errno 10] No child processes"
>
> The minimal testcase showing this problem is as follows:
>
> "import daemon
> import subprocess
>
> daemon.DaemonContext(stderr = open("fakeConsole.txt","w+")).open()
> subprocess.Popen('echo 1').wait()"
>
> So there is no threading going on (I've found some bugs relating to
> subprocess and threading). I'm using Fedora 10, Python 2.5.2, and

> python-daemon 1.4.6 from herehttp://pypi.python.org/pypi/python-daemon/

Ben Finney

unread,
Jul 10, 2009, 7:30:41 AM7/10/09
to
Andy Clegg <and...@gmail.com> writes:

> "import daemon
> import subprocess
>
> daemon.DaemonContext(stderr = open("fakeConsole.txt","w+")).open()
> subprocess.Popen(['echo','1']).wait()"
>
> However the error remains the same.

The error appears in the file specified for the stderr output of the
DaemonContext. Here it is without the unwanted extra line-wrapping that
seems to plague all Google Mail users (seriously, folks: get a real mail
provider that won't mangle your messages)::

Traceback (most recent call last):
File "/home/bignose/Projects/python/python-daemon/bin/andy-clegg-test", line 7, in <module>


subprocess.Popen(['echo', '1']).wait()

File "/usr/lib/python2.5/subprocess.py", line 1184, in wait
pid, sts = self._waitpid_no_intr(self.pid, 0)
File "/usr/lib/python2.5/subprocess.py", line 1014, in _waitpid_no_intr
return os.waitpid(pid, options)


OSError: [Errno 10] No child processes

I confirm that I'm getting the same error; Python 2.5.4.

My first thought was perhaps it's related to the fact that the process
has no stdout? But setting ‘stdout’ and ‘stderr’ to the same file::

#! /usr/bin/python

import daemon
import subprocess

fake_console = open("fake_console.txt", "w+")
daemon.DaemonContext(stdout=fake_console, stderr=fake_console).open()


subprocess.Popen(['echo', '1']).wait()

still gives the same error::

1
Traceback (most recent call last):
File "/home/bignose/Projects/python/python-daemon/bin/andy-clegg-test", line 8, in <module>


subprocess.Popen(['echo', '1']).wait()

File "/usr/lib/python2.5/subprocess.py", line 1184, in wait
pid, sts = self._waitpid_no_intr(self.pid, 0)
File "/usr/lib/python2.5/subprocess.py", line 1014, in _waitpid_no_intr
return os.waitpid(pid, options)


OSError: [Errno 10] No child processes

I'm not familiar enough with the nuances of the ‘subprocess’ module to
know what might be going wrong here. I'd like to know whether it might
be a problem in the ‘python-daemon’ library.

--
\ “Whatever a man prays for, he prays for a miracle. Every prayer |
`\ reduces itself to this: “Great God, grant that twice two be not |
_o__) four.”” —Ivan Turgenev |
Ben Finney

Ben Finney

unread,
Jul 10, 2009, 7:55:10 PM7/10/09
to
Ben Finney <ben+p...@benfinney.id.au> writes:

> Here it is without the unwanted extra line-wrapping that seems to
> plague all Google Mail users (seriously, folks: get a real mail
> provider that won't mangle your messages)

I've been asked off-list whether I have any suggestions for those who
want a better email provider. Handling email (transport and mailbox
services) is certainly a service that's worth getting someone else to
handle, provided you can trust they'll do so reliably and well. For me,
“well” includes “don't mangle my messages”.

My personal solution to this is to manage my own email server machine,
but that's an ongoing investment of time that is not something I would
recommend to everyone.

Nancy McGough maintained (past tense? present tense?) a comprehensive
index <URL:http://www.ii.com/internet/messaging/imap/isps/#providers> of
email service providers with IMAP access, hence allowing you to use
whatever IMAP mail client you like to communicate with your mailboxes
remotely. There are feature lists and price indications for each
provider.

Several of my colleagues swear by Tuffmail <URL:http://tuffmail.com/> as
providing excellent uptime, fine control filtering, responsive service,
flexible customisation, and good overall value.

--
\ “Visitors are expected to complain at the office between the |
`\ hours of 9 and 11 a.m. daily.” —hotel, Athens |
_o__) |
Ben Finney

Ben Finney

unread,
Aug 13, 2009, 6:00:17 AM8/13/09
to
Ben Finney <ben+p...@benfinney.id.au> writes:

> My first thought was perhaps it's related to the fact that the process
> has no stdout? But setting ‘stdout’ and ‘stderr’ to the same file::
>
> #! /usr/bin/python
>
> import daemon
> import subprocess
>
> fake_console = open("fake_console.txt", "w+")
> daemon.DaemonContext(stdout=fake_console, stderr=fake_console).open()
> subprocess.Popen(['echo', '1']).wait()

For newcomers to this thread: the ‘daemon’ module is provided by the
<URL:http://pypi.python.org/pypi/python-daemon> ‘python-daemon’
distribution, of which I am the maintainer.

> still gives the same error::
>
> 1
> Traceback (most recent call last):
> File "/home/bignose/Projects/python/python-daemon/bin/andy-clegg-test", line 8, in <module>
> subprocess.Popen(['echo', '1']).wait()
> File "/usr/lib/python2.5/subprocess.py", line 1184, in wait
> pid, sts = self._waitpid_no_intr(self.pid, 0)
> File "/usr/lib/python2.5/subprocess.py", line 1014, in _waitpid_no_intr
> return os.waitpid(pid, options)
> OSError: [Errno 10] No child processes
>
> I'm not familiar enough with the nuances of the ‘subprocess’ module to
> know what might be going wrong here. I'd like to know whether it might
> be a problem in the ‘python-daemon’ library.

I am no closer to a solution on this one. I don't have enough experience
or knowledge of the ‘subprocess’ module to know whether it's a bug in
‘subprocess’, a bug in ‘python-daemon’, or a hard-to-track interaction
between the two.

Any narrowing of the problem would be appreciated.

--
\ “I bought a dog the other day. I named him Stay. It's fun to |
`\ call him. ‘Come here, Stay! Come here, Stay!’ He went insane. |
_o__) Now he just ignores me and keeps typing.” —Steven Wright |
Ben Finney

Sewar

unread,
Sep 5, 2009, 11:54:09 PM9/5/09
to pytho...@python.org
I got the same bug.

Traceback (most recent call last):

File "./script1.py", line 30, in <module>
call(["python", "script2.py", "arg1"], stdout=sys.stdout, stderr=STDOUT)
File "/usr/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 1123, in wait
pid, sts = os.waitpid(self.pid, 0)

Sewar

unread,
Sep 8, 2009, 1:12:34 PM9/8/09
to pytho...@python.org
I looked at other daemon libraries and snippets, it's clearly the bug is in
subprocess not python-daemon.
Then I found Python bug #1731717 which discusses it.

I wish my project was opensource so I can post more specific test cases.

#1731717 http://bugs.python.org/issue1731717

Thanks

Ben Finney

unread,
Sep 8, 2009, 6:19:33 PM9/8/09
to
Sewar <xse...@gmail.com> writes:

> I looked at other daemon libraries and snippets, it's clearly the bug is in
> subprocess not python-daemon.
> Then I found Python bug #1731717 which discusses it.

Thank you very much! I'm glad to see this is a known issue and that some
investigation has already been done. (It's a bit depressing that the bug
is over two years old, though.)

> I wish my project was opensource so I can post more specific test
> cases.

Can you make a small, complete test case that shows the problem? You
could then post that to the bug report, since it seems they are
currently without a good test case for this bug in Python 2.6.

(Discussing the details further in this thread would be
counter-productive; the discussion should go to the bug report.)

Thanks again for finding this!

--
\ “I was sleeping the other night, alone, thanks to the |
`\ exterminator.” —Emo Philips |
_o__) |
Ben Finney

Joel Martin

unread,
Sep 16, 2009, 5:32:14 PM9/16/09
to
On Sep 8, 5:19 pm, Ben Finney <ben+pyt...@benfinney.id.au> wrote:

> Sewar <xsew...@gmail.com> writes:
> > I looked at other daemon libraries and snippets, it's clearly the bug is in
> > subprocess not python-daemon.
> > Then I found Python bug #1731717 which discusses it.

I'm running python-2.6.2 which supposedly has the fix for #1731717.
However I still still the problem with subprocess after daemonizing.
I've narrowed it down to just the setting of the SIGCLD signal.

You can reproduce the problem thus (in 2.4.6, 2.5.4 and 2.6.2):

import subprocess, signal
signal.signal(signal.SIGCLD, signal.SIG_IGN)
subprocess.Popen(['echo','foo']).wait()


This patch allows python-daemon to work with subprocess:

--- daemon.py (revision 146)
+++ daemon.py (working copy)
@@ -140,8 +140,6 @@
actually defined in the ``signal`` module will appear in
the
default map:

- * ``signal.SIGCLD``: ``None``
-
* ``signal.SIGTTIN``: ``None``

* ``signal.SIGTTOU``: ``None``
@@ -730,7 +728,6 @@

"""
name_map = {
- 'SIGCLD': None,
'SIGTSTP': None,
'SIGTTIN': None,
'SIGTTOU': None,

Ben Finney

unread,
Sep 16, 2009, 7:07:39 PM9/16/09
to
Joel Martin <nos...@martintribe.org> writes:

> I'm running python-2.6.2 which supposedly has the fix for #1731717.
> However I still still the problem with subprocess after daemonizing.
> I've narrowed it down to just the setting of the SIGCLD signal.
>
> You can reproduce the problem thus (in 2.4.6, 2.5.4 and 2.6.2):
>
> import subprocess, signal
> signal.signal(signal.SIGCLD, signal.SIG_IGN)
> subprocess.Popen(['echo','foo']).wait()

This would be good to have in the bug report; please add this test case
to <URL:http://bugs.python.org/issue1731717> so the responsible parties
can reproduce the problem as you've shown.

> This patch allows python-daemon to work with subprocess:

Thank you, this *really* helps narrow down the problem. I don't know if
I'll simply be removing the handling as you suggest, but it certainly
makes the range of solutions much clearer.

--
\ “The process by which banks create money is so simple that the |
`\ mind is repelled.” —John Kenneth Galbraith, _Money: Whence It |
_o__) Came, Where It Went_, 1975 |
Ben Finney

Ben Finney

unread,
Sep 16, 2009, 11:21:32 PM9/16/09
to
Ben Finney <ben+p...@benfinney.id.au> writes:

> I'm not familiar enough with the nuances of the ‘subprocess’ module to
> know what might be going wrong here. I'd like to know whether it might
> be a problem in the ‘python-daemon’ library.

My test case for this is now::

=====
#! /usr/bin/python

import daemon
import os
import sys
import subprocess

fake_console = open("fake_console.txt", "w+")
daemon.DaemonContext(stdout=fake_console, stderr=fake_console).open()

sys.stdout.write("Parent daemon process.\n")
os.system("echo Child process via 'os.system'.")
subprocess.Popen(["echo", "Child process via 'subprocess.Popen'."]).wait()
sys.stdout.write("Parent daemon process done.\n")
=====

causing (in the ‘fake_console.txt’ output file)::

=====
Child process via os.system.
Child process via 'subprocess.Popen'.
Parent daemon process.


Traceback (most recent call last):

File "/home/bignose/Projects/python/python-daemon/python-daemon.devel/bin/clegg-example", line 13, in <module>
subprocess.Popen(["echo", "Child process via 'subprocess.Popen'."]).wait()


File "/usr/lib/python2.5/subprocess.py", line 1184, in wait
pid, sts = self._waitpid_no_intr(self.pid, 0)
File "/usr/lib/python2.5/subprocess.py", line 1014, in _waitpid_no_intr
return os.waitpid(pid, options)
OSError: [Errno 10] No child processes

=====


Ben Finney <ben+p...@benfinney.id.au> writes:

> Joel Martin <nos...@martintribe.org> writes:
>
> > I'm running python-2.6.2 which supposedly has the fix for #1731717.
> > However I still still the problem with subprocess after daemonizing.
> > I've narrowed it down to just the setting of the SIGCLD signal.

[…]

> Thank you, this *really* helps narrow down the problem. I don't know
> if I'll simply be removing the handling as you suggest, but it
> certainly makes the range of solutions much clearer.

I'm also glad to see a test case that causes exactly the same error with
or without the presence of a ‘daemon.DaemonContext’.

Further research shows that handling of ‘SIGCLD’ (or ‘SIGCLD’) is fairly
OS-specific, with “ignore it” or “handle it specifically” being correct
on different systems. I think Python's default handling of this signal
is already good (modulo bug #1731717 to be addressed in ‘subprocess’).

So I will apply a change similar to Joel Martin's suggestion, to default
to avoid touching the ‘SIGCLD’ signal at all, and with extra notes in
the documentation that anyone using child processes needs to be wary of
signal handling.

This causes the above test case to succeed; the output file contains::

=====
Child process via os.system.
Child process via 'subprocess.Popen'.
Parent daemon process.
Parent daemon process done.
=====

--
\ “Reality must take precedence over public relations, for nature |
`\ cannot be fooled.” —Richard P. Feynman |
_o__) |
Ben Finney

Reply all
Reply to author
Forward
0 new messages