Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Passing File Descriptors To Subprocesses
827 views
Skip to first unread message
Lawrence D’Oliveiro
Jul 16, 2016, 9:59:58 PM7/16/16
to
A few years ago I wrote a tool <https://github.com/ldo/apitizer> to do comparisons between different versions of the Android API. Then one day, after a Python update (from the dates, it had to have been some version of 3.2), it stopped working. It took quite a while--over a year--until I figured out what was happening, and I could move my code from Python 2 back to Python 3 again.
The various subprocess functions <https://docs.python.org/3/library/subprocess.html> have arguments called “close_fds” and “pass_fds”, which specify which file descriptors are to be left open in the child process. Yet no matter what I set these to, it seemed I could not pass my pipes to a subprocess.
What the docs *don’t* tell you is that these arguments do not control what happens after the exec. The file descriptors that are kept open are only those which do not have the FD_CLOEXEC flags set in their fcntl settings.
Remember how the subprocess functions work:
* first, a fork(2) call is executed, then
* the child process does execve(2) (or some convenience variant of this) to actually execute the command or program that you specified.
The “close_fds” and “pass_fds” args only matter in the first step. You need to specify these, *and* have the right CLOEXEC settings on those file descriptors for the second step. Leave out either one, and your child process does not get the file descriptors.
Python 3.4 has added special calls <https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors> to manage this CLOEXEC setting. My code uses fcntl <https://docs.python.org/3/library/fcntl.html>.
The docs for the subprocess module need to make this requirement clear.
The various subprocess functions <https://docs.python.org/3/library/subprocess.html> have arguments called “close_fds” and “pass_fds”, which specify which file descriptors are to be left open in the child process. Yet no matter what I set these to, it seemed I could not pass my pipes to a subprocess.
What the docs *don’t* tell you is that these arguments do not control what happens after the exec. The file descriptors that are kept open are only those which do not have the FD_CLOEXEC flags set in their fcntl settings.
Remember how the subprocess functions work:
* first, a fork(2) call is executed, then
* the child process does execve(2) (or some convenience variant of this) to actually execute the command or program that you specified.
The “close_fds” and “pass_fds” args only matter in the first step. You need to specify these, *and* have the right CLOEXEC settings on those file descriptors for the second step. Leave out either one, and your child process does not get the file descriptors.
Python 3.4 has added special calls <https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors> to manage this CLOEXEC setting. My code uses fcntl <https://docs.python.org/3/library/fcntl.html>.
The docs for the subprocess module need to make this requirement clear.
eryk sun
Jul 16, 2016, 11:01:36 PM7/16/16
to
On Sun, Jul 17, 2016 at 1:59 AM, Lawrence D’Oliveiro
<lawren...@gmail.com> wrote:
> The various subprocess functions <https://docs.python.org/3/library/subprocess.html> have
> arguments called “close_fds” and “pass_fds”, which specify which file descriptors are to be
> left open in the child process. Yet no matter what I set these to, it seemed I could not pass
> my pipes to a subprocess.
>
> What the docs *don’t* tell you is that these arguments do not control what happens after the
> exec. The file descriptors that are kept open are only those which do not have the
> FD_CLOEXEC flags set in their fcntl settings.
It works correctly in 3.4+, which makes the pass_fds file descriptors
<lawren...@gmail.com> wrote:
> The various subprocess functions <https://docs.python.org/3/library/subprocess.html> have
> arguments called “close_fds” and “pass_fds”, which specify which file descriptors are to be
> left open in the child process. Yet no matter what I set these to, it seemed I could not pass
> my pipes to a subprocess.
>
> What the docs *don’t* tell you is that these arguments do not control what happens after the
> exec. The file descriptors that are kept open are only those which do not have the
> FD_CLOEXEC flags set in their fcntl settings.
inheritable in the child, after fork. See issue 18571 and PEP 446,
section "Other Changes":
http://bugs.python.org/issue18571
https://www.python.org/dev/peps/pep-0446/#other-changes
For example:
Python 3.5.1+ (default, Mar 30 2016, 22:46:26)
[GCC 5.3.1 20160330] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, subprocess
>>> fdr, fdw = os.pipe()
>>> fdw
4
>>> os.get_inheritable(fdw)
False
>>> subprocess.call(['python3'], pass_fds=[fdw])
child:
Python 3.5.1+ (default, Mar 30 2016, 22:46:26)
[GCC 5.3.1 20160330] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.write(4, b'spam')
4
>>> exit()
parent:
0
>>> os.read(fdr, 4)
b'spam'
Lawrence D’Oliveiro
Jul 17, 2016, 3:06:21 AM7/17/16
to
On Sunday, July 17, 2016 at 3:01:36 PM UTC+12, eryk sun wrote:
> It works correctly in 3.4+ ...
Yup, confirmed it happens only in 3.3.
> It works correctly in 3.4+ ...
Yup, confirmed it happens only in 3.3.
pev...@gmail.com
May 11, 2018, 11:23:50 AM5/11/18
to
I encountered the same issue with Python 3.4 on CentOS 7 when using only the close_fds argument. Since I am passing a lot of dynamically obtained file descriptors, using the pass_fds argument is impossible for my case. Setting close_fds to False *but* also explicitly making the fds inheritable upon generation solved the issue. I would be really grateful if there was at least a note in the subprocess module explaining that one might need to explicitly make file descriptors inheritable since if it wasn't this post I would have lost even more time than I already did.
Terry Reedy
May 11, 2018, 5:15:16 PM5/11/18
to
On 5/11/2018 11:23 AM, pev...@gmail.com wrote:
> I encountered the same issue with Python 3.4 on CentOS 7 when using only the close_fds argument. Since I am passing a lot of dynamically obtained file descriptors, using the pass_fds argument is impossible for my case. Setting close_fds to False *but* also explicitly making the fds inheritable upon generation solved the issue. I would be really grateful if there was at least a note in the subprocess module explaining that one might need to explicitly make file descriptors inheritable since if it wasn't this post I would have lost even more time than I already did.
Make specific proposal on the tracker bugs.python.org.
> I encountered the same issue with Python 3.4 on CentOS 7 when using only the close_fds argument. Since I am passing a lot of dynamically obtained file descriptors, using the pass_fds argument is impossible for my case. Setting close_fds to False *but* also explicitly making the fds inheritable upon generation solved the issue. I would be really grateful if there was at least a note in the subprocess module explaining that one might need to explicitly make file descriptors inheritable since if it wasn't this post I would have lost even more time than I already did.
--
Terry Jan Reedy
0 new messages