fork/exec & close file descriptors

[Skip Montanaro]

Due to presumed bugs in an underlying library over which I have no control, I'm considering a restart in the wee hours of the morning. The basic fork/exec dance is not a problem, but how do I discover all the open file descriptors in the new child process to make sure they get closed? Do I simply start at fd 3 and call os.close() on everything up to some largish fd number? Some of these file descriptors will have been opened by stuff well below the Python level, so I don't know them a priori.

Thx,

Skip

[Chris Angelico]

What Python version are you targeting? Are you aware of PEP 446?

https://www.python.org/dev/peps/pep-0446/

tl;dr: As of Python 3.4, the default is to close file descriptors on
exec automatically - and atomically, where possible.

I'm not sure if there's a 2.7 backport available, though even if there
is, you'll need to manually set your files non-inheritable; AIUI the
default in 2.7 can't be changed for backward compatibility reasons
(the change in 3.4 *will* break code that was relying on automatic
inheritability of FDs - they now have to be explicitly tagged).

ChrisA

[Skip Montanaro]

On Tue, May 19, 2015 at 8:33 AM, Chris Angelico <ros...@gmail.com> wrote:

What Python version are you targeting? Are you aware of PEP 446?

Yeah, I'm still on 2.7, and am aware of PEP 446. Note that many of the file descriptors will not have been created by my Python code. They will have been created by underlying C/C++ libraries, so I can't guarantee which flags were set on file open.

I'm going to continue to pursue solutions which won't require a restart for now, but would like to have a sane restart option in my back pocket should it become necessary.

Thx,

Skip

[Chris Angelico]

Fair enough. What you MAY be able to do is preempt it by going through
your FDs and setting them all CLOEXEC, but it won't make a lot of
difference compared to just going through them all and closing them
between fork and exec.

On Linux (and possibly some other Unixes), /proc/self/fd may be of
use. Enumerating files in that should tell you about your open files.
How useful that is I don't know, though.

ChrisA

[Skip Montanaro]

On Tue, May 19, 2015 at 8:54 AM, Chris Angelico <ros...@gmail.com> wrote:

On Linux (and possibly some other Unixes), /proc/self/fd may be of
use.

Good point. Yes, /proc/PID/fd appears to contain all the entries for open file descriptors (I am on Linux).

Skip

[Jon Ribbens]

On 2015-05-19, Skip Montanaro <skip.mo...@gmail.com> wrote:
> Yeah, I'm still on 2.7, and am aware of PEP 446. Note that many of the file
> descriptors will not have been created by my Python code. They will have
> been created by underlying C/C++ libraries, so I can't guarantee which
> flags were set on file open.

There is no portable way to do this, the problem is Unix not Python.
The below code is a reasonable stab at it, but there is no 100%
guaranteed method. The code is untested but you get the idea.


import errno
import os


def closeall(min=0, max=4096, keep=frozenset()):
"""Close all open file descriptors except for the given exceptions.

Any file descriptors below or equal to `min`, or in the set `keep`
will not be closed. Any file descriptors above `max` *might* not be
closed.
"""
# First try /proc/$$/pid
try:
for fd in os.listdir("/proc/%d/fd" % (os.getpid())):
try:
fd = int(fd)
except ValueError:
continue
if fd >= min and fd not in keep:
os.close(int(fd))
return
except OSError as exc:
if exc[0] != errno.ENOENT:
raise
# If /proc was not available, fall back to closing a lot of descriptors.
for fd in range(min, max):
if fd not in keep:
try:
os.close(fd)
except OSError as exc:
if exc[0] != errno.EBADF:
raise

[Chris Angelico]

Yes, and /proc/self is usually a magic symlink to /proc/<your_pid> so
you can just look at /proc/self/fd instead of explicitly calling up
your own PID.

ChrisA

[Ethan Furman]

On 05/19/2015 05:59 AM, Skip Montanaro wrote:

> Due to presumed bugs in an underlying library over which I have no control, I'm considering a restart in the wee hours of the morning. The basic fork/exec dance is not a problem, but how do I discover
> all the open file descriptors in the new child process to make sure they get closed? Do I simply start at fd 3 and call os.close() on everything up to some largish fd number? Some of these file
> descriptors will have been opened by stuff well below the Python level, so I don't know them a priori.

Pandaemonium [1] (and I believe Ben Finney's daemon [2]) use something akin to the following:

def close_open_files(exclude):
max_files = resource.getrlimit(resource.RLIMIT_NOFILE)[1]
keep = set()
for file in exclude:
if isinstance(file, baseint):
keep.add(file)
elif hasattr(file, 'fileno'):
keep.add(file.fileno())
else:
raise ValueError(
'files to not close should be either an file descriptor, '
'or a file-type object, not %r (%s)' % (type(file), file))
for fd in range(max_files, -1, -1):
if fd in keep:
continue
try:
os.close(fd)
except OSError:
exc = sys.exc_info()[1]
if exc.errno == errno.EBADF:
continue
raise

So, yeah, basically a brute-force method.

--
~Ethan~


[1] https://pypi.python.org/pypi/pandaemonium
[2] https://pypi.python.org/pypi/python-daemon

[Gregory Ewing]

> On Tue, May 19, 2015 at 8:54 AM, Chris Angelico <ros...@gmail.com
> <mailto:ros...@gmail.com>> wrote:
>
> On Linux (and possibly some other Unixes), /proc/self/fd may be of
> use.

On MacOSX, /dev/fd seems to be the equivalent of this.

--
Greg

[Ian Kelly]

Not a perfect equivalent. On Linux, ls -lF /proc/self/fd shows the
contents as symlinks, which is handy since you can just read the links
to see what they're pointing to. On OSX, ls -lF /dev/fd shows three
ttys and two directories.

Though I also note that on my Ubuntu Trusty system, /dev/fd is itself
a symlink to /proc/self/fd.

[Skip Montanaro]

Reviving (and concluding) a thread I started a couple weeks ago, I asked:

> The basic fork/exec dance is not a problem, but how do I discover
> all the open file descriptors in the new child process to make sure
> they get closed? Do I simply start at fd 3 and call os.close() on
> everything up to some largish fd number?

I wanted this again today (for different reasons than before).
Googling for "python close all file descriptors" returned the os
module docs as the first hit, and lo and behold, what do I see
documented? os.closerange (new in 2.6):

os.closerange(fd_low, fd_high)
Close all file descriptors from fd_low (inclusive) to fd_high
(exclusive), ignoring errors.

Guido's time machine strikes again...

Skip

[Marko Rauhamaa]

Skip Montanaro <skip.mo...@gmail.com>:

> os.closerange(fd_low, fd_high)
> Close all file descriptors from fd_low (inclusive) to fd_high
> (exclusive), ignoring errors.
>
> Guido's time machine strikes again...

The only problem is that you don't know how high you need to go in
general.


Marko

[Skip Montanaro]

On Tue, Jun 2, 2015 at 10:28 AM, Marko Rauhamaa <ma...@pacujo.net> wrote:
>
> The only problem is that you don't know how high you need to go in
> general.

Sure, but I didn't know anyway, so no matter what upper bound I choose
(or what function I choose/implement), it's just going to be a guess.
os.closerange just codifies the straightforward procedure.

Skip

[Alain Ketterlin]

The close(2) manpage has the following warning on my Linux system:

| Not checking the return value of close() is a common but nevertheless
| serious programming error. It is quite possible that errors on a pre‐
| vious write(2) operation are first reported at the final close(). Not
| checking the return value when closing the file may lead to silent loss
| of data. This can especially be observed with NFS and with disk quota.
|

(I haven't followed the thread, but if your problem is to make sure fds
are closed on exec, you may be better off using the... close-on-exec
flag. Or simply do the bookkeeping.)

-- Alain.

[Marko Rauhamaa]

Skip Montanaro <skip.mo...@gmail.com>:

Under linux, the cleanest way seems to be going through the files under
/proc/self/fd:

def close_fds(leave_open=[0, 1, 2]):
fds = os.listdir(b'/proc/self/fd')
for fdn in fds:
fd = int(fdn)
if fd not in leave_open:
os.close(fd)

No need for a upper bound.


Marko

[Jon Ribbens]

Or use the more generic code that I already posted in this thread:

def closeall(min=0, max=4096, keep=()):

"""Close all open file descriptors except for the given exceptions.

Any file descriptors below or equal to `min`, or in the set `keep`
will not be closed. Any file descriptors above `max` *might* not be
closed.
"""

# First try /proc/self/pid
try:
for fd in os.listdir("/proc/self/fd"):

try:
fd = int(fd)
except ValueError:
continue
if fd >= min and fd not in keep:
os.close(int(fd))
return
except OSError as exc:
if exc[0] != errno.ENOENT:
raise
# If /proc was not available, fall back to closing a lot of descriptors.
for fd in range(min, max):
if fd not in keep:
try:
os.close(fd)
except OSError as exc:
if exc[0] != errno.EBADF:
raise

This function could use os.closerange(), but if the documentation is
correct and it ignores *all* errors and not just EBADF, then it
sounds like os.closerange() should not in fact ever be used for any
purpose.

[Marko Rauhamaa]

Alain Ketterlin <al...@universite-de-strasbourg.fr.invalid>:

> The close(2) manpage has the following warning on my Linux system:
>
> | Not checking the return value of close() is a common but
> | nevertheless serious programming error. It is quite possible that

> | errors on a previous write(2) operation are first reported at the

> | final close(). Not checking the return value when closing the file
> | may lead to silent loss of data. This can especially be observed
> | with NFS and with disk quota.
> |
>
> (I haven't followed the thread, but if your problem is to make sure
> fds are closed on exec, you may be better off using the...
> close-on-exec flag. Or simply do the bookkeeping.)

The quoted man page passage is a bit untenable.

First, if close() fails, what's a poor program to do? Try again? How
do you get rid of an obnoxious file descriptor? How would close-on-exec
help? Would exec*() fail?

What if an implicit close() fails on _exit(), will _exit() fail then?
(The man page doesn't allow it.)

The need to close all open file descriptors comes between fork() and
exec*(). The kernel (module) does not see the close() system call unless
the reference count drops to zero. Normally, those function calls
between fork() and exec*() are therefore no-ops.

However, there's no guarantee of that. So the parent process might get
to call close() before the child that is about to call exec*(). Then,
the parent would not get the error that the man page talks about.
Instead, the error goes to the child, which has no reasonable way of
dealing with the situation.

I think having NFS et al postpone their I/O errors till close() is
shifting the blame to the victim.


Marko

[Alain Ketterlin]

Marko Rauhamaa <ma...@pacujo.net> writes:

> Alain Ketterlin <al...@universite-de-strasbourg.fr.invalid>:
>
>> The close(2) manpage has the following warning on my Linux system:
>>
>> | Not checking the return value of close() is a common but
>> | nevertheless serious programming error. It is quite possible that
>> | errors on a previous write(2) operation are first reported at the
>> | final close(). Not checking the return value when closing the file
>> | may lead to silent loss of data. This can especially be observed
>> | with NFS and with disk quota.
>> |
>>
>> (I haven't followed the thread, but if your problem is to make sure
>> fds are closed on exec, you may be better off using the...
>> close-on-exec flag. Or simply do the bookkeeping.)
>
> The quoted man page passage is a bit untenable.
>
> First, if close() fails, what's a poor program to do?

Warn the user? Not assume everything went well? It all depends on the
application, and what the file descriptor represents.

> Try again?

Could be a good idea on NFS or other kind of mounts.

> How do you get rid of an obnoxious file descriptor?

You don't, you check everything before closing the file, with fsync()
for example.

I've no idea what the OP's program was doing, so I'm not going to split
hairs. I can't imagine why one would like to mass-close an arbitrary set
of file descriptors, and I think APIs like os.closerange() are toxic and
an appeal to sloppy programming.

-- Alain.

[Marko Rauhamaa]

Alain Ketterlin <al...@universite-de-strasbourg.fr.invalid>:

> Marko Rauhamaa <ma...@pacujo.net> writes:
>> First, if close() fails, what's a poor program to do?
>
> Warn the user? Not assume everything went well? It all depends on the
> application, and what the file descriptor represents.

The problem here is in the system call contract, which is broken.
There's no fix. The man page admonition is just hand-waving without
constructive advice.

>> Try again?
> Could be a good idea on NFS or other kind of mounts.

Maybe close() will fail for ever.

> I can't imagine why one would like to mass-close an arbitrary set of
> file descriptors,

That's standard practice before execking a file. Failure to do that can
seriously hurt the parent process. For example, the parent (or child)
will never read an EOF from file descriptor if its duplicate is open in
an unwitting child process. Also, the number of open files in the system
may grow over all limits or simply waste kernel resources.

Close-on-exec is nice, maybe. However, you don't have control over all
file descriptors. Loggers, high-level library calls and others open
files without the application programmer knowing or having direct
control over.

> and I think APIs like os.closerange() are toxic and an appeal to
> sloppy programming.

And you recommend what instead?


Marko

[Chris Angelico]

On Wed, Jun 3, 2015 at 7:06 AM, Alain Ketterlin
<al...@universite-de-strasbourg.fr.invalid> wrote:
> I've no idea what the OP's program was doing, so I'm not going to split
> hairs. I can't imagine why one would like to mass-close an arbitrary set
> of file descriptors, and I think APIs like os.closerange() are toxic and
> an appeal to sloppy programming.

When you fork, you get a duplicate referent to every open file in both
parent and child. Closing them all in the child is very common, as it
allows the parent to continue owning those file descriptors (so that
when you close it in the parent, the resource is really closed). One
notable example is with listening sockets; bind/listen in the parent,
then fork (maybe to handle a client), then terminate the parent
process. You now cannot restart the parent without aborting the child,
as the child now owns that listening socket (even if it never wants to
use it). There are some specific ways around this, but not on all OSes
(eg Linux only added support for SO_REUSEPORT in 3.9), and the best
way has always been to make sure the children don't hang onto the
listening socket. (There are other good reasons for doing this, too.)

ChrisA

[Skip Montanaro]

Folks, I'm not interested in rehashing this. I don't know what all the
open file descriptors are. Some/many/most will have been opened by
underlying C++ libraries and will not have been opened by code at the
Python level. I do think the addition of an os.fsync() attempt on each
file descriptor before calling os.close() makes some sense to me.
Beyond that, I'm not sure there's more that can be done.

I sent my last message simply because I was surprised os.closerange()
existed. I have no idea why it didn't turn up in my original
searching.

Skip

[Alain Ketterlin]

Chris Angelico <ros...@gmail.com> writes:

> On Wed, Jun 3, 2015 at 7:06 AM, Alain Ketterlin
> <al...@universite-de-strasbourg.fr.invalid> wrote:
>> I've no idea what the OP's program was doing, so I'm not going to split
>> hairs. I can't imagine why one would like to mass-close an arbitrary set
>> of file descriptors, and I think APIs like os.closerange() are toxic and
>> an appeal to sloppy programming.
>
> When you fork, you get a duplicate referent to every open file in both

> parent and child. [...]

Thank you, I know this. What I mean is: what are the reasons that you
cannot access your file descriptors one by one? To me closing a range of
descriptors has absolutely no meaning, simply because ranges have no
meaning for file descriptors (they're not ordered in any way). What if
some library uses its own descriptors that happen to lie in your
"range"? Etc.

-- Alain.

[Alain Ketterlin]

Marko Rauhamaa <ma...@pacujo.net> writes:

> Alain Ketterlin <al...@universite-de-strasbourg.fr.invalid>:
>
>> Marko Rauhamaa <ma...@pacujo.net> writes:
>>> First, if close() fails, what's a poor program to do?
>>
>> Warn the user? Not assume everything went well? It all depends on the
>> application, and what the file descriptor represents.
>
> The problem here is in the system call contract, which is broken.
> There's no fix. The man page admonition is just hand-waving without
> constructive advice.
>
>>> Try again?
>> Could be a good idea on NFS or other kind of mounts.
>
> Maybe close() will fail for ever.

Your program has to deal with this, something is going wrong, it can't
just close and go on.

>> I can't imagine why one would like to mass-close an arbitrary set of
>> file descriptors,
>

> That's standard practice before execking a file. [...]

>> and I think APIs like os.closerange() are toxic and an appeal to
>> sloppy programming.
>
> And you recommend what instead?

Keeping them around and closing the ones you own, plus doing whatever is
necessary to have libraires (loggers, database connectors, etc.) finish
properly. And not taking the risk of messing with descriptors your
program is not responsible of.

-- Alain.

[Marko Rauhamaa]

Alain Ketterlin <al...@universite-de-strasbourg.fr.invalid>:

> Marko Rauhamaa <ma...@pacujo.net> writes:
>> Maybe close() will fail for ever.
>
> Your program has to deal with this, something is going wrong, it can't
> just close and go on.

Here's the deal: the child process is saddled with file descriptors it
never wanted in the first place. It can't decline them. Now you're
saying it can't even dispose of them.

The reason this has been allowed to go on is that everybody just closes
the file descriptors and ignores the possibility or repercussions of a
failure. I haven't read about horror stories of this failing.

I readily admit this is very dirty, but since the API doesn't offer a
clean alternative, there's nothing you can/should do about it.


Marko

[alister]

On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote:

> Alain Ketterlin <al...@universite-de-strasbourg.fr.invalid>:
>
>> Marko Rauhamaa <ma...@pacujo.net> writes:
>>> Maybe close() will fail for ever.
>>
>> Your program has to deal with this, something is going wrong, it can't
>> just close and go on.
>
> Here's the deal: the child process is saddled with file descriptors it
> never wanted in the first place. It can't decline them. Now you're
> saying it can't even dispose of them.
>

No You cab dispose of them you just need to warn the user that the action
did not complete correctly & there may be errors with the data.

Example What does your test editor do if you try to save a file back to a
USB stick that has been removed? does it simply let you think the file
has been successfully saved? i hope not.

> The reason this has been allowed to go on is that everybody just closes
> the file descriptors and ignores the possibility or repercussions of a
> failure. I haven't read about horror stories of this failing.
>
> I readily admit this is very dirty, but since the API doesn't offer a
> clean alternative, there's nothing you can/should do about it.
>
>
> Marko

--
Never be afraid to tell the world who you are.
-- Anonymous

[Steven D'Aprano]

On Wed, 3 Jun 2015 07:38 pm, alister wrote:

> On Wed, 03 Jun 2015 10:41:44 +0300, Marko Rauhamaa wrote:

[...]

>> Here's the deal: the child process is saddled with file descriptors it
>> never wanted in the first place. It can't decline them. Now you're
>> saying it can't even dispose of them.
>>
> No You cab dispose of them you just need to warn the user that the action
> did not complete correctly & there may be errors with the data.

*What* action? Pay closer attention to what Marko wrote:

"the child process ..."

How does the child process know what action didn't complete? What error
message are you going to display to the user?

"Error when closing file descriptor 123456"

What action do you think the user can take on seeing this error message?

> Example What does your test editor do if you try to save a file back to a
> USB stick that has been removed? does it simply let you think the file
> has been successfully saved? i hope not.

Correct me if I'm wrong, but don't you get an error when you try to *open* a
file on a missing USB stick?

There are at least three opportunities to get an error when writing a file:

* opening the file;
* writing to the file;
* closing the file.

It's not clear what conditions will lead to the first two succeeding but the
third failing, or what to do if you get such an error. If you don't know
what caused close() to fail, then you can't do anything about it, and if
you can't do anything about it, what's the point in reporting the error to
the user who will just get frustrated and nervous and not be able to do
anything about it either?

I'm sure that there are circumstances where an error when closing a file is
a clear and obvious fault that should be reported. But in the scenario that
Marko is describing, I'm not so sure that is the case.



--
Steven

[alister]

open file for appending data
write some data
remove mem stick
now try to close file

> then you can't do anything about it,

Correct you cant do anything about it.

> and if you can't do anything about it, what's the point in reporting the
> error to the user who will just get frustrated and nervous and not be
> able to do anything about it either?

Because they need to know they may have just lost important data whilst
they still have a chance to do it again. finding out 6 months later that
they had screwed up saving data on a multi-million pound account will not
be good & quite rightly considered to be due to a bug in your software.

>
> I'm sure that there are circumstances where an error when closing a file
> is a clear and obvious fault that should be reported. But in the
> scenario that Marko is describing, I'm not so sure that is the case.

from the scenario Marco is reporting I get the impression that he is
having to interact with a system that is fundamentally flawed from the
ground up.



--
Standards are crucial. And the best thing about standards is: there are
so ____ many to choose from!

[rand...@fastmail.us]

On Wed, Jun 3, 2015, at 03:11, Alain Ketterlin wrote:
> Thank you, I know this. What I mean is: what are the reasons that you
> cannot access your file descriptors one by one? To me closing a range of
> descriptors has absolutely no meaning, simply because ranges have no
> meaning for file descriptors (they're not ordered in any way). What if
> some library uses its own descriptors that happen to lie in your
> "range"? Etc.

The context in which this is useful is that you've just forked, and
you're about to exec. "Some library" isn't going to ever get back
control within the current process. Generally the range of file
descriptors you want to close is (e.g.) 3-Infinity, after you've already
got 0 1 and 2 pointing to where you want them (whatever redirected file
or pipe).

[Marko Rauhamaa]

Steven D'Aprano <st...@pearwood.info>:

> How does the child process know what action didn't complete? What
> error message are you going to display to the user?
>
> "Error when closing file descriptor 123456"
>
> What action do you think the user can take on seeing this error
> message?

Besides, even if you wanted to display that error message, you might not
be able to since maybe you have unknowingly closed fd 123433, which is
the hidden connection to the display server. So the program might just
crash in the attempt. Or the hidden fd is still open but won't tolerate
two processes sharing the connection.

You definitely must not use sys.stderr from the child process nor are
you allowed to exit through an exception but must call os._exit()
instead.

The child process is highly limited in what it can do between os.fork()
and os.execve(). To put it simply, it must not call any library
functions except system calls.

You might be able to reserve a special pipe to convey such errors to the
parent. Only don't rely on getting an EOF as closing the pipe might
fail...


Marko

[Marko Rauhamaa]

alister <alister.n...@ntlworld.com>:

> from the scenario Marco is reporting I get the impression that he is
> having to interact with a system that is fundamentally flawed from the
> ground up.

Well, yes. It's called linux, but it's not all bad. I just think that
man page was being sanctimonious.


Marko

[rand...@fastmail.us]

On Wed, Jun 3, 2015, at 08:25, Marko Rauhamaa wrote:
> Steven D'Aprano <st...@pearwood.info>:
>
> > How does the child process know what action didn't complete? What
> > error message are you going to display to the user?
>

> You definitely must not use sys.stderr from the child process nor are
> you allowed to exit through an exception but must call os._exit()
> instead.

Why does the child process need to report the error at all? The parent
process will find out naturally when *it* tries to close the same file
descriptor.

[alister]

I meant the program that is supplying your app with file handles willy-
nilly without caring what happens to them



--
"...a most excellent barbarian ... Genghis Kahn!"
-- _Bill And Ted's Excellent Adventure_

[Marko Rauhamaa]

rand...@fastmail.us:

> Why does the child process need to report the error at all? The parent
> process will find out naturally when *it* tries to close the same file
> descriptor.

That's not how it goes.

File descriptors are reference counted in the Linux kernel. Closes are
no-ops except for the last one that brings the reference count to zero.

If the parent should close the file before the child, no error is
returned to the parent.


Marko

[Alain Ketterlin]

rand...@fastmail.us writes:

> On Wed, Jun 3, 2015, at 03:11, Alain Ketterlin wrote:
>> Thank you, I know this. What I mean is: what are the reasons that you
>> cannot access your file descriptors one by one? To me closing a range of
>> descriptors has absolutely no meaning, simply because ranges have no
>> meaning for file descriptors (they're not ordered in any way). What if
>> some library uses its own descriptors that happen to lie in your
>> "range"? Etc.
>
> The context in which this is useful is that you've just forked, and
> you're about to exec. "Some library" isn't going to ever get back
> control within the current process.

Any decent library will (on linux) use close-on-exec, or even have
fini/dtor functions to clean up. Any that does not is buggy. But of
course, if you shoot in their feet...

> Generally the range of file descriptors you want to close is (e.g.)
> 3-Infinity, after you've already got 0 1 and 2 pointing to where you
> want them (whatever redirected file or pipe).

Closing 3-... is meaningless, probably useless, and potentially harmful.

-- Alain.

[rand...@fastmail.us]

Why would the parent close it before the child? Your scenario doesn't
seem to have anything to do with how people actually use subprocesses.

[Chris Angelico]

On Wed, Jun 3, 2015 at 11:21 PM, <rand...@fastmail.us> wrote:
> On Wed, Jun 3, 2015, at 09:08, Marko Rauhamaa wrote:

> Why would the parent close it before the child? Your scenario doesn't
> seem to have anything to do with how people actually use subprocesses.

Write an editor that opens a file and holds it open until the user's
done with it. Have something that lets you shell out for whatever
reason. Then trigger the shell-out, and instantly SIGSTOP the child
process, before it does its work - or just have a really heavily
loaded system, so it can't get a time slice. Now close the file in the
UI, which results in the file being closed in the parent. Right, now
let the child run... and there it goes, closing the file.

Mightn't be a common situation (given that the amount of code executed
between forking and closing FDs is not going to be much, and isn't
going to involve reading from disk or anything), but it can certainly
happen.

ChrisA

[Marko Rauhamaa]

Marko Rauhamaa <ma...@pacujo.net>:

First of all, it's not the descriptors that are refcounted -- it's the
files referred to by the descriptors.

However, I was also wrong about close() being a no-operation. Here's the
relevant kernel source code snippet:

========================================================================
int __close_fd(struct files_struct *files, unsigned fd)
{
struct file *file;
struct fdtable *fdt;

spin_lock(&files->file_lock);
fdt = files_fdtable(files);
if (fd >= fdt->max_fds)
goto out_unlock;
file = fdt->fd[fd];
if (!file)
goto out_unlock;
rcu_assign_pointer(fdt->fd[fd], NULL);
__clear_close_on_exec(fd, fdt);
__put_unused_fd(files, fd);
spin_unlock(&files->file_lock);
return filp_close(file, files);

out_unlock:
spin_unlock(&files->file_lock);
return -EBADF;
}

int filp_close(struct file *filp, fl_owner_t id)
{
int retval = 0;

if (!file_count(filp)) {
printk(KERN_ERR "VFS: Close: file count is 0\n");
return 0;
}

if (filp->f_op->flush)
retval = filp->f_op->flush(filp, id);

if (likely(!(filp->f_mode & FMODE_PATH))) {
dnotify_flush(filp, id);
locks_remove_posix(filp, id);
}
fput(filp);
return retval;
}
========================================================================

What is revealed is that:

1. The file descriptor is released regardless of the return value of
close(2):

__put_unused_fd(files, fd);

2. The file object's refcount is decremented accordingly regardless of
the return value of close(2).

3. The return value reflects the success of the optional flush() method
of the file object.

4. The flush() method is called with each call to close(2), not only
the last one.

IOW, os.close() closes the file even if it should report a failure. I
couldn't have guessed that behavior from the man page.

So the strategy you proposed is the right one: have the child process
ignore any possible errors from os.close(). The parent will have an
opportunity to deal with them.

And now Linux is back in the good graces, only the man page is
misleading.


Marko

[Marko Rauhamaa]

Marko Rauhamaa <ma...@pacujo.net>:

> So the strategy you proposed is the right one: have the child process
> ignore any possible errors from os.close(). The parent will have an
> opportunity to deal with them.
>
> And now Linux is back in the good graces, only the man page is
> misleading.

However, the child process needs to be prepared for os.close() to block
indefinitely because of an NFS problem or because SO_LINGER has been
specified by the parent, for example. Setting the close-on-exec flag
doesn't help there.


Marko

[Marko Rauhamaa]

alister <alister.n...@ntlworld.com>:

> I meant the program that is supplying your app with file handles
> willy- nilly without caring what happens to them

You seem to be advocating a strategy whereby the application keeps close
track of all file descriptors and closes them individually as needed.

Thing is, processes can be forked by library calls. For example, you
might have developed a class that converts a synchronous database API
into an asynchronous one. You'll have your object fork a helper process
that makes blocking calls while the object methods make sure never to
block the caller.

The application doesn't know the library would be starting a child
process. On the other hand, the library has no idea on what files the
application might have open. That's why the library traverses all file
descriptors and closes them categorically after forking.


Marko

[rand...@fastmail.us]

On Wed, Jun 3, 2015, at 09:32, Chris Angelico wrote:
> Write an editor that opens a file and holds it open until the user's
> done with it. Have something that lets you shell out for whatever
> reason. Then trigger the shell-out, and instantly SIGSTOP the child
> process, before it does its work - or just have a really heavily
> loaded system, so it can't get a time slice. Now close the file in the
> UI, which results in the file being closed in the parent. Right, now
> let the child run... and there it goes, closing the file.

The parent should be waiting for the child process. If it shouldn't wait
for the command, then the child process should spawn a grandchild
process, after closing the file descriptors. This is how the text editor
I use actually works (more or less. In fact, the way to run a process it
won't wait for is to run it as a background shell command. If you STOP
the shell itself, the editor will be stuck waiting.)

[rand...@fastmail.us]

On Wed, Jun 3, 2015, at 10:43, Marko Rauhamaa wrote:
> However, the child process needs to be prepared for os.close() to block
> indefinitely because of an NFS problem or because SO_LINGER has been
> specified by the parent, for example. Setting the close-on-exec flag
> doesn't help there.

Out of curiosity, does exec block in this situation?

[Marko Rauhamaa]

rand...@fastmail.us:

I didn't try it, but it is apparent in the source code:

========================================================================
void do_close_on_exec(struct files_struct *files)
{
unsigned i;
struct fdtable *fdt;

/* exec unshares first */
spin_lock(&files->file_lock);
for (i = 0; ; i++) {
unsigned long set;
unsigned fd = i * BITS_PER_LONG;

fdt = files_fdtable(files);
if (fd >= fdt->max_fds)

break;
set = fdt->close_on_exec[i];
if (!set)
continue;
fdt->close_on_exec[i] = 0;
for ( ; set ; fd++, set >>= 1) {
struct file *file;
if (!(set & 1))
continue;

file = fdt->fd[fd];
if (!file)

continue;

rcu_assign_pointer(fdt->fd[fd], NULL);

__put_unused_fd(files, fd);
spin_unlock(&files->file_lock);
filp_close(file, files);
cond_resched();
spin_lock(&files->file_lock);
}

}
spin_unlock(&files->file_lock);

}

int filp_close(struct file *filp, fl_owner_t id)
{
int retval = 0;

if (!file_count(filp)) {
printk(KERN_ERR "VFS: Close: file count is 0\n");
return 0;
}

if (filp->f_op->flush)
retval = filp->f_op->flush(filp, id);

if (likely(!(filp->f_mode & FMODE_PATH))) {
dnotify_flush(filp, id);
locks_remove_posix(filp, id);
}
fput(filp);
return retval;
}
========================================================================

Now, the kernel NFS code specifies a flush() method, which can block and
even fail.

Sockets don't have a flush() method. So closing a socket cannot fail.
However, fput(), which decrements the reference count, may block if
lingering has been specified for the socket.

So I wasn't all that wrong earlier after all: whoever closes a socket
last will linger. Thus, the parent (who wanted to linger) might zip
through closing a socket while the unwitting child process will suffer
the lingering delay before it gets to exec.

However, there's this comment in inet_release():

/* [...]
* If the close is due to the process exiting, we never
* linger..
*/


Marko

[Chris Angelico]

Really? I thought forking, execing, and not immediately waiting, was a
standard way to trigger an asynchronous action. My editor lets me
start something and then keep working in the editor, and see the
output from the command when it's ready. (Simple example: A "git push"
might take a long time if the network's slow, and I want to know if it
errors out, but most likely I'll just see the expected messages come
up and that's that.) The parent definitely shouldn't immediately wait
on the child; and it shouldn't disown the child via a second forking
because it wants to report on the child's completion. So no, I don't
think insta-waiting is right in all situations.

ChrisA

[MrJean1]

The subprocess module uses upper bound MAXFD which is defined as

<code>
try:
MAXFD = os.sysconf("SC_OPEN_MAX")
except:
MAXFD = 256
</code>

/Jean