Bug#920913: fakeroot hangs when nofile limit is large

[Dan Martins]

Package: fakeroot
Version: 1.23

On systems where the nofile limit is large, fakeroot hangs
For example on an Arch Linux system, where DefaultLimitNOFILE=infinity
in /etc/systemd/system.conf:

$ ulimit -Hn
1073741816
$ ulimit -Sn
1073741816
$ cat /proc/sys/fs/file-max
18446744073709551615
$ cat /proc/sys/fs/nr_open
1073741816

$ fakeroot /bin/bash
...
Several minutes pass
...
fakeroot shell is presented:
[root@lab1 ~]#


End users do not expect this long delay and kill the process before
then.

The long delay appears to be in this section of faked.c, where
getdtablesize returns 1073741816

========================================================
From line 1486:

if(!foreground){
/* literally copied from the linux klogd code, go
to background */
if ((pid=fork()) == 0){
int fl;
int
num_fds = getdtablesize();

fflush(stdout);

/* This is the child closing its file descriptors. */
for (fl= 0; fl <= num_fds; ++fl)
#ifdef FAKEROOT_FAKENET
if (fl != sd)
#endif /* FAKEROOT_FAKENET */
close(fl);
=========================================================

[Clint Adams]

On Wed, Jan 30, 2019 at 09:51:42AM -0500, Dan Martins wrote:
> On systems where the nofile limit is large, fakeroot hangs
> For example on an Arch Linux system, where DefaultLimitNOFILE=infinity
> in /etc/systemd/system.conf:

Any idea how other programs are being patched to cope with this?

[Dan Martins]

No.. Initially, I was hoping you might have some idea.
After digging some more, I found a few reports in the debian bug
tracker for other software (kdeinit, mount.nfs) having similar issues.
Things in debian seem a bit more complicated due to a patched
pam_limits, but the workaround appears to be limiting the hard limit
via systemd or pam. So infinity doesn't really mean infinity.

I didn't find any reports where the program itself was patched to
handle this better.

https://github.com/systemd/systemd/issues/10921
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917167
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917168
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917357

It may be this commit
https://salsa.debian.org/systemd-team/systemd/commit/084e84e33a403868b7f84159da745689e2ff0ba9

Which on a debian system appears to limit nr_open
$ cat /proc/sys/fs/nr_open
1048576

And in turn caps the hard limit, even when 'infinity' is used in the
systemd configuration:
ulimit -Hn
1048576

With this limit, fakeroot operates without significant delay.

Arch typically doesn't patch upstream so I think it is unlikely they
will take a similar approach. With that said, I'm not aware of a real
reason to set DefaultLimitNOFILE to infinity.

[Bernhard Übelacker]

Hello all,

> Any idea how other programs are being patched to cope with this?

This issue sounds similar to following older bugs, but
the actions taken there might not be applicable here.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917293
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=917167

Kind regards,
Bernhard

[Julian Andres Klode]

Here is the patch to make it use /proc/self/fd, similar to what we do in apt,
and other software does. This uses dirfd to avoid closing the directory
fd under us while iterating over it; an alternative can be to just not
use close() but instead fcntl(fl, F_SETFD, FD_CLOEXEC) such that the
fds will all be closed by the time you exec - which is the concrete
approach taken in apt.

You'll have to see what works better for you in terms of standard
compliance; dirfd was a BSD 4.3 extension standardised only in
POSIX.1-2008.

/proc/self/fd itself is implemented on Linux, and I guess BSDs
also have a /proc compat layer, but it's been a while since I looked
into it.

Neither approach of closing fds is POSIX compliant, given that the
system may open and require arbitrary descriptors, but oh well, in
practice it works fine, and there is no approach that is compliant :D
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en

[Olliver Schinagl]

I've been using a docker container (either debian or alpine based) to
build openwrt on my Arch system :)

OpenWRT pulls and builds fakeroot to do stuff with, but both containers
choke in some form.

The debian based container launches 1 faked 100% process at a time, and
takes forever to do things (but finishes eventually). A 'build' takes
about 30 - 60 minutes.


```

buildbot@99419141cf36:/workdir$ cat /proc/sys/fs/file-max
9223372036854775807
buildbot@99419141cf36:/workdir$ cat /proc/sys/fs/nr_open
1073741816
buildbot@99419141cf36:/workdir$ ulimit
unlimited
buildbot@99419141cf36:/workdir$ ulimit -Hn
1073741816
buildbot@99419141cf36:/workdir$
```

On alpine (the same numbers) faked also gets lauched, and also takes
100% CPU on a single core, but for some reason on alpine I see dozens of
faked processes launched. Not sure how they related. Builds finish in 10
or so minutes, so at least that's not so bad.


On my host, with systemd, I get far different numbers


```

% cat /proc/sys/fs/file-max
9223372036854775807
% cat /proc/sys/fs/nr_open
1073741816
%  ulimit -Hn
524288
% ulimit
unlimited

```

I'll find out if I can set ulimit during container creation/start to
workaround the issue, but it's still quite annoying.

The reason for the ping/follow up? There where some good suggestions in
this thread earlier, as having to close 524288 file descriptors isn't as
bad as a billion, its still quite a lot of wasted resources for nothing.
Think of the tree's ;)


Olliver