Doctests leaking file handles

[Tobias Hansen]

Hi,

I am having problems with a file handle leak during doctesting in the sagemath Debian package. This already occurred sporadically with sage 9.0 but with 9.1 (rc2) it happens more consistently. When the file handle limit (ulimit -n) is reached, the remaining tests are killed and one gets this:

Traceback (most recent call last):
  File "/home/thansen/src/sage/sagemath/sagemath/sage/src/bin/sage-runtests", line 178, in <module>
    err = DC.run()
  File "/home/thansen/src/sage/sagemath/sagemath/debian/build/usr/lib/python3/dist-packages/sage/doctest/control.py", line 1227, in run
    self.run_doctests()
  File "/home/thansen/src/sage/sagemath/sagemath/debian/build/usr/lib/python3/dist-packages/sage/doctest/control.py", line 928, in run_doctests
    self.dispatcher.dispatch()
  File "/home/thansen/src/sage/sagemath/sagemath/debian/build/usr/lib/python3/dist-packages/sage/doctest/forker.py", line 2024, in dispatch
    self.parallel_dispatch()
  File "/home/thansen/src/sage/sagemath/sagemath/debian/build/usr/lib/python3/dist-packages/sage/doctest/forker.py", line 1916, in parallel_dispatch
    w.start()  # This might take some time
  File "/home/thansen/src/sage/sagemath/sagemath/debian/build/usr/lib/python3/dist-packages/sage/doctest/forker.py", line 2191, in start
    super(DocTestWorker, self).start()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/context.py", line 276, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 69, in _launch
    child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

The number of tests that run before this is similar to the file handle limit (e.g. I saw 1234 tests for a limit of 1024 and 103 tests for a limit of 100).

I don't know if this comes from our patches, an external program or sage itself. At least our patches change only doctest results in sage/doctest and I didn't find anything suspicious elsewhere. It could also be that it happens only with python 3.8 which is not in sage yet. Has anyone else seen this? Any idea where to look or how to debug this?

Best,

Tobias

[Tobias Hansen]

Hi,

the problem is still there with sage 9.1 (which I pushed to the Debian packaging repo). Since the number of tested files before the error is similar to the file descriptor limit, it seems plausible that each DocTestWorker in sage/doctest/forker.py is leaking one pipe. I saw that there are many open pipes by looking at /proc/$PID/fd/, however this does not give me any additional info about these pipes.

Unfortunately without deeper knowledge about forking in Python, I don't see what causes the leak. Could anyone please have a look at forker.py to help me?

Best,
Tobias