make ptestlong on Big Sur

[Zachary Scherr]

Has anyone tried to run "make -j8 ptestlong" on Big Sur? Every time I run it, doctesting breaks with the following traceback:

Traceback (most recent call last):

  File "/Users/zscherr/sage/develop/src/bin/sage-runtests", line 182, in <module>

    err = DC.run()

  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/doctest/control.py", line 1236, in run

    self.run_doctests()

  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/doctest/control.py", line 937, in run_doctests

    self.dispatcher.dispatch()

  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2040, in dispatch

    self.parallel_dispatch()

  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 1935, in parallel_dispatch

    w.start()  # This might take some time

  File "/Users/zscherr/sage/develop/local/lib/python3.9/site-packages/sage/doctest/forker.py", line 2207, in start

    super(DocTestWorker, self).start()

  File "/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start

    self._popen = self._Popen(self)

  File "/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen

    return _default_context.get_context().Process._Popen(process_obj)

  File "/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 277, in _Popen

    return Popen(process_obj)

  File "/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__

    self._launch(process_obj)

  File "/usr/local/Cellar/python@3.9/3.9.1_4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 65, in _launch

    child_r, parent_w = os.pipe()

OSError: [Errno 24] Too many open files

make: *** [ptestlong] Error 1

Since I was trying to test in parallel I don't know exactly what's going on, but the last test that was attempted before this error was

sage -t --long --warn-long 53.1 --random-seed=0 src/sage/doctest/test.py

by itself this test seems to pass but causes a lot of random segfaults, although this is with homebrew's python 3.9.1.

[Matthias Koeppe]

I think this has started to show up, at least sporadically, with the multiprocessing changes made in/for python 3.8, see https://trac.sagemath.org/ticket/27754

Something definitely needs fixing.

[Zachary Scherr]

Interesting.  I've noticed the segfaults in the past, but never the "too many open files" error.  I'll try to play around with my ulimit and see if I can get the testing to finish.

[Zachary Scherr]

For the record, I was able to get make ptestlong it to finish by doing:

ulimit -n 4096