Python 3 error: RuntimeError: maximum recursion depth exceeded while pickling an object

[Tim Graham]

I've observed a transient issue when building Django's docs (traceback below). It only seems to happen on Python 3. Reproduced with Sphinx 1.3.1 and 1.2.3. I am not sure if the issue is with sphinx itself or with something else. Any debugging advice would be appreciated.

# Sphinx version: 1.3.1
# Python version: 3.4.0 (CPython)
# Docutils version: 0.12 release
# Jinja2 version: 2.7.3
# Last messages:
#   reading sources... [ 48%] ref/models/index
#   reading sources... [ 49%] ref/models/instances
#   reading sources... [ 49%] ref/models/lookups
#   reading sources... [ 49%] ref/models/meta
#   reading sources... [ 50%] ref/models/options
#   reading sources... [ 50%] ref/models/querysets
#   reading sources... [ 50%] ref/models/relations
#   reading sources... [ 50%] ref/request-response
#   reading sources... [ 51%] ref/schema-editor
#   reading sources... [ 51%] ref/settings
# Loaded extensions:
#   alabaster (0.7.3) from /home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/alabaster/__init__.py
#   djangodocs (unknown version) from /home/tim/code/djangoproject/djangoproject.com/djangodocs/1.8/docs/_ext/djangodocs.py
#   sphinx.ext.intersphinx (1.3.1) from /home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/ext/intersphinx.py
#   ticket_role (unknown version) from /home/tim/code/djangoproject/djangoproject.com/djangodocs/1.8/docs/_ext/ticket_role.py
#   sphinx.ext.viewcode (1.3.1) from /home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/ext/viewcode.py
Traceback (most recent call last):
  File "/home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/cmdline.py", line 245, in main
    app.build(opts.force_all, filenames)
  File "/home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/application.py", line 264, in build
    self.builder.build_update()
  File "/home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/builders/__init__.py", line 245, in build_update
    'out of date' % len(to_build))
  File "/home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/builders/__init__.py", line 259, in build
    self.doctreedir, self.app))
  File "/home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/environment.py", line 618, in update
    self._read_serial(docnames, app)
  File "/home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/environment.py", line 638, in _read_serial
    self.read_doc(docname, app)
  File "/home/tim/.virtualenvs/djangoproject-1.7/lib/python3.4/site-packages/sphinx/environment.py", line 863, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
RuntimeError: maximum recursion depth exceeded while calling a Python object

[Tim Graham]

I'm still seeing this with Sphinx 1.3.4 and found another report [1] that suggested sharing doctree files between builds may be the cause. Can anyone suggest whether this may or may not be a bug in Sphinx and what further debugging could be done to diagnose the issue? I could try similar measures found in the other report. Thank you!

Here's how we invoke Sphinx: https://github.com/django/djangoproject.com/blob/9a81c230955524ce74dbdbf2832615c9767db077/docs/management/commands/update_docs.py

[1] https://ghc.haskell.org/trac/ghc/ticket/10950

[Takayuki Shimizukawa]

Hi Tim,

I've never tried to build django documentation, but it seems it depends to depth of doctree structure.

Python reference said: "Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a RecursionError will be raised in this case. You can carefully raise this limit with sys.setrecursionlimit()." [1]_, please try it on your conf.py.

9 days before, haskell team seems to have resolved the issue by using this tweak [2]_.

.. [1] https://docs.python.org/3.5/library/pickle.html#what-can-be-pickled-and-unpickled

.. [2] https://git.haskell.org/ghc.git/commitdiff/e6bbaefcf7d0aa9ce5d6845cf82329e473fd88c0

Regards,

--

Takayuki SHIMIZUKAWA

http://about.me/shimizukawa

--
You received this message because you are subscribed to the Google Groups "sphinx-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sphinx-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Graham]

Thanks for the reply. I've done some more investigation and it looks like the problem is writing the doctree of a particularly long document docs/ref/settings.txt in the Django repo. The "maximum recursion depth exceeded" problem isn't deterministic though. I grabbed the doctree files of building the same documentation for both a crashing and non-crashing run.

settings.doctree-okay is 734kB
settings.doctree-error is 803kB and I had to set sys.setrecursionlimit(1137) to make it work.

I can provide these files in a bug report if you think it's worth further investigation.

Meanwhile, should we document the sys.setrecursionlimit() workaround?

[Takayuki Shimizukawa]

On Fri, Feb 12, 2016 at 4:25 AM Tim Graham <timog...@gmail.com> wrote:

I can provide these files in a bug report if you think it's worth further investigation.

 

Thanks!

If you have time, please file the problem to github issue.

Meanwhile, should we document the sys.setrecursionlimit() workaround?

Yes, please.

Regards, 

[Tim Graham]

Thanks.
Issue: https://github.com/sphinx-doc/sphinx/issues/2324
Doc patch: https://github.com/sphinx-doc/sphinx/pull/2325