Building some big python packages requires a lot of time relating to xref-resolving logic

8 views
Skip to first unread message

Jun Shi

unread,
Apr 11, 2023, 1:17:37 AM4/11/23
to sphin...@googlegroups.com
Hi, I need to use Sphinx to build documents for some big python packages, but I found it took a lot of time. I created an issue[link] on Github. It's a blocking issue of our project, so I'm sending this mail and asking help from the dev channel.  

The problem is, building bigger packages in Sphinx takes much more time than smaller ones. As an example, building documents for azure-mgmt-containerservice costs nearly an hourWith python profiler, I found that most of the building time is spent by "fuzzy search" mode of the `PythonDomain.find_obj` method[link] instead of reading, transforming and writing document contents.
Picture3.png

I think there may be 2 problems causing this: 

1. Too much reference-parsing enters the "fuzzy search" mode when finding a matching object. For example, resolving a declaration ":type arg: int" will enter fuzzy search mode. However, I think from the code of a related method “domain.python.parse_reftarget” [link], only when “reftarget” startswith ".",  it should enter the "fuzzy search" mode in ` find_obj`. 

I added a UT to test_domain_py on my fork repo [link]. we can see when parsing a variable type of int, a 'refspecific' attribute is appended to the “pending_xref “node, and it will enter the “fuzzy search” mode, when the package is bigger, it’s costing a lot of time. 

The UT to parse variable declaration with `int` type: 

Picture2.png

pending_xref” node in the parsed doctree, it has “refspecific” attribute. 

Picture5.png

Then “find_obj” method will enter fuzzy mode. 

Picture6.png
Picture7.png
2. Searching all loaded objects for resolving “pending_xref” is inefficient. Can we add a cache for xref-resolving results, or can we have a switch (in config file) to turn off fuzzy reference resolving (if “fullname” doesn't have an exact match, directly trigger `missing_reference` event)? 

Thanks!

Reply all
Reply to author
Forward
0 new messages