Searchfox Update: Searchfox knows (some) Python

[Andrew Sutherland]

Have you always felt like JavaScript was a big mistake and always
regretted that PyXPCOM never landed into Gecko?

While searchfox is neutral on issues like this, searchfox now has
significantly better support for Python[1] than JavaScript[2]!

For example, if you click on
https://searchfox.org/mozilla-central/query/default?q=calls-to%3A%27mozprofile.profile%3A%3AProfile%3A%3Aset_preferences%27+depth%3A4
searchfox will produce an SVG diagram[3] demonstrating control-flow
understanding of a method in mozprofile being called by mozrunner and
from there marionette_driver and wptrunner.

Searchfox's understanding of Python is thanks to
https://github.com/sourcegraph/scip-python which is built on top of
https://github.com/microsoft/pyright.  There is definitely more that
could be done here[4], but presumably if you add type annotations nice
things will happen.

Searchfox's choice of what Python to understand is encoded at
https://github.com/mozsearch/mozsearch-mozilla/blob/436855242ac7e4f56f37aecf12532e5db4c5b2a7/mozilla-central/build#L18-L56
and basically amounts to staying out of the WPT subtree because pyright
did not like the WPT subtree[5].  It also stays out of "third_party"
directories because I removed them first but maybe they could be added back.

Andrew

1: (Some) python.

2: As always, please do let your manager know if you think improvements
to searchfox could help make you more productive. While
https://github.com/sourcegraph/scip-typescript provides a potential
avenue to an improved JavaScript/TypeScript experience, a major
constraint is that because searchfox already provides a soupy[6]
semantic JavaScript experience it's very hard to make incremental
improvements without potentially regressing searchfox's existing JS
behavior.

3: The diagram is not currently accessible in any meaningful way and I
feel it's important to call this out.  Expanding on my most recent mail
about improvements to diagramming functionality, searchfox is now
populating a data structure that provides information about nodes and
the edges between them which is intended to allow
https://github.com/cmudig/data-navigator or a similar mechanism to
understand the graph at a semantic level and I am working towards this. 
Please do note that the improvements in searchfox's semantic
understanding of python are not limited to diagramming, however!  For
example
https://searchfox.org/mozilla-central/search?q=symbol:S_py_python_mozprofile.profile%2FProfile%23set_preferences().&redirect=false
is a search link on the root node of the diagram.

4: There are cases where a field is initialized to None and then
subsequently populated by a very straightforward class construction. 
Member accesses of that field lack semantic information, but it seems
quite reasonable to assume that the field is of the constructed type and
perform a lookup on its field/method accordingly.  This does not happen,
but feels like low-hanging fruit.

5: scip-python seems to feed pyright a bunch of data and then turns a
crank until the types converge or something like that. The types did not
converge!

6: Searchfox's JS analysis assumes all JS is loaded into a single global
soup.

[Mike Conley]

Exciting updates to Searchfox! Thank you!

I don't think I say this enough - Searchfox is probably the tool I use
most to get my job done. I suspect I spend just as much (or more) of my
time reading and examining code in Searchfox than I do writing or
modifying code in my editor. It's truly indispensable. Thanks for your work!

-Mike