Searchfox Update: calls-between and other diagram improvements

[Andrew Sutherland]

Did you know you can use searchfox's pre-alpha diagramming functionality
to try and figure out how one class might interact with another class or
type?  Drawing from a real world example in #xpcom, what if you knew a
class did something involving a delay that almost certainly involved a
timer, but weren't sure where the class or its related classes might be
involved?

Now you can write a query like
`calls-between-source:'mozilla::ProcessPriorityManager'
calls-between-target:'nsTimer'` and find out!
https://searchfox.org/mozilla-central/query/default?q=calls-between-source%3A%27mozilla%3A%3AProcessPriorityManager%27%20calls-between-target%3A%27nsTimer%27

Or if you were wondering what could make a docshell want to talk to the
external helper app service, a query like
`calls-between-source:nsDocShell
calls-between-target:nsExternalHelperAppService depth:10` can show you!
https://searchfox.org/mozilla-central/query/default?q=calls-between-source%3AnsDocShell%20calls-between-target%3AnsExternalHelperAppService%20depth%3A10

Both of those examples involve telling searchfox a class name. Searchfox
knows all the methods on each class and acts like you named all of the
methods individually.  (That's why the second diagram ends up including
a bunch of internal structure of the nsExternalHeleprAppService. 
Because they are reachable from nsDocShell.)  But for times when you
know one or more specific methods, you can specify them.  This can be
helpful for classes like nsGlobalWindowInner which frequently can have a
lot going on that may be interesting, but not what you were interested in.

For example, the query `calls-between-source:'nsGlobalWindowInner'
calls-between-target:'nsTimer' depth:8`[1] is the first depth level that
ClearTimeout starts to show up, but we were already learning about how
nsCaret::ResetBlinking could get involved at lower depths than that.  So
instead we could issue a query of
`calls-between-source:'nsGlobalWindowInner::SetTimeout'
calls-between-source:'nsGlobalWindowInner::ClearTimeout'
calls-between-target:'nsTimer' depth:9 paths-between-node-limit:12000`
https://searchfox.org/mozilla-central/query/default?q=calls-between-source%3A%27nsGlobalWindowInner%3A%3ASetTimeout%27+calls-between-source%3A%27nsGlobalWindowInner%3A%3AClearTimeout%27+calls-between-target%3A%27nsTimer%27+depth%3A9+paths-between-node-limit%3A12000

The existing "calls-between" mechanism still works, and in fact can be
mixed with "calls-between-source" and "calls-between-target" if
desired.  Symbols included by calls-between are treated as both sources
and targets which means that when a class is specified, internal
control-flow will always be present in the resulting diagram.  This is
great if you wanted to know how a class was arranged, like say with the
query `calls-between:'mozilla::dom::TimeoutManager'`[2], but can be
annoying if you just wanted to know how two distinct classes interact.

If you've tried "calls-between" before and found it slow or it felt like
it was maybe deciding the class you pointed it at had too many methods,
then there's good news!  Performance is now several orders of magnitude
improved, the limits on method combinations are gone, and in particular
a number of worst-case edge cases have been eliminated.  Node limits
will still apply. Specifically `paths-between-node-limit` (valid up to
16k) controls the number of nodes that are allowed to go into the graph
that we try to find the paths through, and then `node-limit` (up to 1k)
controls how many nodes can be in the resulting graph.  The `depth`
controls both the maximum depth of the breadth-first traversal as well
as the maximum path the graph algorithms are allowed to find.  If you
are hitting the node limit, you can lower the `path-limit` threshold
from 96 to something lower so that any node that has more than the
limit's in-edges will not be considered.

Other notable changes:

- Identifiers are now constrained such that you do need to provide
absolute pretty identifiers.  If your class is 'foo::Bar' you used to be
able to do 'Bar' and things would be fine, but this made it impossible
to avoid getting 'baz::Bar'.  If you use the existing diagram context
menus, it will always generate absolute pretty identifiers for you, so
you can use that as the start of your query.

- `calls-from` now knows to avoid traversing into methods like
`NS_DebugBreak` that otherwise clutter up your diagram.

- The same goes for `calls-to` and `calls-between` when it comes to
methods like "nsIObserver::Observe" which will not be useful until
searchfox is able to tuple the observer notifications over the specific
string notification that is used in those cases. Otherwise all code gets
related to all other code.  This can still happen for other interfaces,
but few interfaces are as analytically problematic.  Well,
nsISupports::{AddRef, Release, QueryInterface} were too, but that's why
they also got annotated. (Also some telemetry stuff that adds paths to
nsTimer.)

- There's now a ulimit -v in place for the pipeline-server if my
optimizations/improvements weren't good enough so that if the "query"
endpoint ends up using too much memory, its process will be terminated
rather than tar-pitting down the entire searchfox instance.  I'm not
saying that when I first went to write this email that I issued a query
so awesome that it took down the searchfox server, but... well, I did
that[3].  It should now be impossible to do that[4].

The usual caveats apply:

- This is only for C++ and languages searchfox has SCIP indexing support
for (Java/Kotlin/Python), not JavaScript/TypeScript.  If you work with
JS/TS and think functionality like this could help you be more efficient
by enabling you to more quickly and more deeply understand our complex
systems, please let your manager know so that they can pass the feedback
along to appropriate channels.

- If you want diagrams in your context menu, you gotta turn it on at
https://searchfox.org/mozilla-central/pages/settings.html although
calls-between does not show up in the context menu at this time.  (It
will probably end up in a cool, extremely optional sidebar thing.)

- The diagrams still do not generate a usable accessibility-tree.

- It will take up to 12 hours for trees other than mozilla-central to
switch over to the latest release with all the enhancements/fixes.

Andrew

1:
https://searchfox.org/mozilla-central/query/default?q=calls-between-source%3A%27nsGlobalWindowInner%27%20calls-between-target%3A%27nsTimer%27%20depth%3A8
2:
https://searchfox.org/mozilla-central/query/default?q=calls-between%3A%27mozilla%3A%3Adom%3A%3ATimeoutManager%27
3: We always have a spare mozilla-central server around for reasons like
this.  Not specifically this, mind you.  It's a generic "hot spare"
server, not an "Andrew oopsie" server.
4: Specifically, to write a query that takes down the entire server. 
You can still write awesome searchfox queries.  Also, if you put your
mind to it, maybe you can break diagrams for everyone else.