Searchfox Update: Searchfox result limits have been increased and it is now explicit when limits have been hit

[Andrew Sutherland]

Everyone is always talking about how great quality is, but what about
quantity?  To help address this shortage of quantity, Searchfox now will
display more results!  And because those people talking up quality make
a good point too, Searchfox results quality in the face of limits being
hit has also been improved. But because we don't want to forget about
quantity, Searchfox will now also tell you the quantities of results in
the various result headers!

Before the fix for https://bugzilla.mozilla.org/show_bug.cgi?id=1459188
landed, Searchfox would apply a result limit of 1,000 all over the
place; when looking at file names, when asking the livegrep codesearch
fulltext search server for results, and then when processing the
results.  Unfortunately, there were a few sub-optimal things going on:

- The result limit was only 1,000!  That's hardly any quantity! The
result limit is now 4,000.

- The hard file name limit of 1,000 was applied before categorizing
paths into Core/Third-Party/Test/Generated.  This meant that if there
were more than 1,000 test files that matched, Searchfox might not tell
you about any of the "Core" files that matched.  Whoops!  Searchfox now
uses a cut-off of 32,000 files before categorization and the 4,000
result limit is only applied after categorization, favoring Searchfox's
Core, Third-Party, Test, and then Generated prioritization.

- Searchfox was needlessly secretive about when limits were being hit,
leaving you to infer if a limit was hit by telling you the number of
results and what the maximum was.  When the string was "Number of
results: 1000 (maximum is 1000)" this was fairly clear.  But because the
fulltext search server won't add a batch of results that would put its
result set over the 1,000 limit, you could end up seeing a number of
results that was 20-30 shy of 1000 and it wouldn't be clear what that
would mean.  Now Searchfox is more explicit about when limits are hit. 
For example, if you search for just ".js" you will get a warning like
"Warning: The following limits were hit in your search: fulltext search
hit limit, file pre-filter limit, result count limit".  Is that more
clear? Eh, maybe not.  But it is more explicit!

Other notes:

- As the adage goes, there's no quantity in I, but there is a quantity
in team.  Many thanks to :Gijs for bringing up this issue in
https://chat.mozilla.org/#/room/#searchfox:mozilla.org today (and for
related concerns previously) and providing feedback as we iterated
towards a solution, :jwatt for having filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1748757 and :kats having
filed https://bugzilla.mozilla.org/show_bug.cgi?id=1459188 and his
investigation there[1].

Andrew

1: Note that while these bug numbers may raise the question of "hey! why
did it take so long to address this?!", it's only relatively recently
that we've had the spare performance capacity and the quantity AND
quality of tests necessary to make these changes with confidence.