Searchfox (new code search tool)

[Bill McCloskey]

Hi everyone,

I would like to announce a new tool I've been working on for source
code searching called Searchfox (http://searchfox.org). If you use MXR
or DXR, I recommend you try Searchfox. Here are some of the benefits:

- Besides C++ code, Searchfox indexes JavaScript, XBL, and IDL. You
  can search by property name and, in some cases, qualified property
  name (e.g., SessionStore.duplicateTab). IDL files link to both JS
  and C++ implementations and users.

- Blame in Searchfox is fast and easy. Every file includes blame
  information in a gray bar on the left side, and walking through the
  blame history takes only one click per revision. Each file in the
  blame chain downloads quickly, and blame goes all the way back to
  1998. Say goodbye to the frustration of reaching "Free the
  (distributed) Lizard" at hg.mozilla.org and finding that GitHub
  blame times out!

- Searchfox jumps to the actual definition of methods rather than the
  header file declaration.

- C++ template handling is a little better, files download a little
  quicker, and other smaller improvements.

If you would like to try out Searchfox, I recommend that you change
your keyword searches to point to it. Otherwise it's too easy to
forget and revert to muscle memory.

Keyword search:
http://searchfox.org/mozilla-central/search?q=%s

Keyword search to find a particular file:
http://searchfox.org/mozilla-central/search?q=&path=%s

Some help on using Searchfox can be found at
http://searchfox.org. Also, you can see some screenshots at my blog:
https://billmccloskey.wordpress.com/2016/06/07/searchfox/

Also, here are some reasons not to use Searchfox:

- You frequently look at repositories besides
  mozilla-central. Searchfox only handles m-c.

- You like MXR's ability to sorta index all platforms. Like DXR,
  Searchfox uses a clang plugin that only analyzes Linux64 debug
  builds. I'm very eager to fix this problem, but it will take some
  time. Full-text search finds everything, of course.

-Bill

[Mike Hommey]

On Mon, Jun 06, 2016 at 09:35:34PM -0700, Bill McCloskey wrote:
> Hi everyone,
>
> I would like to announce a new tool I've been working on for source
> code searching called Searchfox (http://searchfox.org). If you use MXR
> or DXR, I recommend you try Searchfox. Here are some of the benefits:

I'm going to sound negative, but why? Or more precisely, why not
contribute to DXR to add those features that you implemented in
searchfox that DXR doesn't have?

MXR is already taking too long to fade out of existence, do we really
want yet another different tool?

Mike
_______________________________________________
firefox-dev mailing list
firef...@mozilla.org
https://mail.mozilla.org/listinfo/firefox-dev

[Sebastian Zartner]

On 7 June 2016 at 07:18, Mike Hommey <m...@glandium.org> wrote:
> MXR is already taking too long to fade out of existence, do we really
> want yet another different tool?

For some contributors (like me) this may be, because they simply don't
know that DXR is meant to replace MXR. If you want people to use DXR,
you should mention that on the MXR homepage and other places.

Sebastian

[Matthew N.]

It's more due to lack of mxr-parity rather than usage that is delaying the MXR retirement. See the dependency tree: https://bugzilla.mozilla.org/showdependencytree.cgi?id=1097091&hide_resolved=1

Matthew

[Gabor Krizsanits]

Wow, this is amazing. Million thanks for this, especially for the speed, the easy blame walk and the highlight. I always wanted to have a tool for the traversing history like that. One thing I miss is to see the revision number / contributor name on the side bar for each line on the sidebar in blame view. One question, is the source code public?

By the way I do not believe we all want the same thing from a code search tool, so I see nothing wrong in someone implementing a personalized version for his own taste, and some of us will benefit from it greatly.

- Gabor

_______________________________________________
dev-platform mailing list
dev-pl...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

[Andrew Halberstadt]

On 07/06/16 01:18 AM, Mike Hommey wrote:
> I'm going to sound negative, but why? Or more precisely, why not
> contribute to DXR to add those features that you implemented in
> searchfox that DXR doesn't have?
>

> MXR is already taking too long to fade out of existence, do we really
> want yet another different tool?
>

> Mike

+1. The technical achievement here is very impressive, and there looks
to be a ton of useful features. The blame walking looks especially
useful for me. But DXR is very actively developed, has 45 contributors
and 146 forks:
https://github.com/mozilla/dxr

I just don't see the point of competition between internal code
searching tools. I would love if ideas from searchfox (like blame
walking) could be incorporated into DXR.

Andrew

[Bobby Holley]

Searchfox is the best thing ever. It made a huge difference in the speed and ease with which I can query the codebase and get the answers I need.

My understanding is that Bill achieved a lot of the performance and correctness wins over dxr with different architectural choices, and by using Rust. I think the results validate his design choices, and I think it is very reasonable for him to have started from scratch in order to experiment without needing to align with the dxr roadmap.

I would be thrilled if dxr could incorporate the technology in searchfox or somehow achieve similar performance and features in some other way. But until that is shown to be possible, the existence of searchfox is a huge win for gecko hacking, and I am very grateful for the work Bill put into it (largely on his own time, iiuc).

bholley

On 07/06/16 01:18 AM, Mike Hommey wrote:

I'm going to sound negative, but why? Or more precisely, why not
contribute to DXR to add those features that you implemented in
searchfox that DXR doesn't have?

MXR is already taking too long to fade out of existence, do we really
want yet another different tool?

Mike

+1. The technical achievement here is very impressive, and there looks
to be a ton of useful features. The blame walking looks especially
useful for me. But DXR is very actively developed, has 45 contributors
and 146 forks:
https://github.com/mozilla/dxr

I just don't see the point of competition between internal code
searching tools. I would love if ideas from searchfox (like blame
walking) could be incorporated into DXR.

Andrew
_______________________________________________

[Astley Chen]

Agreed. As long as the tool helps and run faster, it is worth a try.

I’m also hoping that most of functions only on DXR can be ported to Searchfox soon,

for exp, “Jump to file”, “Jump to bugzilla”.

For syntax highlighting, just found literal string color and comment color are switched,

is there any reason to do so ? To me, the styling on DXR feels better.

—--
Best Regards,
Astley Chen | Mozilla Taiwan

[Richard Z]

On Mon, Jun 06, 2016 at 09:35:34PM -0700, Bill McCloskey wrote:

> Hi everyone,
>
> I would like to announce a new tool I've been working on for source
> code searching called Searchfox (http://searchfox.org). If you use MXR
> or DXR, I recommend you try Searchfox. Here are some of the benefits:

cool.

Btw - is there any tool to do code search in addons that are published
on AMO and allow source viewing?

Richard

--
Name and OpenPGP keys available from pgp key servers

[Bill McCloskey]

On Mon, Jun 6, 2016 at 10:18 PM, Mike Hommey <m...@glandium.org> wrote:

I'm going to sound negative, but why? Or more precisely, why not
contribute to DXR to add those features that you implemented in
searchfox that DXR doesn't have?

I started trying to contribute small changes to DXR, but I didn't like the design of the backend. At the time, DXR was switching from a SQL database to Elasticsearch. The idea of using a database or a big Java project to do something as simple as exact text matching and in-memory hashtable lookup (for C++ identifiers) seemed wrong to me. But I didn't think I could convince enough people working on DXR that I was right, so I forked it. Since then, I've made several other major backend changes: indexing is done mostly in Rust (for speed), and the whole thing runs in AWS (which means I'm not blocked on ops people to deploy new changes). I think those were both great decisions, but again, it would be a lot harder to do them in DXR.

I've talked to Erik Rose and Peter Elmers about how we can merge the projects back together. They've started to port my JS analyzer to DXR, and I'm guessing that work like that will continue. I'm also hopeful that we can work together on some projects, like indexing other platforms. It doesn't seem to likely that we'll be able to fully merge the projects anytime soon, but I don't think there's any great need to do so.

You can find the code for Searchfox at https://github.com/bill-mccloskey/mozsearch. I'm hoping to make it more friendly to contributions soon. It would be a great project to work on if you want to learn Rust or AWS or clang. That's why it's been fun for me.

 

MXR is already taking too long to fade out of existence, do we really
want yet another different tool?

I think all that matters is the pace of improvements to these tools. I think the existence of Searchfox will only speed this up.

 

Mike

[Gregory Szorc]

On Mon, Jun 6, 2016 at 9:35 PM, Bill McCloskey <wmccl...@mozilla.com> wrote:

Hi everyone,

I would like to announce a new tool I've been working on for source
code searching called Searchfox (http://searchfox.org). If you use MXR
or DXR, I recommend you try Searchfox. Here are some of the benefits:

- Besides C++ code, Searchfox indexes JavaScript, XBL, and IDL. You
  can search by property name and, in some cases, qualified property
  name (e.g., SessionStore.duplicateTab). IDL files link to both JS
  and C++ implementations and users.

- Blame in Searchfox is fast and easy. Every file includes blame
  information in a gray bar on the left side, and walking through the
  blame history takes only one click per revision. Each file in the
  blame chain downloads quickly, and blame goes all the way back to
  1998. Say goodbye to the frustration of reaching "Free the
  (distributed) Lizard" at hg.mozilla.org and finding that GitHub
  blame times out!

In case you aren't aware https://hg.mozilla.org/experimental/gecko-dev/ has the full CVS history and https://bugzilla.mozilla.org/show_bug.cgi?id=1265493 tracks making that more production suitable.

Furthermore, the MOSS grant awarded to the Mercurial project has a large component to improve the blame interface. That work is partially described at https://www.mercurial-scm.org/wiki/BlamePlan and is starting to land upstream. If all works out, its end state will leapfrog Searchfox's.

Also, hg.mozilla.org has JSON HTTP APIs to facilitate the building of alternate interfaces and tools. https://hg.mozilla.org/mozilla-central/help/hgweb contains the endpoints. Add "json-" in front of the command to get JSON. e.g. https://hg.mozilla.org/mozilla-central/json-annotate/tip/client.mk.

It is also often trivial to modify the hg.mozilla.org web interface. If you want features, file bugs and ask for them.

 

- Searchfox jumps to the actual definition of methods rather than the
  header file declaration.

- C++ template handling is a little better, files download a little
  quicker, and other smaller improvements.

If you would like to try out Searchfox, I recommend that you change
your keyword searches to point to it. Otherwise it's too easy to
forget and revert to muscle memory.

Keyword search:
http://searchfox.org/mozilla-central/search?q=%s

Keyword search to find a particular file:
http://searchfox.org/mozilla-central/search?q=&path=%s

Some help on using Searchfox can be found at
http://searchfox.org. Also, you can see some screenshots at my blog:
https://billmccloskey.wordpress.com/2016/06/07/searchfox/

Also, here are some reasons not to use Searchfox:

- You frequently look at repositories besides
  mozilla-central. Searchfox only handles m-c.

- You like MXR's ability to sorta index all platforms. Like DXR,
  Searchfox uses a clang plugin that only analyzes Linux64 debug
  builds. I'm very eager to fix this problem, but it will take some
  time. Full-text search finds everything, of course.

-Bill

[Peter Elmers]

As a DXR contributor, I'd like to point out it hasn't been sitting still recently either: xpidl (find definitions/jump to headers) and JavaScript (ported from Searchfox, in fact) support have landed and are just waiting a config change in the production builder to be live. C++ search, particularly on template usage, has improved recently.

Fun things like path promotion and context around search results are currently under review.

Peter (:new_one)