DXR 2.0 staged. Feedback please!

[Erik Rose]

DXR 2.0 is about to land! This is a major revision touching every part of the system, swapping out SQLite for elasticsearch, and replacing many hard-coded C++ assumptions with a language-independent plugin interface.

Please take it for a spin on the staging server at http://dxr.allizom.org/, and see if you find any regressions from the production version at dxr.mozilla.org. You can file them directly at https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=DXR&status_whiteboard=es or just reply. Barring showstoppers, we plan to put it into prod within a few weeks.

What's new?

* Improved C/C++ analysis
* Multi-language support—Python and Rust, for starters, soon to be enabled for moz-central
* All queries are fast—and will be even faster in prod, once our webheads and elasticsearch servers are colocated
* Browsing of images
* Listing of binary files
* Result counts (so jrudermann can have googlefights)
* Independent tree indexing, so one build failure won't scuttle updates for the rest of the trees. (This will help us get all the trees currently under MXR indexed.)
* Parallel indexing so we can set the DC on fire
* New plugin architecture so we can add new languages, query types, and cross references easily (https://dxr.readthedocs.org/en/es/development.html#writing-plugins)

This is really a backend-focused release, but you can see some of the new possibilities start to leak out. I'm enthusiastic about the features this will enable next: better surfacing of symbols without having to know their type ahead of time, faceted drill-down, context for search results, and permalinks (our last major blocker to decommissioning MXR).

Thanks for helping test it out!

Erik Rose
DXR Lead

[Nick Fitzgerald]

Looking forward to eventually getting better JS searching via the
multi-language support!

What is the relative priority of permalinks to specific revisions?

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Erik Rose]

> What is the relative priority of permalinks to specific revisions?

High! My great new intern Peter has a PR open right now: https://github.com/mozilla/dxr/pull/419

[Ehsan Akhgari]

This is great to see Erik! Thanks everyone for their hard work!

I am wondering, how close are we to be able to index IDL/WebIDL files,
and navigate through JS and C++ callers and implementations of those
attributes/methods? That is currently the biggest reason why I have to
use MXR from time to time, and it would be nice to see DXR growing
support for this...

Cheers,
Ehsan

[Erik Rose]

> I am wondering, how close are we to be able to index IDL/WebIDL files, and navigate through JS and C++ callers and implementations of those attributes/methods? That is currently the biggest reason why I have to use MXR from time to time, and it would be nice to see DXR growing support for this...

With the ability to implement languages as plugins and have them cooperate on individual files, I don't think we're far. I do have a ticket open (https://bugzilla.mozilla.org/show_bug.cgi?id=1109804), but I know next to nothing about those IDLs. Would you be willing to spend a few minutes giving me a tour of what you'd like sometime after 2.0 lands?

[Jeff Muizelaar]

It looks like finding of overrides of virtual methods is missing from
DXR 2.0. Is this intentional?

-Jeff

On Wed, Jun 3, 2015 at 3:10 PM, Erik Rose <er...@mozilla.com> wrote:
> DXR 2.0 is about to land! This is a major revision touching every part of the system, swapping out SQLite for elasticsearch, and replacing many hard-coded C++ assumptions with a language-independent plugin interface.
>
> Please take it for a spin on the staging server at http://dxr.allizom.org/, and see if you find any regressions from the production version at dxr.mozilla.org. You can file them directly at https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=DXR&status_whiteboard=es or just reply. Barring showstoppers, we plan to put it into prod within a few weeks.
>
> What's new?
>
> * Improved C/C++ analysis
> * Multi-language support—Python and Rust, for starters, soon to be enabled for moz-central
> * All queries are fast—and will be even faster in prod, once our webheads and elasticsearch servers are colocated
> * Browsing of images
> * Listing of binary files
> * Result counts (so jrudermann can have googlefights)
> * Independent tree indexing, so one build failure won't scuttle updates for the rest of the trees. (This will help us get all the trees currently under MXR indexed.)
> * Parallel indexing so we can set the DC on fire

> * New plugin architecture so we can add new languages, query types, and cross references easily (https://dxr.readthedocs.org/en/es/development.html#writing-plugins)

[Erik Rose]

> It looks like finding of overrides of virtual methods is missing from
> DXR 2.0. Is this intentional?

Hmm, no. The tests seem to pass (https://github.com/mozilla/dxr/blob/es/dxr/plugins/clang/tests/test_overrides.py). Where are you seeing it?