Status of the Lanedo project

[Lionel Dricot]

Hello guys,

As you know, Lanedo is working on DXR and I thought it worth posting a
summary of what we have been doing and what are our objectives. I will
take the tasks that are in the current project plan.


Search improvements
1) Search should be case insensitive
+ Done

2) Multiple words search should return the intersection, not the union
+ Situation has been vastly improved: the search on multiple words
return now results where those multiple words are in the same file and
are nearby. Ideally, the union should only take the cases where the
words are on the same line but this will require subsantial work on the
tokenizer.

3) Allow searching by clicking on any word in the page
+ This is currently done but we are voluntaly avoiding searches for
words that have a meaning. Clicking on "anything" seems a bit confusing.
Discussion should happen to understand what is the desired behaviour.

4) Search results should be on par with MXR.
+ Carlos is in contact with the frontend team for that. Some use cases
should be given.

5) Searching for a file name should first returns the file if a perfect
match exists
+ Upon investigation, it appears that implementing this would way a lot
easier if the html was dynamic. We suggest investigating the switch to
mod_python or to Django before implementing this feature (Django is
proposed here : https://wiki.mozilla.org/DXR_Future_Work_Plan )

Bug fixing
6) Navigation between functions appears to be broken
+ The link on the icon should be implemented this week

7) Declaration/function switching doesn't work well
+ Lately, we haven't been able to reproduce that bug and all the
usecases we had seems fixed. We consider this bug as fixed except if we
receive an usecase.


Infrastructure
8) An automatic daily indexing process is required.
+ Some awesome improvements have happened on the indexing performances.
Remaining work implies making the update.sh more robust and setting the
cronjob. Nearly done.

User Interface
9) Navigation by clicking on sources should be possible in a new tab
using the middle button.
+ This is done
10) When clicking on an indentifier, DXR displays a popup with three
controls. Those will be flattened out and made into links that are
middle-clickable.
+ Not started yet

Not in the project plan:
11) We have made the logo on the frontpage configurable. There is still
the logo on the search page to make also configurable.


Lionel

[Josh Matthews]

On 12-03-28 11:53 AM, Lionel Dricot wrote:
> 4) Search results should be on par with MXR.
> + Carlos is in contact with the frontend team for that. Some use cases
> should be given.

I had a list of issues in https://etherpad.mozilla.org/DXRplan (under
jdm's bugs) - have these been looked at? For example, I just checked the
results for callers:nsIObserverService::NotifyObservers, and there are a
bunch of weird results that show up.

Cheers,
Josh

[Lionel Dricot]

Hi Josh,

You're right, I overlooked the Etherpad.

Could you group under one item all the bugs related to search? That way,
it is easy to know what is fixed and what should bedone for that
particular task.

(currently, it looks a bit like a brainstorm of ideas)

Lionel

Le 28/03/12 18:25, Josh Matthews a écrit :

> _______________________________________________
> dev-static-analysis mailing list
> dev-stati...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-static-analysis

[Josh Matthews]

Done. Please note that the issues I listed (in red) are almost entirely
ones I encountered while performing searches that frustrated me, not
simply things brainstorming "nice to haves".

Cheers,
Josh

[Carlos Garnacho]

Hey,

I didn't look yet into fixing those, but result coherence is indeed
important, so fixing that kind of bugs would be my next goal after basic
usability issues.

Carlos

[Ehsan Akhgari]

This looks very good, thanks!

On Wed, Mar 28, 2012 at 8:53 AM, Lionel Dricot <lio...@lanedo.com> wrote:

> 2) Multiple words search should return the intersection, not the union
> + Situation has been vastly improved: the search on multiple words
> return now results where those multiple words are in the same file and
> are nearby. Ideally, the union should only take the cases where the
> words are on the same line but this will require subsantial work on the
> tokenizer.
>

This is a lot better now.

> 3) Allow searching by clicking on any word in the page
> + This is currently done but we are voluntaly avoiding searches for
> words that have a meaning. Clicking on "anything" seems a bit confusing.
> Discussion should happen to understand what is the desired behaviour.
>

Hmm, for example clicking on eTypedBR here <
http://dxr.lanedo.com/mozilla-central/editor/libeditor/text/nsPlaintextEditor.h.html#l77>
doesn't work...

> 5) Searching for a file name should first returns the file if a perfect
> match exists
> + Upon investigation, it appears that implementing this would way a lot
> easier if the html was dynamic. We suggest investigating the switch to
> mod_python or to Django before implementing this feature (Django is
> proposed here : https://wiki.mozilla.org/DXR_Future_Work_Plan
>

Can you please explain why?

> 7) Declaration/function switching doesn't work well
> + Lately, we haven't been able to reproduce that bug and all the
> usecases we had seems fixed. We consider this bug as fixed except if we
> receive an usecase.
>

OK, I'll report if I see this again.

Cheers,
--
Ehsan
<http://ehsanakhgari.org/>

[Carlos Garnacho]

On mié, 2012-03-28 at 12:23 -0700, Ehsan Akhgari wrote:

> Hmm, for example clicking on eTypedBR here <
> http://dxr.lanedo.com/mozilla-central/editor/libeditor/text/nsPlaintextEditor.h.html#l77>
> doesn't work...

Right... that's something that deserves linking, I was mostly hesitant
about enabling this for parts of the language syntax (eg. searching for
"while" won't give you anything really meaningful), comments and such.

>
>
> > 5) Searching for a file name should first returns the file if a perfect
> > match exists
> > + Upon investigation, it appears that implementing this would way a lot
> > easier if the html was dynamic. We suggest investigating the switch to
> > mod_python or to Django before implementing this feature (Django is
> > proposed here : https://wiki.mozilla.org/DXR_Future_Work_Plan
> >
>
> Can you please explain why?

The html pages for source code are generated at index time and served
statically, yet we need something dynamic for the "did you mean...?"
box. I guess we can do this in a div loading another page, although it
would be more feasible if that content were generated dynamically, the
figures from the performance work keeps me optimistic about that :)

Cheers,
Carlos

[Ehsan Akhgari]

On Fri, Mar 30, 2012 at 8:46 AM, Carlos Garnacho <car...@lanedo.com> wrote:

> On mié, 2012-03-28 at 12:23 -0700, Ehsan Akhgari wrote:
>
> > Hmm, for example clicking on eTypedBR here <
> >
> http://dxr.lanedo.com/mozilla-central/editor/libeditor/text/nsPlaintextEditor.h.html#l77
> >
> > doesn't work...
>
> Right... that's something that deserves linking, I was mostly hesitant
> about enabling this for parts of the language syntax (eg. searching for
> "while" won't give you anything really meaningful), comments and such.
>

I agree that searching on language keywords is not useful, but comments and
strings could contain potentially useful things to search for:

// The logic here should match that of AwesomeClass::DoWonders
NS_ASSERTION(condition, "Make sure AwesomeClass::mMember is initialized");

> >
> >
> > > 5) Searching for a file name should first returns the file if a perfect
> > > match exists
> > > + Upon investigation, it appears that implementing this would way a lot
> > > easier if the html was dynamic. We suggest investigating the switch to
> > > mod_python or to Django before implementing this feature (Django is
> > > proposed here : https://wiki.mozilla.org/DXR_Future_Work_Plan
> > >
> >
> > Can you please explain why?
>
> The html pages for source code are generated at index time and served
> statically, yet we need something dynamic for the "did you mean...?"
> box. I guess we can do this in a div loading another page, although it
> would be more feasible if that content were generated dynamically, the
> figures from the performance work keeps me optimistic about that :)
>

I think the best way to solve this is to stick something in the URL like:

http://dxr.mozilla.org/path/to/file.cpp.html?searchterm=nsFooBar

And having a static script which tries to find "searchterm" in the query
string, and if it does, showing the "did you mean?" text. This has the
advantage of being more efficient to serve (no dynamic pages) and would not
affect the indexing speed (since the script can go in an external file
which will be linked to from the template.

How does this sound?

[Carlos Garnacho]

Hey :),

On vie, 2012-03-30 at 09:07 -0700, Ehsan Akhgari wrote:
> On Fri, Mar 30, 2012 at 8:46 AM, Carlos Garnacho <car...@lanedo.com> wrote:
>
> > On mié, 2012-03-28 at 12:23 -0700, Ehsan Akhgari wrote:
> >
> > > Hmm, for example clicking on eTypedBR here <
> > >
> > http://dxr.lanedo.com/mozilla-central/editor/libeditor/text/nsPlaintextEditor.h.html#l77
> > >
> > > doesn't work...
> >
> > Right... that's something that deserves linking, I was mostly hesitant
> > about enabling this for parts of the language syntax (eg. searching for
> > "while" won't give you anything really meaningful), comments and such.
> >
>
> I agree that searching on language keywords is not useful, but comments and
> strings could contain potentially useful things to search for:
>
> // The logic here should match that of AwesomeClass::DoWonders
> NS_ASSERTION(condition, "Make sure AwesomeClass::mMember is initialized");

That's a good point, I guess the HTMLifier could notice strings and
comments and set some ID so clicking on non-boilerplate only applies to
those.

>
>
> > >
> > >
> > > > 5) Searching for a file name should first returns the file if a perfect
> > > > match exists
> > > > + Upon investigation, it appears that implementing this would way a lot
> > > > easier if the html was dynamic. We suggest investigating the switch to
> > > > mod_python or to Django before implementing this feature (Django is
> > > > proposed here : https://wiki.mozilla.org/DXR_Future_Work_Plan
> > > >
> > >
> > > Can you please explain why?
> >
> > The html pages for source code are generated at index time and served
> > statically, yet we need something dynamic for the "did you mean...?"
> > box. I guess we can do this in a div loading another page, although it
> > would be more feasible if that content were generated dynamically, the
> > figures from the performance work keeps me optimistic about that :)
> >
>
> I think the best way to solve this is to stick something in the URL like:
>
> http://dxr.mozilla.org/path/to/file.cpp.html?searchterm=nsFooBar
>
> And having a static script which tries to find "searchterm" in the query
> string, and if it does, showing the "did you mean?" text. This has the
> advantage of being more efficient to serve (no dynamic pages) and would not
> affect the indexing speed (since the script can go in an external file
> which will be linked to from the template.
>
> How does this sound?

Nice suggestion, certainly requires less changes. I've went ahead and
implemented this for types/functions/filenames, it's now live in
dxr.lanedo.com :)

Carlos