DXR performance work

[Carlos Garnacho]

Hey hey,

Even though I know performance wasn't part of this sprint's focus, I
felt the long indexing times were slowing down development and testing
more than necessary, so I've spent some time improving dxr-index.py and
the clang plugin to have it perform better.

With these patches [1], more operations happen directly on the database
instead of building the data map on memory, the result is that memory
usage has been decreased to 25% of the original one and the overall
indexing + html generation time has gone down from ~3h30m to 40-45m [2]

Of that reduced time, the first 1/3 is spent on creating the database,
the other 2/3 are spent on generating the html pages (~56K), which
results on an avg of 35-37 pages per second being created, I think this
timing is good enough to consider dynamic generation of source code
pages :).

dxr.lanedo.com now uses those patches for indexing, there shouldn't be
any major visible difference, so please tell me if there's some
inconsistency I oversaw.

Cheers,
Carlos

[1] https://github.com/garnacho/dxr/compare/2f6a5d348a...da902df303
[2] using mozilla-central

[Ehsan Akhgari]

This looks great, thanks a lot, Carlos! Please upstream this into the
mozilla/dxr repository.

Thanks!
--
Ehsan
<http://ehsanakhgari.org/>

> _______________________________________________
> dev-static-analysis mailing list
> dev-stati...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-static-analysis
>

[Josh Matthews]

On 12-03-30 11:26 AM, Carlos Garnacho wrote:
> With these patches [1], more operations happen directly on the database
> instead of building the data map on memory, the result is that memory
> usage has been decreased to 25% of the original one and the overall
> indexing + html generation time has gone down from ~3h30m to 40-45m [2]

Wow; nice job!

[Taras Glek]

On 3/30/2012 8:26 AM, Carlos Garnacho wrote:
> Hey hey,
>
> Even though I know performance wasn't part of this sprint's focus, I
> felt the long indexing times were slowing down development and testing
> more than necessary, so I've spent some time improving dxr-index.py and
> the clang plugin to have it perform better.
>
> With these patches [1], more operations happen directly on the database
> instead of building the data map on memory, the result is that memory
> usage has been decreased to 25% of the original one and the overall
> indexing + html generation time has gone down from ~3h30m to 40-45m [2]
>
> Of that reduced time, the first 1/3 is spent on creating the database,
> the other 2/3 are spent on generating the html pages (~56K), which
> results on an avg of 35-37 pages per second being created, I think this
> timing is good enough to consider dynamic generation of source code
> pages :).

Not sure if this is related, but clicking on stuff is completely broken atm.

Eg
http://dxr.lanedo.com/mozilla-central/dom/base/nsIScriptRuntime.h.html#l65

There one can't click on parseversion and clicking on nsIScriptContext
bring up a busted popup.

Can we rollback to less busted version?

Taras

[Carlos Garnacho]

Hey,

On lun, 2012-04-02 at 13:34 -0700, Taras Glek wrote:
<snip>

> Eg
> http://dxr.lanedo.com/mozilla-central/dom/base/nsIScriptRuntime.h.html#l65
>
> There one can't click on parseversion and clicking on nsIScriptContext
> bring up a busted popup.
>
> Can we rollback to less busted version?

Just a heads up that the performance changes are again live in
dxr.lanedo.com, I've fixed those issues and thoroughly compared
before/after outputs, it all looks pretty consistent now.

Carlos

[Taras Glek]

Since we now have json + testcase support. Can these regressions be
testcased so we don't run into them again?

Taras

[Taras Glek]

On 4/4/2012 7:03 AM, Carlos Garnacho wrote:

I did a search with following url
http://dxr.lanedo.com/search.cgi?tree=mozilla-central&string=SamplerStackFrameRAII


that ended up with following error
http://dl.dropbox.com/u/5961467/moz/error.html

Reloading the page corrected the error. I haven't seen this before, is
this due to recent changes?

Taras

[Taras Glek]

Guessing not as this seems to be caused by trailing whitespace in searches
>
> Taras

[Jean-Marc Desperrier]

Taras Glek a écrit :

>
> I did a search with following url
> http://dxr.lanedo.com/search.cgi?tree=mozilla-central&string=SamplerStackFrameRAII
>
>
> that ended up with following error
> http://dl.dropbox.com/u/5961467/moz/error.html
>
> Reloading the page corrected the error. I haven't seen this before, is
> this due to recent changes?

Managed to find another failure

- Load
http://dxr.lanedo.com/mozilla-central/tools/profiler/sps_sampler.h.html?string=SamplerStackFrameRAII#l144
- the yellow bar is a bit annoying as it can't be dismissed
- click on "SamplerStackFrameRAII" in line 144
- in the pop-up click on "Declared at tools/profiler/sps_sampler.h:147:3"
- loading
http://dxr.lanedo.com/mozilla-central/tools/profiler/None/mozilla-central/tools/profiler/sps_sampler.h.html#l147
which fails with a "Not Found" error.

http://dxr.lanedo.com/mozilla-central/tools/profiler/sps_sampler.h.html#l147
works correctly, so somehow a wrong "tools/profiler/None" got inserted
in side the URL.

[Lionel Dricot]

Le 22/05/2012 21:20, Jean-Marc Desperrier a écrit :
>
> Managed to find another failure
>
> - Load
> http://dxr.lanedo.com/mozilla-central/tools/profiler/sps_sampler.h.html?string=SamplerStackFrameRAII#l144
> - the yellow bar is a bit annoying as it can't be dismissed

For me, clicking anywhere on the yellow bar dismiss it. Is it not
working for you or should we make that more discoverable by putting a
close link on the right ?

> - click on "SamplerStackFrameRAII" in line 144
> - in the pop-up click on "Declared at tools/profiler/sps_sampler.h:147:3"
> - loading
> http://dxr.lanedo.com/mozilla-central/tools/profiler/None/mozilla-central/tools/profiler/sps_sampler.h.html#l147
> which fails with a "Not Found" error.

The link you've pasted looks wrong, indeed. But I cannot reproduce that
on dxr.lanedo.com. It might have been fixed in the mean time. Could you
try to reproduce ?

>
> http://dxr.lanedo.com/mozilla-central/tools/profiler/sps_sampler.h.html#l147
> works correctly, so somehow a wrong "tools/profiler/None" got inserted
> in side the URL.

Not for me :-(

Lionel

[Jean-Marc Desperrier]

Lionel Dricot a écrit :

> Le 22/05/2012 21:20, Jean-Marc Desperrier a écrit :
>> - the yellow bar is a bit annoying as it can't be dismissed
>
> For me, clicking anywhere on the yellow bar dismiss it. Is it not
> working for you or should we make that more discoverable by putting a
> close link on the right ?

You're right, I didn't expect I could dismiss it by clicking anywhere.

An hard to hit close link wouldn't be great, I think the best would be a
large "dismiss" button.

>> - click on "SamplerStackFrameRAII" in line 144
>> - in the pop-up click on "Declared at tools/profiler/sps_sampler.h:147:3"
>> - loading
>> http://dxr.lanedo.com/mozilla-central/tools/profiler/None/mozilla-central/tools/profiler/sps_sampler.h.html#l147
>> which fails with a "Not Found" error.
>
> The link you've pasted looks wrong, indeed. But I cannot reproduce that
> on dxr.lanedo.com. It might have been fixed in the mean time. Could you
> try to reproduce ?

I can't repro anymore this morning. But it's a bit annoying if you have
no idea why there was this result yesterday.

Globally DXR does seem to work quite well, even if it feels a bit rough.

I have a list of point which fell a bit annoying even if none is big
problem :
- the search when opening a pop-up feels a bit long. Given the kind of
interface that is used, you want this to be snappy and it's just not yet
- At least it would be help if the result was held in a local cache, and
reused the next time instead of searching again

The pop-up hides the rest of the code, I think it pushes you to open it,
want to check the code below so decide to close it first, then need to
reopen it.

- it should be easier to dismiss the pop-up. I feel clicking inside the
"Search for "xx"..." line should dismiss it.
- when the target of a link is within the last lines of the file, you
should still scroll it to the middle of the screen, showing a blank
space below it.
- the "Calls"/"Called by" lists are great, but it feels like the "Called
By" should be integrated more with the "References:" list.

OK, here's what I'm thinking (taking the example of the push function at
http://dxr.lanedo.com/mozilla-central/tools/profiler/sps_sampler.h.html#l207)
:
void ProfileStack::push(const char *)
Definition at tools/profiler/sps_sampler.h:207:8
Declared at tools/profiler/sps_sampler.h:207:8

References:
Called at tools/profiler/sps_sampler.h:261:10
by: void * mozilla_sampler_call_enter(const char *)

Calls:
Size_t mozilla::ArrayLength(T (&)[N]):

- I think the "Definition/Declared lines" for the function listed inside
"Calls/Called by" should not be shown by default, only after you click
on the function line
- Taking the example of mStack at
http://dxr.lanedo.com/mozilla-central/tools/profiler/sps_sampler.h.html#l235
If your are able to do the called by thing for function, it would be
great to do also this for variables (yes, always asking for more):
References:
Used at tools/profiler/sps_sampler.h:261:10
by: void ProfileStack::push(const char *aName)
- the same goes for things like "struct ProfileStack" at
http://dxr.lanedo.com/mozilla-central/tools/profiler/sps_sampler.h.html?string=ProfileStack#l164.
There's a long list of references, it would be great to know inside
which function each reference is.
- When there's a list of reference like that, an usual cinematic will be
to click on one of them, access the code, and then click again on the
identifier to display the list and go to another reference.
It therefore would be *really* useful to load the result from a local
cache instead of searching again. Will help the server too.

[Lionel Dricot]

Hi Jean-Marc,

It would be a lot easier if you could report all your issues directly on
bugzilla (under webtools -> DXR). That's the only task list we are using
for now.

Lionel