Improve performance of qlog <dir>

[craig Hewetson]

I noted that bzr qlog <dir> was slow to show results. So I ran bzr log <dir> and found that although its slow to complete, it does show what it has available must faster than bzr qlog.

I suspect that qlog is trying to get a complete chunk of revisions before it shows them. Won't it be better for it to show whatever it has after a predefined period of time?

Also like we have already discussed before: I think that background threads will have to be used a bit more so as not to hang up the ui when doing any long operation.

[Gary van der Merwe]

HI Craig.

Let me first explain why qlog dir is so slow. The list of files in a
dir can change at each revision. Files can get added, and removed, and
moved in or out of the dir, so it has to load the inventory for every
revision, to get a list of the files in the dir at that revision.

bzr log dir is just doing this for the mainline revisions. bzr qlog
has to do this for all revisions. As I was writing this I was thinking
I could change qlog to first look at the main line, and then the other
revisions, but after I read the code, I remembered that this is not
possible.

I also thought that 2a format has a big performance improvement in
loading inventories, but quick wall clock timing of qlog dir showed it
is slower on a 2a repo. I need to investigate why.

So the other thing that seams to be irritating you is the ui lock up...

* Yes, we load in chucks. You can experiment with changing the chunk
size. lib/loggraphprovider.py line 792 (method load_filter_file_id)

* A threaded approach will solve the ui lock up problem. But this is a
huge undertaking.

There are other things I need to look at. Will do when I have some time...

Gary

[Gary van der Merwe]

On Fri, Sep 18, 2009 at 10:58 AM, Gary van der Merwe <gar...@gmail.com> wrote:
> bzr log dir is just doing this for the mainline revisions. bzr qlog
> has to do this for all revisions. As I was writing this I was thinking
> I could change qlog to first look at the main line, and then the other
> revisions, but after I read the code, I remembered that this is not
> possible.

This is because bzr log is doing a delta between the inventory and the
inventory of the rev -1.

qlog just looks for text id's, which is overall faster. May be we can
first do deltas for the main line, and then look for text id's for all
other revisions. This will be slower over all, but will give you
useful info quicker.

Maybe we could do just the main line first, and only filter the other
revisions that are on screen...

[Gary van der Merwe]

On Fri, Sep 18, 2009 at 10:58 AM, Gary van der Merwe <gar...@gmail.com> wrote:

> So the other thing that seams to be irritating you is the ui lock up...

Please try the latest version of trunk. I've changed something to keep
the ui more responsive. Please let me know if this is an improvement.

[craig Hewetson]

Thanks for looking at this issue.

I've asked this question because our development shop (15 developers) has just moved over to Bazaar from VSS.
And this was the one question/complaint that they had and I'm trying to "solve it".

The one approach is the make qlog more efficient and the other (probably more attainable approach) was to somehow get bzr search to index on paths and then allow qlog to make use of it.

Having problems connecting to launchpad but when I can connect I will test you improvements and give you feedback.

Thanks

[Gary van der Merwe]

I've just pushed a change the more than hales the total load time for
bzr qlog lib on the qbzr branch. \o/

[Gary van der Merwe]

On Fri, Sep 18, 2009 at 2:22 PM, Gary van der Merwe <gar...@gmail.com> wrote:
> I've just pushed a change the more than hales the total load time for
> bzr qlog lib on the qbzr branch. \o/

Note that this improvement was on a 2a branch. It speed stays the same
for 1.9 branches.

[Gary van der Merwe]

Related bug for bzr log : https://bugs.launchpad.net/bzr/+bug/374730