possible optimisation of index process

[Treeve Jelbert]

when i run bup-index on a windows partition from linux, i see messages
like:

# ./bin/bup-win A4200 /mnt/wxp/
Indexing: 2, done.
bup: merging indexes (153251/153251), done.
Indexing: 1, done.
bup: merging indexes (153253/153253), done.
Indexing: 35, done.
bup: merging indexes (153288/153288), done.
Indexing: 1, done.
bup: merging indexes (153289/153289), done.
Indexing: 1, done.
bup: merging indexes (153290/153290), done.
Indexing: 82, done.
bup: merging indexes (153372/153372), done.
Indexing: 24975, done.
bup: merging indexes (178347/178347), done.
Indexing: 646, done.
bup: merging indexes (178993/178993), done.
Indexing: 1, done.
bup: merging indexes (178994/178994), done.
Indexing: 1, done.
bup: merging indexes (178995/178995), done.
Indexing: 9758, done.
bup: merging indexes (188753/188753), done.
Indexing: 1, done.
bup: merging indexes (188754/188754), done.
Indexing: 1, done.
bup: merging indexes (188755/188755), done.
Indexing: 1, done.
bup: merging indexes (188756/188756), done.
Indexing: 1, done.
bup: merging indexes (188757/188757), done.
Indexing: 5325, done.
bup: merging indexes (194082/194082), done.
Indexing: 1, done.
bup: merging indexes (194083/194083), done.
Indexing: 1, done.
bup: merging indexes (194084/194084), done.
Indexing: 491, done.
bup: merging indexes (194575/194575), done.
Reading index: 41326, done.
Saving: 16.06% (1290898/8036300k, 5953/41326 files) 12m28 8600k/s



Each merge step takes a long time.
There appears to a merge for each item in the /mnt/wxp/ directory.

Would it not be better to defer the merge until all the subtrees have
been indexed ?

I do not see this behaviour on ext4 systems, only with ntfs-3g

# ./bin/bup-root horos /
Indexing: 238737, done.
bup: merging indexes (414896/414896), done.
Reading index: 158374, done.
Saving: 100.00% (264582/264586k, 158374/158374 files), done.
bloom: adding 1 file (11165 objects).
4.9G /mnt/bup/root.bup
4.9G total

[Damien Robert]

Treeve Jelbert wrote in message
<30f3a7d1138fd58a...@scarlet.be>:

> when i run bup-index on a windows partition from linux, i see messages
> like:

> Would it not be better to defer the merge until all the subtrees have
> been indexed ?

> I do not see this behaviour on ext4 systems, only with ntfs-3g

When you do
bup index NAME path1 path2 path3
then bup index path1, merges, index path2, merge, and so on.


I am not sure why it does that too with only one NFS path.
In the code of index-cmd, there is:

paths = index.reduce_paths(extra)
update_index(paths, excluded_paths)

so maybe index.reduce_paths generate several paths, but i don't see how it
could be possible from the source code.

Anyway, in
https://groups.google.com/forum/?fromgroups=#!topic/bup-list/SxS3-P4Oars
I have posted a patch to update_index that only do the merge after it has
processed all the paths. Maybe it can help you in your situation too?

Cheers,
Damien