Replication Progress Bar on CBL 1.1.0

[David Quon]

I'm upgrading to CBL 1.1.0 from CBL 1.0.3.1.  Upon initial replication from CouchDB to CBL we show a progress bar using code similar to this.

    double progress = 0.0;

    double total = self.pullReplication.changesCount + self.pushReplication.changesCount;

    unsigned completed = self.pullReplication.completedChangesCount + self.pushReplication.completedChangesCount;

    

    progress = completed / total;

    NSLog(@"%f%@ - %u / %f", progress * 100, @"%", completed, total);

We then use `progress` to show a progress bar.  However the `total` seems to grow over a period of time during the replication making the `progress` percentage kind of useless as the percentage goes up and down not giving the user any useful information of actual progress.

Is there anything to be done to determine the actual total `changesCount` for pushing and pulling so the `progress` bar becomes useful again?  Is this a CBL 1.1.0 bug or just the expected behavior that just happened to work previously in CBL 1.0.3.1?

Here's sample data output from the `NSLog` above.

0.000000% - 0 / 13.000000

0.000000% - 0 / 38.000000

0.000000% - 0 / 52.000000

0.000000% - 0 / 65.000000

0.000000% - 0 / 78.000000

0.000000% - 0 / 86.000000

0.000000% - 0 / 103.000000

0.000000% - 0 / 117.000000

0.000000% - 0 / 150.000000

0.000000% - 0 / 168.000000

0.000000% - 0 / 182.000000

0.000000% - 0 / 190.000000

0.000000% - 0 / 197.000000

0.000000% - 0 / 201.000000

5.555556% - 13 / 234.000000

79.477612% - 213 / 268.000000

70.529801% - 213 / 302.000000

63.392857% - 213 / 336.000000

57.567568% - 213 / 370.000000

52.722772% - 213 / 404.000000

48.741419% - 213 / 437.000000

87.685775% - 413 / 471.000000

81.782178% - 413 / 505.000000

76.623377% - 413 / 539.000000

72.202797% - 413 / 572.000000

68.151815% - 413 / 606.000000

64.632238% - 413 / 639.000000

91.084695% - 613 / 673.000000

86.704385% - 613 / 707.000000

82.837838% - 613 / 740.000000

79.198966% - 613 / 774.000000

75.960347% - 613 / 807.000000

72.889417% - 613 / 841.000000

93.020595% - 813 / 874.000000

89.537445% - 813 / 908.000000

86.305732% - 813 / 942.000000

83.043922% - 813 / 979.000000

79.784102% - 813 / 1019.000000

95.566038% - 1013 / 1060.000000

92.090909% - 1013 / 1100.000000

88.859649% - 1013 / 1140.000000

85.774767% - 1013 / 1181.000000

82.964783% - 1013 / 1221.000000

96.193497% - 1213 / 1261.000000

93.164363% - 1213 / 1302.000000

90.454884% - 1213 / 1341.000000

87.834902% - 1213 / 1381.000000

85.422535% - 1213 / 1420.000000

96.847156% - 1413 / 1459.000000

94.262842% - 1413 / 1499.000000

91.872562% - 1413 / 1538.000000

89.600507% - 1413 / 1577.000000

87.384045% - 1413 / 1617.000000

97.403382% - 1613 / 1656.000000

95.106132% - 1613 / 1696.000000

92.968300% - 1613 / 1735.000000

90.924464% - 1613 / 1774.000000

88.968560% - 1613 / 1813.000000

97.894168% - 1813 / 1852.000000

95.875198% - 1813 / 1891.000000

93.937824% - 1813 / 1930.000000

92.077197% - 1813 / 1969.000000

90.288845% - 1813 / 2008.000000

88.568637% - 1813 / 2047.000000

96.500479% - 2013 / 2086.000000

94.729412% - 2013 / 2125.000000

93.022181% - 2013 / 2164.000000

91.375397% - 2013 / 2203.000000

89.785905% - 2013 / 2242.000000

97.018851% - 2213 / 2281.000000

95.429064% - 2213 / 2319.000000

93.890539% - 2213 / 2357.000000

92.555416% - 2213 / 2391.000000

91.257732% - 2213 / 2425.000000

98.009748% - 2413 / 2462.000000

96.713427% - 2413 / 2495.000000

95.413207% - 2413 / 2529.000000

94.147483% - 2413 / 2563.000000

92.914902% - 2413 / 2597.000000

91.714177% - 2413 / 2631.000000

97.791916% - 2613 / 2672.000000

96.670366% - 2613 / 2703.000000

95.574250% - 2613 / 2734.000000

94.571118% - 2613 / 2763.000000

93.421523% - 2613 / 2797.000000

92.266949% - 2613 / 2832.000000

98.150733% - 2813 / 2866.000000

97.133978% - 2813 / 2896.000000

96.072404% - 2813 / 2928.000000

95.001689% - 2813 / 2961.000000

93.954576% - 2813 / 2994.000000

92.930294% - 2813 / 3027.000000

98.431885% - 3013 / 3061.000000

97.350565% - 3013 / 3095.000000

96.292745% - 3013 / 3129.000000

95.257667% - 3013 / 3163.000000

94.009360% - 3013 / 3205.000000

92.936459% - 3013 / 3242.000000

98.046994% - 3213 / 3277.000000

97.040169% - 3213 / 3311.000000

96.082536% - 3213 / 3344.000000

95.115453% - 3213 / 3378.000000

94.195251% - 3213 / 3411.000000

93.265602% - 3213 / 3445.000000

98.102903% - 3413 / 3479.000000

97.153430% - 3413 / 3513.000000

96.249295% - 3413 / 3546.000000

95.335196% - 3413 / 3580.000000

94.464434% - 3413 / 3613.000000

99.067727% - 3613 / 3647.000000

98.152676% - 3613 / 3681.000000

97.280560% - 3613 / 3714.000000

96.398079% - 3613 / 3748.000000

95.556731% - 3613 / 3781.000000

94.705111% - 3613 / 3815.000000

99.064692% - 3813 / 3849.000000

98.222566% - 3813 / 3882.000000

97.369765% - 3813 / 3916.000000

96.556090% - 3813 / 3949.000000

95.731860% - 3813 / 3983.000000

94.945219% - 3813 / 4016.000000

99.086420% - 4013 / 4050.000000

98.261508% - 4013 / 4084.000000

97.473889% - 4013 / 4117.000000

96.675500% - 4013 / 4151.000000

95.913002% - 4013 / 4184.000000

95.139877% - 4013 / 4218.000000

99.082785% - 4213 / 4252.000000

98.319720% - 4213 / 4285.000000

97.500579% - 4213 / 4321.000000

96.761599% - 4213 / 4354.000000

96.033736% - 4213 / 4387.000000

95.230561% - 4213 / 4424.000000

98.922317% - 4406 / 4454.000000

98.260482% - 4406 / 4484.000000

97.715680% - 4406 / 4509.000000

97.826569% - 4411 / 4509.000000

97.848747% - 4412 / 4509.000000

97.893103% - 4414 / 4509.000000

98.070526% - 4422 / 4509.000000

98.092703% - 4423 / 4509.000000

98.624972% - 4447 / 4509.000000

100.000000% - 4509 / 4509.000000

[Jens Alfke]

> On Sep 2, 2015, at 3:50 PM, David Quon <dav...@amcoonline.net> wrote:
>
> Is there anything to be done to determine the actual total `changesCount` for pushing and pulling so the `progress` bar becomes useful again?

I feel your pain :( Unfortunately there’s a trade-off between accurate progress reporting, vs. performance and memory footprint. Specifically, if we determined the total number of docs (changesCount) ahead of time, it would slow down replication and/or consume more memory.

> Is this a CBL 1.1.0 bug or just the expected behavior that just happened to work previously in CBL 1.0.3.1?

I know of some specific changes that reduce memory usage during a first-time pull; they were made quite a while ago, but it’s likely they were after 1.0.3. The fix I’m thinking of was to address a problem where one developer’s app was literally crashing with an out-of-memory error when pulling 60,000+ documents.

To give you an idea of what’s going on: the puller has a couple of parallel tasks. One of them is reading the “_changes feed” from the server, which is a stream of document/revision IDs that are newer than what the client already has. Another task is sending HTTP requests to download those docs, and a third task is inserting the downloaded docs into the local db.

The puller doesn’t know how many docs it’s going to pull until it reaches EOF on the changes feed. On the other hand, if it reads the feed as fast as possible, it has to buffer all the doc/revision IDs in memory because the actual download process can’t keep up. That turned out to cause problems with large downloads. So now the feed is throttled so it doesn’t get too far ahead of the downloading … but that means the puller doesn’t know how many documents there are until fairly late in the process.

Another possibility would be to ask the server how many documents are in the feed before reading the feed. Unfortunately that would require Sync Gateway to do a lot of redundant work, and the task of generating that feed is already pretty expensive (and complex) on the server side, so I don’t think that’s feasible.

—Jens

[Brendan Duddridge]

So does this mean that this blog post that shows how to make a sync progress indicator on iOS is no longer relevant?

http://blog.couchbase.com/display-sync-progress-indicator-ios

I was hoping to add one to my app to show sync progress.

And as an aside, to respond to one of your points above, I can easily see my app requesting a hundred thousand documents to sync or more. I've been performance testing with one of my forms which has over 117,000 records in it, not to mention all the ancillary documents which will also need to be synced along with them. I'm assuming other than the time it takes to sync that much data that it shouldn't be an issue. I haven't begun adding sync into my app just yet (but I'm getting close).

Thanks,

Brendan