Achieving Sync Performance

[Tobin Harris]

Hi folks

It's taking about 5 minutes for our iPhone 4 to sync (pull) about 6000 documents over WiFi. These documents have about 15 fields - short strings and numbers.

If we want to reduce sync duration, are there any general guidelines? 

Smaller documents will sync faster because there's less data moving over the network, but I'm wondering if reducing 15 fields to 5 would give a significant improvement?

Cheers

T

[Jens Alfke]

On Oct 28, 2012, at 3:55 AM, Tobin Harris <to...@engineroomapps.com> wrote:

It's taking about 5 minutes for our iPhone 4 to sync (pull) about 6000 documents over WiFi. These documents have about 15 fields - short strings and numbers.

When I’ve done profiling in the past, the performance had almost nothing to do with the size of the documents (though I didn’t look at huge documents.) The bottlenecks were more to do with the parallelism of the replicator code. I made a bunch of improvements at the time, but that was months ago (in the spring?) and there have been a lot of changes since.

There are also some limitations of the CouchDB sync protocol that reduce efficiency — the biggest one is that most of the time it can only pull one revision per HTTP request. I have a partial workaround for that, but it only applies to first-generation revisions with no attachments.

If you want to investigate, use Instruments to do a time profile of a sync. (Make sure to do it on a real device, not the simulator, because they have very different performance characteristics.)

—Jens

[Tobin Harris]

Thanks Jens

So, does that mean for 6000 documents there's going to be 6000 HTTP connections?

For my context, I need to download download a lot of documents, then have ongoing sync. I wonder if there's any faster way to move the initial documents to TouchDB. Like downloading a single file and then bulk inserting. I guess the tricky bit will be making the revisions and ID's match up with that on the server. 

Tobin

[Jens Alfke]

On Oct 28, 2012, at 1:07 PM, Tobin Harris <to...@engineroomapps.com> wrote:

So, does that mean for 6000 documents there's going to be 6000 HTTP connections?

Any of the documents that are first-generation without attachments will be fetched in bulk (100 at a time IIRC).

The rest will be individual HTTP requests. Remember, keep-alive is used so that’s not individual connections!

For my context, I need to download download a lot of documents, then have ongoing sync. I wonder if there's any faster way to move the initial documents to TouchDB. Like downloading a single file and then bulk inserting. I guess the tricky bit will be making the revisions and ID's match up with that on the server. 

A workaround that was discussed once was downloading an entire TouchDB database file over HTTP, saving it into the same directory as your app’s other databases, opening it, then doing a local sync with it. I don’t know whether anyone got it working. (The hard part would be getting the server to generate a TouchDB-format file; it’s a completely different file format than CouchDB.)

—Jens

[Tobin Harris]

Thanks for the info Jens.

Downloading a pre-made TouchDb could be pretty cool. Is using the bulk insert method (putChanges) an option?

T

—Jens

[Jens Alfke]

On Oct 28, 2012, at 4:43 PM, Tobin Harris <to...@engineroomapps.com> wrote:

Downloading a pre-made TouchDb could be pretty cool. Is using the bulk insert method (putChanges) an option?

After downloading a database you’d use the replication API to pull from it.

—Jens

[Nicolas Lapomarda]

For what it's worth, I've used the -replaceWithDatabaseFile API before to install a canned TouchDB database downloaded from a server on first launch. It works fine but requires that a separate process generates the TouchDB db (in my case, a Mac app linked to TouchDB does it) and uploads it to the server whenever it needs to be updated.

Nico

--
 
 

[Paul Capestany]

Instead of making everyone always download all those docs, couldn't you just pre-populate TouchDB and have that included in the actual AppStore app with the docs already there? 

That way, while you might have some old data on first load, at least people would have something to use immediately, and only the changes would be downloaded to the device (instead of everything)?

[Jens Alfke]

On Oct 29, 2012, at 2:13 PM, Paul Capestany <cape...@gmail.com> wrote:

Instead of making everyone always download all those docs, couldn't you just pre-populate TouchDB and have that included in the actual AppStore app with the docs already there? 

It’s the same amount of data whether it gets downloaded along with the app, or on first launch. There are some benefits to doing it on the first launch, like keeping the app download smaller, and being able to change that database without having to go through App Store approval again.

—Jens

[Tobin Harris]

Including a baseline data se would be great, but for this app a completely different baseline is needed every day. So, we need to grab a ton of data each day that is used for 3-12 hours, then dumped.

[Eduardo Scoz]

For the record, I just wanted to point out a few of the performance issues we've had so far after a lot of testing syncing 10-20MBs of data at a time:

- when syncing against readonly db, local checkpoint is not used, which slows down replication startup - https://github.com/couchbaselabs/TouchDB-iOS/issues/184

- When syncing against cloudant, SSL causes the synchronization speed to slow down by as much as 50%. Using a username/password causes it to slowdown 6-8 times. Cloudant is/was investigating this; I need to check the ticket status. Removing both things speeds things up (from 6 mins to 30secs on my test case).

Anybody else had issues with SSL/auth performance?

[dyowee]

Yup, been having this issue with Cloudant for the past couple of days...