TouchDB: Possible to sync attachments in wifi only?

[Paul Spencer]

We are implementing TouchDB iOS app with a feature that allows users to take photos.  Right now the photos are added as attachments to documents and eventually make their way to the central database via continuous sync'ing.  We are adding a big switch to allow users to turn off data transfer when they only have a 3G connection.  This controls a number of aspects of the application and basically hamstrings it unnecessarily if you are willing to transfer *some* but not *lots* of data.  What we would like to be able to do is to provide an option to enable data in 3G mode but disable transfer of large files, i.e. attachments.  Is this a planned (or perhaps already existing?) feature of TouchDB?  Can it be implemented on top of TouchDB in some reasonable fashion?  Is it even a sane idea?

Cheers

Paul

[J Chris Anderson]

On May 10, 2012, at 1:05 PM, Paul Spencer wrote:

We are implementing TouchDB iOS app with a feature that allows users to take photos.  Right now the photos are added as attachments to documents and eventually make their way to the central database via continuous sync'ing.  We are adding a big switch to allow users to turn off data transfer when they only have a 3G connection.  This controls a number of aspects of the application and basically hamstrings it unnecessarily if you are willing to transfer *some* but not *lots* of data.  What we would like to be able to do is to provide an option to enable data in 3G mode but disable transfer of large files, i.e. attachments.  Is this a planned (or perhaps already existing?) feature of TouchDB?  Can it be implemented on top of TouchDB in some reasonable fashion?  Is it even a sane idea?

It's a great idea and something we hear from lots of users.

I'd do it by using filtered replication so you can create a replication just of documents that don't have attachments over a certain size. Here's some code that filters based on a document property: https://github.com/couchbaselabs/TouchDB-iOS/blob/master/Demo-Mac/DemoAppController.m#L90

Note that replication is all-or-nothing for a document, so you'll need to keep your large photos as attachments to document you don't want to replicate. Eg you can't replicate a document and not it's attachments, you need to have documents that don't replicate at all, and documents that do.

Chris

[Dave Cottlehuber]

In "core" CouchDB, we are considering supporting storing attachments separately (e.g. on disk, in cloud/S3/flickr/CDN) and in that case this approach would definitely be worth adding.

A+

Dave

[Hank]

FYI there was a similar discussion here:  https://groups.google.com/forum/?fromgroups#!topic/mobile-couchbase/LnHJpZkAM9U 

[Kevin Nelson]

Note that replication is all-or-nothing for a document, so you'll need to keep your large photos as attachments to document you don't want to replicate. Eg you can't replicate a document and not it's attachments, you need to have documents that don't replicate at all, and documents that do.

I'm really glad I found this bit of info.  It might be good to put this in the docs somewhere accessible.

I'm still a complete n00b with Couchbase stuff, but I'm working in a similar problem space where it doesn't always make sense to sync an attachment, but it does always make sense to sync the metadata about the attachment.

This also gives way to the idea of lazy loading without actually syncing -- for example, if you were building an iPhoto replacement with TouchDB, you wouldn't necessarily want all of your photos from the past 5 years on one device since it might take up all of the storage you have on your phone, but you would still want to see thumbnails, or know that there are photos exist but are missing from the particular device.  And, as mentioned above, if you've taken photos that another client should acknowledge the existence of, but that haven't been uploaded because you've set the app to not upload over cellular data networks.

Unless it's realistic that that sort of functionality will get added (or that it will be accepted as a contribution), it sounds like TouchDB might not be the right thing for my particular use case. :(

[Kevin Nelson]

Sorry for the double post, but I also meant to ask, how would you model your documents to join the metadata to the attachment?

My first guess is that you'd do it in a view, and maybe set up a channel per device, and only add attachments to that channel if they should be synced to the device?

Thanks,

Kevin

[Jens Alfke]

On Dec 10, 2014, at 10:44 AM, Kevin Nelson <ke...@rkn.la> wrote:

Unless it's realistic that that sort of functionality will get added (or that it will be accepted as a contribution), it sounds like TouchDB might not be the right thing for my particular use case. :(

Hopefully "TouchDB" was a typo :)

Sorry for the double post, but I also meant to ask, how would you model your documents to join the metadata to the attachment?

Hmm, I think I'd have the main doc be like:

"type": "photo"

"name": "landscape.jpg"

"size": "1138527"

"image_id": "9AC6A1AC-C8CE-4765-85A0-6B0C92451A6B"

and then the doc with the attachment would be like

"_id": "9AC6A1AC-C8CE-4765-85A0-6B0C92451A6B"

"type": "attachment"

"_attachments": { … actual image attachment … }

In other words, the "image_id" property in the main/metadata doc is a "foreign key" that points to the attachment doc.

The sync function would assign "photo" docs to a "photos" channel, and "attachment" docs to a separate "attachments" channel. All users would have access to both channels channel, but the client's pull replication would explicitly only sync the "photos" channel.

Then you need a way to pull the attachment docs on demand; you can do that by starting a new pull replication with all channels, and give it a list of docIDs to pull.

—Jens