Storing large files especially zip in couchbase

[parvez....@decurtis.com]

For an iOS application backed by couchbase-lite, how one can upload zip files in couchbase database? Is using attachment the best way to achieve this? For example if an ios application contains a zip file of size 27MB that needs to be uploaded to a couchbase database. What API in couchbase lite iOS SDK will be useful to accomplish this ? Any code sample will be useful to understand this. If one use attachment approach then the files at iOS end will be stored in probably encrypted format. Unzipping of such files will also be required. How one can achieve this?

Thanks

[Brendan Duddridge]

Just use this API:

- (void) setAttachmentNamed: (NSString*)name

            withContentType: (NSString*)mimeType

                 contentURL: (NSURL*)fileURL;

You just have to provide a filename, a mimeType and a URL to the file.

To unzip the file you'll have to first get the attachment, write it to your own location and then unzip it.

- (nullable CBLAttachment*) attachmentNamed: (NSString*)name;

To unzip you'll need a zip library. UnzipKit is a good one which I use:

https://github.com/abbeycode/UnzipKit

The nice thing about Couchbase Lite is when you encrypt your database, it also encrypts all the attachments.

Thanks,

Brendan

[Brendan Duddridge]

Sorry, I forgot to mention that setAttachmentNamed is on CBLModel.

[Todd Freese]

Is there still a 20meg max limit on file attachments?

T

[Jens Alfke]

On Apr 7, 2017, at 10:30 AM, Todd Freese <to...@filmworkers.com> wrote:

Is there still a 20meg max limit on file attachments?

There is on Sync Gateway. Couchbase Lite has no limit.

—Jens

[Todd Freese]

Do you think the SG limit will ever be raised?

T

[Traun Leyden]

Yeah this is definitely possible, but requires some Sync Gateway work to be able to interface with a different storage system for the attachments.  (eg, something like https://minio.io/ or similar)

I would get in touch with Sachin Smotra who is the Couchbase Mobile product manager and ask to get that feature on the roadmap.  I'll send you a private message with his email address.

[parvez....@decurtis.com]

Since the attachments are stored as blobs in attachments folder where the local sqlite database for couchbase is stored. The blob file has extension .blob so how any unzipping library for example unzipkit is going to unzip it? We also need to keep in mind that attachments are also encrypted like the database.

Thanks

[Jens Alfke]

On Apr 13, 2017, at 12:51 AM, parvez....@decurtis.com wrote:

Since the attachments are stored as blobs in attachments folder where the local sqlite database for couchbase is stored. The blob file has extension .blob so how any unzipping library for example unzipkit is going to unzip it?

A filename extension is usually just a suggestion. If you give the path directly to an unzipping library, it shouldn’t care what the extension is.

We also need to keep in mind that attachments are also encrypted like the database.

This is why we discourage direct access to the file. The only reason it's allowed is because there are some iOS media APIs that insist on taking a file path or URL as input and can’t operate on an NSStream or raw data.

In general you should access attachments as streams or data. I took a look at unzipkit and it doesn’t seem to allow that; but it’s just a wrapper around zlib and I’m sure there are others that can operate on streams.

—Jens

[Brendan Duddridge]

I just read the stream from CBLAttachment's content and write it out to the temp folder or to a folder of the user's choosing. So even though the file on disk is a .blob, when the user chooses to save the file out from the database, the filename I present to them is the same used as the attachment name with the proper .zip file extension. Then you can use whatever program to unzip it.

[parvez....@decurtis.com]

Able to add attachments of sizes 11 MB, 19 MB but not of size 27 MB through the couchbase lite ios application. This means that the limit of 20MB applies to couchbase lite side also. 

Is it a good practice to start all downloads in a for / while loop (Initially the pull replication's downloadsAttachments property is set to false so that initial sync can happen smoothly)? Will the couchbase lite sdk be able to handle multiple downloads simultaneously at its end ? Or should the downloading of attachments be done sequentially i.e. initiating next download only when previous one completed.

Thanks

[parvez....@decurtis.com]

In Swift 3 (iOS), the below code snippet is not getting compiled:

progress.userInfo!.addObserver(self, forKeyPath: kCBLProgressErrorKey, options: [], context: nil)

where "progress" is an instance of Progress returned by calling downloadAttachment method on CBLReplication instance. The encountered error is:

Value of type '[ProgressUserInfoKey : Any]' has no member 'addObserver'

How one can handle error while performing attachment download?

[Jens Alfke]

On Apr 14, 2017, at 12:59 AM, parvez....@decurtis.com wrote:

Able to add attachments of sizes 11 MB, 19 MB but not of size 27 MB through the couchbase lite ios application. This means that the limit of 20MB applies to couchbase lite side also. 

No, it doesn’t. Something else must be going wrong. Please file an issue on Github and include details on the exact error you got, and if possible the code you’re using to add the attachment.

Is it a good practice to start all downloads in a for / while loop (Initially the pull replication's downloadsAttachments property is set to false so that initial sync can happen smoothly)? Will the couchbase lite sdk be able to handle multiple downloads simultaneously at its end ?

Deferred attachment downloads can run simultaneously. The only caveats are that

- The more you download at once, the slower each one downloads, so it takes longer for any of them to finish.

- NSURLSession has a limited number of HTTP sockets to the server, so it can only actually download that many at once (it seems to be 8 on macOS, 4 on iOS), and if you tie up all of those sockets downloading big attachments, you’ll starve the rest of the replicator until one of them finishes*.

—Jens

* One of the reasons for switching to a new non-HTTP-based replication protocol is to avoid problems like starvation. It’s actually come up many times during development, especially with the changes-feed tracker (can’t think how many hours I spent banging my head against that in 2012.)

[parvez....@decurtis.com]

Any idea on why I am getting compilation errors for adding observer on Progress's userInfo property?

Thanks

[Jens Alfke]

On Apr 15, 2017, at 11:54 PM, parvez....@decurtis.com wrote:

Any idea on why I am getting compilation errors for adding observer on Progress's userInfo property?

No. It’s really not going to be possible for someone to help with your compile error, unless you actually post the code and the error message. Bonus points for identifying what language and what version of Couchbase Lite you’re using.

Also, if you’re starting a new topic, don’t reply to an existing thread and take it off-topic. Please post a new message with the above info.

—Jens

[parvez....@decurtis.com]

 As per the issue opened on github it seems there is a limit of 20MB in attachment. 

@ Jens

Can you please clarify over this as there is a big confusion in it.

Thanks

[Jens Alfke]

On Apr 18, 2017, at 5:35 AM, parvez....@decurtis.com wrote:

 As per the issue opened on github it seems there is a limit of 20MB in attachment. 

The error you show in the bug report comes from the replicator, when it fails to pull the attachment from the server. That’s because the attachment couldn’t be pushed to the server, due to being over 20MB. (It looks as though Sync Gateway added the document body even though it failed to save the attachment; that’s a bug. It should have rejected the entire document during the push.)

If you look at the document in the local database, the attachment is there.

If you have a requirement to replicate larger attachments currently, you should look into using CouchDB or Cloudant as your back-end server. It’s also possible to store blobs elsewhere like S3; instead of using the attachment API at all, you just upload the blob there and then store the resulting URL into the document.

Can you please clarify over this as there is a big confusion in it.

I guarantee that there is no limitation on attachment size in Couchbase Lite. I am extremely familiar with the attachment code in the Cocoa implementation, having written most of it. There’s nothing in it that sets a limit. We have at least one unit test that stores a 30MB attachment in a database.

Sync Gateway actually doesn’t have a limit; it’s Couchbase Server that does. As Traun has said, at some point Sync Gateway will switch to using alternative storage for attachments to get around the limit. (I don’t work on Sync Gateway and don’t know their timeframe for doing that.)

—Jens