Multipart Uploader & document body request size

[Brendan Duddridge]

Hi,

So my customers have been informed that Cloudant is moving their shared cluster to a new IBM Cloud system. They want all my customers that are using the old Cloudant to transfer over to new IBM Cloud Cloudant service. 

The problem I'm facing now in relation to Couchbase Lite 1.4.x is that they have reduced the document request size from 64 MB down to only 1 MB.

This is causing a problem with attachments on my CBLDocuments.

If I attach a file that's more than 1 MB, I get a "request entity too large" error.

I used Charles Proxy to look at the request and it seems that Couchbase Lite is sending both the attachment and request body in one request. The attachment in the request is bounded.

But I though the Multipart Uploader was supposed to send attachments in a  separate request? I'm not seeing that in my testing with Charles.

What's interesting is when I look in Charles Proxy, I get this for the request in the top part of the screen:

PUT ... size = 192.58KB Status = Failed

But if I look at the request headers, I see this:

Content-Type	multipart/related; boundary="18B75887-6450-4167-97C0-F2C47D37D700"
Accept-Encoding	br, gzip, deflate
Connection	keep-alive
Accept	*/*
User-Agent	CouchbaseLite/1.3 (Mac OS X 1.4.1)
Content-Length	1585013

So the content length is more like 1.58 MB.

Is there a way to enable attachment uploads as separate requests in Couchbase Lite 1.4.1?

I'm using macOS High Sierra 10.13.1 and iOS 11.1.2

Thanks,

Brendan

[Traun Leyden]

Mutli-part requests are all part of a single request, they are just chunked into multiple parts that are delimited by boundaries.

I would file a support ticket w/ IBM Cloud and try to get some guidance from them, but it sounds to me like you will either need to:

- Find another hosting provider

- Self-host

- Redesign your app so that docs with attachments are less than 1mb

- Redesign your app to use a separate system for storing attachments (eg, AWS S3)

That seems like a it's going to affect a lot of Cloudant users and maybe with some pressure from users they might consider increasing it.

[Brendan Duddridge]

I'm starting to feel like Cloudant is going to be a no-go moving forward.

Adding attachments to a CBLDocument is super convenient and it handles syncing of the documents automatically. By not storing the attachments in the database, I need to handle syncing of them independently. It's especially troublesome when using peer-to-peer syncing, which I offer as a choice to the user. But peer-to-peer isn't as convenient as cloud syncing and certainly not convenient when you want to sync with co-workers or family members.

I've searched for other CouchDB/Couchbase hosting providers, but nobody seems to be doing it outside of IBM. At least none that are still around. I've found some defunct ones or ones that require you to be a developer to understand what's going on. I've got regular consumers of my app who have signed up with IBM Cloudant to be able to sync their data. This is becoming quickly a bit of a nightmare for me and my customers. 

One of the most important things in the way I designed my app, is I never ever wanted to have access to my customers data. So self-hosting or designing my app to use AWS S3 isn't really an option.

The strange thing is IBM told me there was no limit on attachment size. Just a limit on the request size. Inline attachments couldn't be more than 1 MB, but external attachments could. But that doesn't make sense because how can you have a request be under 1 MB, but still allow attachments larger than that?

I'm confused.

Thanks

Brendan

[Traun Leyden]

It's possible that the 1MB limit is only for inline attachments that are base64 encoded.

Can you clarify with IBM Cloud support whether that limit is also true for standalone attachments?  

You might also considering just doing your own experiments to figure out what works and what doesn't.

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/9d3dd347-3a57-4c6f-a0b5-722014374182%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Priya Rajagopal]

Hey Brendan

"I've searched for other CouchDB/Couchbase hosting providers, but nobody seems to be doing it outside of IBM. "

I will DM you regarding this. There are  couple of other hosting partners like zData that our partner teams can make introductions to if you'd like.   We recently went through this exercise with another company who was migrating away from Cloudant (not for the attachment reasons that you mention which seems quite weird). 

regards

-Priya

[Brendan Duddridge]

This is what they told me:

For ordinary attachments, there is no limitation on the size. You can upload any attachment as you like. But for inline attachment, there is a limitation because inline attachments are attachments included as part of the JSON content. The document size limit on the old Cloudant service is 64M.

But I don't do inline attachments as far as I know. I use the setAttachmentNamed: function like this on my CBLModel object:

[self.formEntry setAttachmentNamed:filename

withContentType:mimeType

contentURL:url];

The request being sent to IBM Cloud looks like this:

--005D258A-9C60-4EFE-9E5B-41C6E2EEFC92

Content-Length: 8960

Content-Type: application/json

{"_attachments":{"A-Clean-IND-Long-Beach-Blue-Metallic-BMW-X5-M-Project-8.jpg":{"content_type":"image/jpeg","digest":"sha1-JZ1CrUxQlePeXnyIBs/RbAiHdjg=","follows":true,"length":1575753,"revpos":9}} .... the rest of the JSON document....

--005D258A-9C60-4EFE-9E5B-41C6E2EEFC92

Content-Length: 1575753

Content-Disposition: attachment; filename="A-Clean-IND-Long-Beach-Blue-Metallic-BMW-X5-M-Project-8.jpg"

Content-Type: image/jpeg

... the image attachment data

So it would seem that because the total of the Content-Length from both parts is being used to determine if the request is too large or not rather than just the first part and the second part being treated as an attachment. The main CBLDocument part of the request is only 8960 bytes.

Is there a way to coax Couchbase Lite into telling it to use a separate request for attachments?

Thanks,

Brendan

[Traun Leyden]

It looks like you're fine.  

Couchbase Lite will automatically determine whether to use base64'd inline attachments vs standalone multipart/related attachments.  I can't remember what the cutoff size is, but I think it's around 10K.

The request being sent to IBM Cloud looks like this:
--005D258A-9C60-4EFE-9E5B-41C6E2EEFC92

That's a standalone multipart/related attachment.

 

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchbase+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/1fc6a203-edc3-47bb-a7ec-60cc3c1afa68%40googlegroups.com.

[Brendan Duddridge]

Hi Priya,

I will DM you regarding this. There are  couple of other hosting partners like zData that our partner teams can make introductions to if you'd like.   We recently went through this exercise with another company who was migrating away from Cloudant (not for the attachment reasons that you mention which seems quite weird). 

Thanks for your reply. I am just a one-person company, so I don't have a budget for providing CouchDB hosting services to my customers. That's why the IBM Cloudant solution was perfect. They could sign-up on their own and have access to their own data. I don't have access to their data at all. All they needed was a username and password and my app would login and create the database for them and start syncing. That's it.

If there's other services out there that work like this, I'd love to be able to give them options.

Thanks,

Brendan 

[Jens Alfke]

On Dec 1, 2017, at 1:45 AM, Brendan Duddridge <bren...@gmail.com> wrote:

The problem I'm facing now in relation to Couchbase Lite 1.4.x is that they have reduced the document request size from 64 MB down to only 1 MB.

Are these guidelines public? I’d like to see more detail.

If I attach a file that's more than 1 MB, I get a "request entity too large" error.

I used Charles Proxy to look at the request and it seems that Couchbase Lite is sending both the attachment and request body in one request. The attachment in the request is bounded.

…

Is there a way to enable attachment uploads as separate requests in Couchbase Lite 1.4.1?

The push replicator has to send the body and (changed) attachments in a single HTTP request, because they have to be added to the server’s database atomically. There is no way in the replication protocol to push a document revision without the attachments that go along with it.

—Jens

[Jens Alfke]

On Dec 1, 2017, at 12:32 PM, Traun Leyden <traun....@gmail.com> wrote:

Couchbase Lite will automatically determine whether to use base64'd inline attachments vs standalone multipart/related attachments.  I can't remember what the cutoff size is, but I think it's around 10K.

Looking at the code, I think the only case where the push replicator sends attachments as inline base64 is if the server doesn’t support multipart … I believe this was for compatibility with old versions of PouchDB. Otherwise it’s always multipart.

—Jens

PS: If you’re following along in the code, the relevant variable is _dontSendMultipart in CBLRestPusher.m.

[Brendan Duddridge]

Hi Jens,

Are these guidelines public? I’d like to see more detail.

Yes, here's some docs mentioning the 1 MB limit:

https://console.bluemix.net/docs/services/Cloudant/api/document.html#documents

Specifically:

Note: If you are using a Cloudant NoSQL DB service on IBM Bluemix, documents are limited to a maximum size of 1 MB. Exceeding this limit causes a 413 error. 

 

 

The push replicator has to send the body and (changed) attachments in a single HTTP request, because they have to be added to the server’s database atomically. There is no way in the replication protocol to push a document revision without the attachments that go along with it.

So is it normal to consider the total of all the parts in a multi-part request to count against this request limit?

Thanks,

Brendan 

[Brendan Duddridge]

Looking at the code, I think the only case where the push replicator sends attachments as inline base64 is if the server doesn’t support multipart … I believe this was for compatibility with old versions of PouchDB. Otherwise it’s always multipart.

—Jens

PS: If you’re following along in the code, the relevant variable is _dontSendMultipart in CBLRestPusher.m.

Yes, I was inspecting the code and I turned on the Sync logging and I see the multipart uploader is being used:

Sync: CBLRestPusher[https://blahblah-bluemix.cloudant.com/db-71ac9de1a8d541f5bf3e4e544309a61c]: Queuing CBLMultipartUploader[PUT https://blahblah-bluemix.cloudant.com/db-71ac9de1a8d541f5bf3e4e544309a61c/rec-41e615fd7b7747ebb00a047ac412a2c5?new_edits=false] (multipart, 1547kb)

Thanks,

Brendan

 

[Jens Alfke]

On Dec 1, 2017, at 12:45 PM, Jens Alfke <je...@couchbase.com> wrote:

Looking at the code, I think the only case where the push replicator sends attachments as inline base64 is if the server doesn’t support multipart

Nope, Priya dug deeper and found that attachments under 2048 bytes will be sent inline.

But changing that limit won’t help; sending large attachments inline will just make the HTTP body larger (by 50%.)

—Jens

[Jens Alfke]

On Dec 1, 2017, at 12:51 PM, Brendan Duddridge <bren...@gmail.com> wrote:

Note: If you are using a Cloudant NoSQL DB service on IBM Bluemix, documents are limited to a maximum size of 1 MB. Exceeding this limit causes a 413 error. 

1MB for the body (JSON) of a document is reasonable, but from what’s happening to you, it sounds like they’re including attachments in this limit as well. You should probably contact Cloudant tech support and ask for details.

If they didn’t mean to limit docs+attachments to 1MB, then it sounds like their code that enforces the limit in HTTP requests was written without considering attachments, and needs to be fixed.

—Jens

[Brendan Duddridge]

So here's something interesting. IBM has their own Cloudant CDTDatastore framework for iOS that syncs with their Cloudant system.

Most of the code was written by you Jens :)

And they also do multipart/related uploads.

So I suspect the developers who used their own CDTDatastore framework are having the same problem.

Brendan

[Jens Alfke]

On Dec 8, 2017, at 5:17 PM, Brendan Duddridge <bren...@gmail.com> wrote:

So here's something interesting. IBM has their own Cloudant CDTDatastore framework for iOS that syncs with their Cloudant system.

Most of the code was written by you Jens :)

Yup, Cloudant forked the CBL codebase back in early 2013 (when it was still called “TouchDB”) and went their own way with it, unfortunately.

—Jens

Jens Alfke — Mobile Architect — Couchbase, Inc.