Uploading attachments using Multipart/related

[JC de Villa]

Hi guys,

I'm playing with multiple attachments using multipart/related as explained
in

http://docs.couchdb.org/en/latest/api/document/common.html#creating-multiple-attachments

I wind up with the error below

[Tue, 25 Mar 2014 08:33:59 GMT] [error] [emulator] Error in process
<0.28322.2> with exit value: {{nocatch,<<18
bytes>>},[{couch_httpd_db,receive_request_data,2},{couch_httpd,read_until,3},{couch_httpd,parse_part_header,1},{couch_httpd,parse_multipart_request,3},{couch_doc,'-doc_from_multi_part_stream/2-fun-1-',3}]}

This is on couchdb 1.5, on ubuntu 12.04 server, using curl from CLI.

Considering how simple the example is, I'm completely at a loss as to why I
can't even get that to work.

Any help would be appreciated.

--
JC de Villa

[Alexander Shorin]

Hi,

Could you show your curl command for upload?
--
,,,^..^,,,

[JC de Villa]

Sure.

curl -X PUT -H 'Content-Type: multipart/related; boundary="abc123"'
http://localhost:5984/odms_contents/abc -...@test.data

I have the data in a file called test.data, formatted exactly as in the
example.

Not sure what else I could be missing here.

--
JC de Villa

[Alexander Shorin]

On Tue, Mar 25, 2014 at 1:19 PM, JC de Villa <jc.de...@gmail.com> wrote:
> curl -X PUT -H 'Content-Type: multipart/related; boundary="abc123"'
> http://localhost:5984/odms_contents/abc -...@test.data

That's the problem. -d sends data as text. you should use
--data-binary or -T argument to send it as binary.

--
,,,^..^,,,

[JC de Villa]

Hmm,

I get the same using -T and --data-binary (tried again just now).

--
JC de Villa

[Nick North]

Do you have the same initial line breaks as in the example? I can't
remember the exact details at the moment, but mime parsing can be sensitive
about that and, if you are missing the line break before the first boundary
string, it might skip over the JSON body.

Nick

[JC de Villa]

I had the same thought in that it might be the parser complaining about
formatting, and I've made sure that it matches the example exactly.

--
JC de Villa

[Nick North]

Sounds as if that should work then. I'm not much of a curl user, so don't
know if there are any other potential problems in the command line. I'm not
around my CouchDb installations today, but I can construct a working
request this evening if no-one else beats me to it.

Nick

[JC de Villa]

Thanks Nick, much appreciated.

--
JC de Villa

[JC de Villa]

This is absolutely driving me nuts.

I'm sure it's easy. Uploading multiple attachments just don't seem to want
to work for me.

Argh!

--
JC de Villa

[Whorn]

you may want to read the src of the nodejs modules nano/request - search for "multipart"

[Jens Alfke]

On Mar 25, 2014, at 7:37 AM, JC de Villa <jc.de...@gmail.com> wrote:

> This is absolutely driving me nuts.
> I'm sure it's easy. Uploading multiple attachments just don't seem to want to work for me.

This sounds like my experience getting my replicator to interoperate with CouchDB a few years ago :)

Here’s a brain dump of things I remember:

* Make sure the line breaks in the MIME separators/headers are CRLF, not just LF!
* CouchDB crashes if a multipart body is sent in HTTP ‘chunked’ mode (COUCHDB-1403, filed by me two years ago and still unresolved. My colleague working on the Java port of my replicator just ran into this a few weeks ago.)
* I remember there being a bug in CouchDB where it required a CRLF after the closing MIME separator, i.e. the body had to end “--separator--\r\n” not just “--separator--“) but I can’t find a reference to the bug in my source code anymore. It may have been fixed.
* CouchDB used to ignore the headers in attachment MIME parts and assumed that the attachments appeared in the same order as in the “_attachments” object in the main JSON body. I believe this has been fixed and that it now looks at the Content-Disposition header to find the attachment’s filename, but I can’t remember for sure.

Hope this helps!

—Jens

[Nick North]

The "chunked" problem is fixed in branch 1956-attachment-handling, but I'm
not sure if that is ready to merge yet, so you're right that it is still a
problem at the moment.

I also vaguely recall the final CRLF problem, and have a feeling it was
patched a while back, but I'm not completely sure. I'll check....

The MIME headers are still ignored. I'd like to sort this out, but it
raises a host of questions about backward-compatibility, and what to do if
there is a mixture of _attachments entries and MIME headers, especially if
they are incompatible with each other. I haven't got around to thinking out
a proposal for answering those questions yet.

Nick

> --Jens

[Nick North]

Following up on my last message: the final newline is no longer required.

You could take a look at https://gist.github.com/NorthNick/9769261 - this
is a dump of a request that works on CouchDb 1.5 for me. As Jens says, the
crucial difference may well be in the use of CRLF around the boundary. The
first boundary string is preceded by CRLFCRLF and followed by CRLF; the
later ones are preceded by CRLF and followed by CRLFCRLF.

Nick

[JC de Villa]

Thanks everyone for looking into this.

I just got in a few minutes ago to tinker with this again. I'll post back
once I get it working (or not... :( )

--
JC de Villa

[JC de Villa]

Good lord. It WAS the newlines.

The only hurdle I have to get over now is when uploading multiple binary
attachments. But at least it's working now, and I can move on to figuring
that out.

Thanks Nick, Jens for the help. :)

--
JC de Villa

[Nick North]

Glad you got it working. Working with MIME is a whole world of pain; if
your have the option of using an existing library to construct your
requests, your life will be a lot easier. Or if you can use a library to
construct some test requests and then use them as a template for your own
code, that might also speed things up.

Nick

[JC de Villa]

We were initially planning on using pythons email.mime module to build the
request, but the ability to change the line separator is available only on
python >= 3.2.

Oh well, looks like we're rolling our own.

Thanks again. :)

--
JC de Villa

[Jens Alfke]

On Mar 26, 2014, at 3:07 AM, JC de Villa <jc.de...@gmail.com> wrote:

> We were initially planning on using pythons email.mime module to build the
> request, but the ability to change the line separator is available only on
> python >= 3.2.

Really? The line separator in a multipart document MUST be a CRLF. That’s hardwired into the specification*.

—Jens

* http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html

[JC de Villa]

On Wed, Mar 26, 2014 at 11:24 PM, Jens Alfke <je...@couchbase.com> wrote:

>
> On Mar 26, 2014, at 3:07 AM, JC de Villa <jc.de...@gmail.com> wrote:
>
> > We were initially planning on using pythons email.mime module to build
> the
> > request, but the ability to change the line separator is available only
> on
> > python >= 3.2.
>
> Really? The line separator in a multipart document MUST be a CRLF. That's
> hardwired into the specification*.

--Jens
>
> * http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html


Yeah. I saw that too. The gnarly details of the bug is in
http://bugs.python.org/issue1349106

It was around since 2005 and the fix was committed in 2010, which landed it
in 3.2.

And of course, nothing ships with python 3 by default...

Fun, fun, fun.


--
JC de Villa